Released: benchee 0.6.0, benchee_csv 0.5.0, benchee_json and benchee_html – HTML reports and nice graphs!

Released: benchee 0.6.0, benchee_csv 0.5.0, benchee_json and benchee_html – HTML reports and nice graphs!

The last days I’ve been hard at work to polish up and finish releases of benchee (0.6.0 – Changelog), benchee_csv (0.5.0 – Changelog) as well as the initial releases of benchee_html and benchee_json!

I’m the proudest and happiest of finally getting benchee_html out of the door along with great HTML reports including plenty of graphs and the ability to export them! You can check out the example online report or glance at this screenshot of it:

reportWhile benchee_csv had some mere updates for compatibility and benchee_json just transforms the general suite to JSON (which is then used in the HTML formatter) I’m particularly excited about the big new features in benchee and of course benchee_html!

Benchee

The 0.6.0 is probably the biggest release of the “core” benchee library with some needed API changes and great features.

New run API – options last as keyword list

The “old” way you’d optionally pass in options as the first argument into run as a map and then define the jobs to benchmark in another map. I did this because in my mind the configuration comes first and maps are much easier to work with through pattern matching as opposed to keyword lists. However, having an optional first argument already felt kind of weird…

Thing is, that’s not the most elixir way to do this. It is rather conventional to pass in options as the last argument and as a keyword list. After voicing my concerns in the elixirforum, the solution was to allow passing in options as keyword lists but convert to maps internally to still have the advantage of good pattern matching among other advantages.

The old style still works (thanks to pattern matching!) – but it might get deprecated in the future. In this process though the run interface of the very first version of run, which used a list of tuples, doesn’t work anymore😦

Multiple inputs

The great new feature is that benchee now supports multiple inputs – so that in one suite you can run the same functions against multiple different inputs. That is important as functions can behave very differently on inputs of different sizes or a different structure. Therefore it’s good to check the functions against multiple inputs. The feature was inspired by a discussion on an elixir issue with José Valim.

So what does this look like? Here it goes:

The hard thing about it was that it changed how benchmarking results had to be represented internally, as another level to represent the different inputs was needed. This lead to quite some work both in benchee and in plugins – but in the end it was all worth it🙂

benchee_html

This has been in the making for way too long, should have released a month or 2 ago. But now it’s here! It provides a nice HTML table and four different graphs – 2 for comparing the different benchmarking jobs and 2 graphs for each individual job to take a closer look at the distribution of run times of this particular job. There is a wiki page at benchee_html to discern between the different graphs highlighting what they might be useful for. You can also export PNG images of the graphs at click of a simple icon🙂

Wonder how to use it? Well it was already shown earlier in this post when showing off the new API. You just specify the formatters and the file where it should be written to🙂

But without further ado you can check out the sample report or just take a look at these images🙂

ipsboxplothistogram

raw_run_timesClosing Thoughts

Hope you enjoy benchmarking, with different inputs and then see great reports of them. Let me know what you like about benchee or what you don’t like about it and what could be better.

Video: Elixir & Phoenix – fast, concurrent and explicit

And here goes the video from Rubyconf Portugal – which was a blast! This talk mainly focuses on the latter explicit part of the title and how Elixir and Phoenix help with readable and maintainable code. It is also an introduction, quickly glancing at several topics that could also be topics of separate talks. This was at a ruby conference and I’m a ruby programmer, so parts of it are tailored to compare with Ruby, Object Oriented Programming and Functional Programming as well as likenesses and differences between Rails and Phoenix. Hope you enjoy!

You can also have a look at the slides right here or as PDF, speakerdeck and slideshare.

Abstract

Elixir and Phoenix are known for their speed, but that’s far from their only benefit. Elixir isn’t just a fast Ruby and Phoenix isn’t just Rails for Elixir. Through pattern matching, immutable data structures and new idioms your programs can not only become faster but more understandable and maintainable. This talk will take a look at what’s great, what you might miss and augment it with production experience and advice.

Released: deep_merge 0.1.0 for Elixir

As you might have seen on this blog or on twitter I’m thoroughly enjoying elixir. One thing that I found to be thoroughly missing is deep_merge – given two maps merge them together and if a merge conflict occurs but both values are maps go on and recursively merge them as well. In Ruby it is provided by ActiveSupport – you don’t need it THAT often but when you need it, it’s really great to have. The most common use case for me is merging a user specified configuration with some sort of default configuration to get the final configuration.

So at first no one else seemed to have the need for deep_merge, strange huh? About 1.5 months later it seemed others were having the same problem/question as in this stackoverflow question or the gist linked here:

So others do want it! Time to propose it to the elixir-core mailing list! Lots of people seemed to like the idea and were in favor of it, none of the core team members though and the discussion soon went on to be a bit more about implementation details. So, after some time I went and thought “might as well draft up an implementation in a PR so we can discuss this” – and so it was.

As you might have guessed from the blog post title, this PR didn’t get through as the core team thought it was a bit too specific of a use case, the implementation had its flaws (not handling structs properly – I learned something!) and it wasn’t general enough as it didn’t handle keyword lists. During the discussion José also mentioned that using protocols might be the way to go.

Using protocols seems to be the most correct but it feels a very niche feature to justify adding a new protocol to the language.

While I’d have liked to see it in Elixir core I gotta commend the elixir maintainers on rejecting features – I know it can sometimes be hard but in the end it’s for the better keeping the language focused and maintenance low. And one can always write a library so one can pick and choose to get the functionality in.

So what do you do? Well, implement it as a library of course! Meet deep_merge 0.1.0!

Why would you want to use deep_merge?

  • It handles both maps and keyword lists
  • It does not merge structs or maps with structs…
  • …but you can implement the simple DeepMerge.Resolver protocol for types/structs of your choice to also make them deep mergable
  • a deep_merge/3 variant that gets a function similar to Map.merge/3 to modify the merging behavior, for instance in case you don’t want keyword lists to be merged or you want all lists to be appended

All of these features – especially pit falls like the struct merging – are reasons why it might be profitable to adopt a library and not implement it yourself. So, go on check it out on github, hex and hexdocs.

Slides: Introducing elixir the easy way (Rubyconf Portugal)

A small lightning talk I gave at Rubyconf Portugal that is complementary to my “elixir & phoenix – fast, concurrent and explicit” talk in that it goes into how we integrated a Phoenix application into with our existing Rails application and clients.

Slides are available as PDF, speakerdeck and slideshare.

Abstract

Small lightning talk with some practical advice on how we integrated a Phoenix application in our general application landscape with a rails monolith and some frontend clients.

Slides: Elixir & Phoenix – fast, concurrent and explicit (Rubyconf Portugal)

And here go the slides for my elixir and phoenix talk focusing on the great features that both bring to the table and make your development experience nicer.

It is similar to the version presented at Codemotion Berlin, save for some minor tweaks and a hopefully more readable and stronger shade of green😀

So you can get the slides as PDF, speakerdeck and slideshare.

Abstract

Elixir and Phoenix are known for their speed, but that’s far from their only benefit. Elixir isn’t just a fast Ruby and Phoenix isn’t just Rails for Elixir. Through pattern matching, immutable data structures and new idioms your programs can not only become faster but more understandable and maintainable. This talk will take a look at what’s great, what you might miss and augment it with production experience and advice.

Slides: What did AlphaGo do to beat the strongest human Go player?

A talk about AlphGo and techniques it used with no prior knowledge required. Second talk of the Codemotion Berlin series, mostly the same talk I gave at Full Stack Fest. Something was cut/adjusted. A full recording from the Full Stack Fest version is available here.

You can get the slides via PDF, Speakerdeck and Slideshare.

Abstract

This year AlphaGo shocked the  world by decisively beating the strongest human Go player, Lee Sedol. An accomplishment that wasn’t expected for years to come. How did AlphaGo do this? What algorithms did it use? What advances in AI made it possible? This talk will briefly introduce the game of Go, followed by the techniques and algorithms used by AlphaGo to answer these questions.

Slides: Elixir & Phoenix – fast, concurrent and explicit (Codemotion Berlin 2016)

This is a remix and extended version (40 minutes) of the elixir and Phoenix talk I gave in the beginning of the year at the Ruby User Group Berlin and Vilnius.rb. It is an introductory talk about Elixir and Phoenix that shortly dips into the performance aspect but then switches over to features of the Elixir programming languages, principles and how it all comes together nicely in Phoenix. Also known as why do I want to program with Elixir and Phoenix – performance/fault tolerance aside.

This talk was the first talk I gave at Codemotion Berlin 2016.

Slides are embedded here or you can get the PDF, Speakerdeck or Slideshare.

There is no video, sadly – however there is a voice recording for which you can take and then click through the slides: voice recording.

Abstract

Elixir and Phoenix are known for their speed, but that’s far from their only benefit. Elixir isn’t just a fast Ruby and Phoenix isn’t just Rails for Elixir. Through pattern matching, immutable data structures and new idioms your programs can not only become faster but more understandable and maintainable. This talk will take a look at what’s great, what you might miss and augment it with production experience and advice.

Benchee 0.5.0 released – unit scaling for your convenience!

Another month and another release of my Elixir (micro-) benchmarking library benchee. This particular release focusses on unit scaling. E.g. instead of displaying that you had 12_345_678 iterations per second it will now say that there were 12.35 M iterations per second. The same goes for the time as well. There are four different strategies to choose from determining how units should be scaled. My friend and old Shoes (Ruby tooklkit/DSL for building GUIs) companion Eric Watson aka @wasnotrice did the bulk of the work. Thanks!

As usual, the nitty-gritty details are in the Changelog.

Why unit scaling?

The units employed so far were not ideal. Who really works with microseconds all the time and like to read full numbers over a million while only the first couple of places really have an impact? I think it’s easier to work with units closer to what a number really is. If something takes 5_632 microseconds to execute I’m much better off knowing that it takes about 5.63 milliseconds.

So from now on benchee will use one of its four strategies (one of which is none, if you don’t like this behaviour at all) to determine what the best unit to represent the benchmarking results in might be. For the canonical flat_map vs. map.flatten example the result might look like this:

See how the units were automatically scaled to thousands/milliseconds respectively? Now, you might not like that because you always want there to at least be a “1” before the dot. No problem, just use another scaling strategy: smallest!

This is now (in this case) pretty much like the output you’d get in previous benchee versions. Still, smallest is different from none in that if both averages were at least a millisecond they would still be displayed in milliseconds.

Under the hood this is all nicely handled by units (Count, Duration) implementing the Scale and Format behaviours while relying on a Unit struct.

What’s next for benchee?

The next bigger topic that I’ve put quite some time and experiments in is an HTML formatter with fancy graphs and an image export. Want a sneak-peak? Ok, since you asked nicely:

IPS comparisonBoxplot

Video: What did AlphaGo do to beat the strongest human Go player?

The publishing/video partner of Full Stack Fest was amazingly fast in publishing the video. Kudos to them! So after publishing the slides here goes the video!

If you want to have the slides, here they are ( or via links PDF, Speakerdeck, Slideshare):

In case you want to see it live, the talk will be up again at Codemotion Berlin.

Abstract

This year AlphaGo shocked the  world by decisively beating the strongest human Go player, Lee Sedol. An accomplishment that wasn’t expected for years to come. How did AlphaGo do this? What algorithms did it use? What advances in AI made it possible? This talk will briefly introduce the game of Go, followed by the techniques and algorithms used by AlphaGo to answer these questions.

PS: Yes, Lee Sedol probably wasn’t THE STRONGEST human player – more like Top 3 or Top 5 at the time of the game (most people would probably call Ke Jie the strongest player at the moment). Lee Sedol is the dominant of the last decade though, and when the match was announced nobody on the computer-go mailing list complained about the opponent, so I just assumed he was the strongest or among the strongest but only found out after submitting the talk🙂 . Plus, sadly “How did AlphaGo beat one of the top 5 Go Players” isn’t as catchy as a title.

Benchee 0.4.0 released – adjust what is printed

Today I made a little 0.4.0 release of my elixir benchmarking library benchee. As always the Changelog has all the details.

This release mainly focusses on making all non essential output that benchee produces optional. This is mostly rooted in user feedback of people who wanted to disable the fast execution warnings or the comparison report. I decided to go full circle and also make it configurable if benchee prints out which job it is currently benchmarking or if the general configuration information is printed. I like this sort of verbose information and progress feedback – but clearly it’s not to everyone’s taste and that’s just fine🙂

So what’s next for benchee? As a keen github observer might have noticed I’ve taken a few stabs at rendering charts in HTML + JS for benchee and in the process created benchee_json. I’m a bit dissatisfied as of now, as I’d really want to have graphs showing error bars and that seems to be harder to come by than I thought. After D3 and chart.js I’ll probably give highcharts a stab now. However, just reading the non-commercial terms again I’m not too sure if it’s good in all sense (e.g. what happens if someone in a commercial corporation uses and generates the HTML?). Oh, but the wonders of the Internet in a new search I found plotly which seems to have some great error bars support.

Other future plans include benchmarking with multiple input sizes to see how different approaches perform or the good old topic of lessening the impact of garbage collection🙂