Benchmarking, especially micro benchmarking is a pretty complex topic. And the .net runtime makes things even more complicated.
You have your usual problems like making sure your code does the same things in each test, and that your timing is accurate. But .net also introduces JITing, and runtime optimizations. So you could start off writing some code and thinking its slower just to find that you were comparing already JITed code to unJITed code. Or maybe there was a DLL load in there.
Either way, if you’re going to do benchmarks of your code, you should do it right.
What that means in practice, is you should use something like OpenTelemetry traces to identify WHERE you should make improvements, and Benchmark.net to fine tune improvements.
There are plenty of times when you don’t need to do micro benchmarks. But if for example you’ve got one method that gets called thousands or millions of times, then micro benchmarking is the way to go.
I have had more than one occasion when I’ve painstakingly rewritten a method for performance, run the full application and it felt about the same. and then I benchmarked it, and it was worse than before, but the placebo effect tricked me into thinking it had gotten better.
Setup
With newer versions of C# and .net, using benchmark.net has never been easier. You don’t need a main, or any complex setup code. It really only takes two lines of code in your Program.cs file
|
1 2 3 |
using BenchmarkDotNet.Running; BenchmarkRunner.Run<T>(); |
All of your code for the benchmark will go into a class with the right attributes on it.
|
1 2 3 4 5 6 7 8 9 |
[MemoryDiagnoser] public class MyBenchmarks { [Benchmark] public void MyMethod1() {} [Benchmark] public void MyMethod2() {} } |
If all you want is just method timings, you don’t even need to put the [MemoryDiagnoser] attribute. If you leave it off, it will just show you the speed statistics. But I personally find that its useful to include the memory diagnoser because if you can use less time, and less memory, you reduce the risk of random GC cycles causing application stutters.
There are also other diagnosers available, but they’re pretty case specific, whereas the memory diagnoser is just generally useful. See documentation for more details.
Good Practices
You need to run the benchmarks in Release mode. And it should have whatever <Optimize> value you’re using for your real release. Otherwise it won’t be a true indication of what your code will do in the wild. This is most often <Optimize>true</Optimize> and getting confused about where Rider put the button for picking between Debug and Release.
This is where it is right now (classic UI, classic toolbar)
If you try to run in debug mode, it currently gives an error.
While you’re at it, make your environment as similar to your release environment as possible. If you run an x64 benchmark and an x86 application, or you’re running windows deploying in mono, you’re going to have differences.
If the differences in method performance are big, this may not matter. Fast code should be fast code. But I once ran .net core benchmarks for a .net framework application. And a lot of time trying to figure out why the numbers didn’t line up with what I was experiencing with a System.Diagnostics.Stopwatch in the application.
The .net JIT or even compiler can optimize your code to the point where it is no longer testing what you thought. So it is best to make sure you use, return, or both, anything that you’re trying to benchmark
|
1 2 3 4 5 6 7 8 9 10 11 |
//Bad void Blah() { Math.Min(0, 1); } //Good double Blah() { return Math.Min(0, 1); } |
That way the JIT can’t eliminate code, or optimize out any unused variables.
Do as little on your computer as possible while running the benchmarks. If you’re doing stuff, you could impact the results. This isn’t usually too big of a deal if you’re doing the same thing from run to run. But can leave you really confused when you have different results from day to day.
Use the same power/performance settings from run to run. This usually means plugging your laptop in, making sure you’re not thermal throttling, making sure you’re not using a low TDP profile, etc. Once again, its not a big deal if you’re consistent from run to run, but if you’re not, and the differences between timings are small, you may be tricked.