Benchmarking – a marketing tool

There has been a recently heated discussion when http://ormbattle.net/ published benchmarks that demonstrates that NHibernate performance is poor compared with other ORM frameworks. At first sight it looks like an valuable tool for assessing different frameworks speeds, but at a deeper analysis and in the light of the comments on Ayanide post «Benchmarks are useless, yes, again» it isn’t as clear anymore:

Trying to compare different products without taking into account their differences is a flawed approach. Moreover, this benchmark is intentionally trying to measure something that NHibernate was never meant to perform. We don’t need to try to optimize those things, because they are meaningless, we have much better ways to resolve things.

Making a benchmark is simple, make a loop and measure things. But this kind of benchmark only shows as much as the speed of runing some meaningless instructions in a non realistic loop.  Making a benchmark that handles real world scenarios is a much harder undertaking, one that could possibly show real insight into using a framework or another. But the frameworks are usualy very different and one way to do things in the first framework has no direct correspondence  in the other. It will just be a different thing to measure. In the end is all about marketing stuff.

If you want to show something is faster you can certainly make a benchmark to show it as another interesing battle at a whole another level clearly demonstrates. Microsoft published on its site a benchmark report entitled «Benchmarking IBM WebSphere 7 on IBM Power6 and AIX vs. Microsoft .NET on HP BladeSystem and Windows Server 2008» that demonstrates .NET on Windows is much better in terms of speed than WebShepre on AIX. IBM response was to make a benchmark of its own showing the opposite.

The bottom line is that benchmarks should be seen with reserves as most often are just marketing tools.