Comments on: Why many Java performance tests are wrong

By: Kirk

Kirk — Tue, 24 Feb 2009 07:48:23 +0000

Nice posting. I would add that you can see interference in the benchmark by calculating the variance. It should be close to zero in this case. Also you should investigate how the code was optimized to ensure that you don’t lose the effect you are trying to measure. You can use PrintComplied (iirc). Another recommended option is to dump the generated native code using a debug version of the JVM.

By: Bill the Lizard

Bill the Lizard — Fri, 20 Feb 2009 17:34:36 +0000

Good article. I’ve been guilty of this kind of performance non-testing myself in the past. It’s good to read a post that doesn’t just say “That’s wrong,” but also says why it’s wrong and what to do about it.

By: Daan

Daan — Wed, 11 Feb 2009 18:46:29 +0000

Hi Nils,

Glad you liked it!

- Daan

By: Nils

Nils — Wed, 11 Feb 2009 08:58:41 +0000

Really liked that article, gives some good hints… will do some further research on that topic.

Thanks man!

By: Cedric Franz

Cedric Franz — Fri, 06 Feb 2009 06:54:45 +0000

I have run many performance tests on our system and one of the settings that makes a big difference is whether the Java VM is started with the -server option or not. It is not clear cut whether the application will run faster with or without this option set because it largely depends on what the application does, but is is definitely worth trying. Our application runs about 10-15% faster under high load with the -server option.

By: Dror

Dror — Wed, 04 Feb 2009 13:49:10 +0000

http://www.javaedge.net/sessions/So%20you%20want%20to%20write%20your%20own%20benchmark.pdf

By: Chris Gummer

Chris Gummer — Tue, 03 Feb 2009 21:34:25 +0000

Nice article, thanks!

By: links for 2009-01-29 at DeStructUred Blog

links for 2009-01-29 at DeStructUred Blog — Fri, 30 Jan 2009 02:04:11 +0000

[...] Why many Java performance tests are wrong | StuQ.nl (tags: performance java) [...]

By: Jos Hirth

Jos Hirth — Thu, 29 Jan 2009 20:45:35 +0000

@Dimitris

The warm up phase is pretty short with the client VM. With micro benchmarks it’s usually over after less than half a second. Usually you’re interested in improving the performance of programs which run longer than that. Which is the reason why the warm up phase should be usually ignored (since it only skews the measured results for this kind of scenario).

E.g. no one will care if the path finding in your game is somewhat slow for the first 100msec. As long as you reach 60+ fps within a few seconds (and manage to stay there) everything is fine.

So, this isn’t about being objective or not. It’s about getting meaningful numbers. E.g. there are two algorithms. Lets call ‘em A and B. A needs a whopping 500ms for compilation, but only takes 10ms per iteration. Whereas B only needs 100ms for compilation and 50ms per iteration. Would you really want algorithm B in your application?

Now… if it’s a command line application, which quits after 1 second you might care for the warm up. Would be pretty pointless, but go ahead and waste your time. However, if it’s run for a few minutes or hours… or even days/weeks/years… you certainly won’t give a damn about warm up.

Since the warm up gets averaged out in your typical use case, you can get representative times much quicker by ignoring the warm up altogether. Without the warm up algorithm A is the clear winner.

By: turtlewax

turtlewax — Thu, 29 Jan 2009 18:55:59 +0000

>>The statistics you get during the warm up time should be discarded

I agree with Dimitris. Or at the very least, if you do discard warm-up, make sure that point is very well published. Its a well known issue with java, and omitting it will only fuel the criticisms.