The Mystery of the Slow First Run

The causes for performance issues can be found in the strangest places. This simple fact was proven again recently, when we analyzed an issue, a customer had reported to us. A servlet using our formatting processor PDFreactor was running fine in their Bea Oracle Weblogic 10.0 servers except for the fact that the first run after deploying took multiple minutes, which was slower than expected by a factor of 60 ("sixty"). Later runs performed normally, taking only split seconds.

We immediately were able to reproduce the issue in a newly installed Weblogic server. Switching from Bea Oracle JRockit to Sun Oracle Hotspot made no difference. Curiously the logs showed that it was not just a certain part of the conversion process hanging, but many separate steps performing exceptionally slow. So we set up an environment for profiling PDFreactor running inside the servlet in Weblogic using JProfiler. However, when we ran the profiling session the performance issue did no longer occur. So we disabled the profiler to make sure it doesn't influence the Java VM in a way that somehow solves the issue. But the first run still finished in seconds. We were not able to reproduce the issue anymore. So instead of profiling we had to do some detective work to solve this mystery read more...