In the previous post http://www.shunra.com/shunrablog/index.php/2009/08/14/data-center-relocation-questions-and-answers-part-1/ I presented an example of a common performance problem with applications that host executables on a remote shared drive. As common as that problem is, it is usually a legacy problem, most new applications follow a more best practices architecture usually involving a web based front end for the application. However even web based applications can provide their share of performance challenges. The following example presents an Oracle Clinical application commonly used in pharmaceutical companies especially during the clinical trial phases (which is one of the most critical business process a Pharma could have).
In alignment with global trends, more and more clinical trials take place outside of the United States, while the documentation and analysis is done in the US for submission to the FDA. So it is not out of the ordinary to find an Oracle Clinical user in Eastern Europe or in China submitting data to an Oracle Clinical Server hosted in the united states.
Unfortunately, as commonly found with an off the shelf software package that gets customized over time, this type of application can easily morph into a performance challenged application, especially for remote users. The following analysis shows an example of an Oracle Clinical transaction that completes in 8 seconds for local users, while extending to over a minute when users in E. Europe tried to use it.
Clik here to view.
Transaction Response Time Analysis for an Oracle Clinical Application Accessed by Local Users Vs. Remote Users in E. Europe
Obviously, that big jump in response time was a great cause for concern with my client’s management. Which is how I got the privilege to be asked to analyze the root cause of the poor performance.
The first thing I noticed was that one particular transaction (Connect to DB) had an extremely high jump in TRT from 8 seconds locally to over a minute when accessed remotely, so I focused on that one first.
Looking at the network fingerprint of the transaction you can observe the following things:
The transaction size (amount of data downloaded from the server required to complete the transaction) is extremely high (over 5 MB).
The transaction generates 34 HTTP calls. Some of those calls take disproportionally longer time than others (notice the Get jar file calls that each exceed 19 seconds marked below, click on the image for a larger view)
Clik here to view.
Oracle Clinical, Transaction Analysis for "Connect to DB"
When inspecting those JAR files, we saw that they were very heavy in size, over 1 MB each which is pretty big for basically a collection of Java classes that are zipped into a JAR file.
You can also observe in the next bounce diagram that most of the JAR files are downloaded in a serial fashion, blocking other objects from downloading in the mean time. Notice the delta time displayed on the right column
Clik here to view.
Bounce Diagram for an Oracle Clinical Transaction
What can be done to improve the performance of this application?
Oracle Clinical uses JAR files to package Java code that will run on the client. It is not uncommon for customized off the shelf applications to increase in code size over time until the code is very bloated.
There are several best practices that developers can follow to reduce the size of JAR files:
- Rationalize the code – as applications develop over time, multiple classes and sometimes adjacent projects might reference similar libraries and other assets. If not careful those libraries and common assets end up packaged multiple times inside the JAR file, causing it to inflate in size.
- Minify the code – a quick Google search for “reduce the size of JAR files” will reveal several free tools that can minimize the size of JAR files, usually through eliminating white spaces, comments, shortening variable names, etc.
- Defer loading – chances are that not all the code in the JAR file is needed for the “connect to DB” transaction, which means that users that only want to perform a small transaction are penalized by the download time of code that will never get executed by them. Deferred loading is a design pattern that simply says “only download assets or code when it is needed” I wrote more on that design pattern in this previous post “http://www.shunra.com/shunrablog/index.php/2009/01/29/content-loading-when-being-lazy-pays-off/”
In summary, as more and more web 2.0 technologies hit main stream, new forms of performance pitfalls may present themselves, requiring us to consider application performance for remote users even for applications that traditionally were considered “remote user friendly” such as web based applications.
How was this Analysis Performed?
This analysis was used by integrating a HP LoadRunner script modeling an Oracle Clinical business user with Shunra VE Analyzer. So as the scripted transactions executed across a virtual WAN (simulated by the Shunra VE) the beginning and end of each transaction were marked by the VE Transaction Manager. At the end of the test, the VE Analyzer had sufficient data to generate the attached reports, enabling us to pin point the root cause of the performance problem. You can learn more on how to set this up by going through Shunra Certified Performance Engineering training, either on site or at the next Shunra University. Learn more about Shunra training here: http://www.shunra.com/shunra-university-overview.php and here http://www.shunra.com/training-overview.php?keyword=services
Questions, comments, feel free to write me at amichai.lesser at Shunra dot com. Or comment on any of my posts.
BTW, we just created a new Application Performance Management Group on LinkedIn, feel free to look it up and join here http://www.linkedin.com/groupRegistration?gid=2200667
Best,
Amichai Lesser