The Performance Analysis ExerciseThe remainder of this chapter is devoted to the actual tuning exercise of the Trade3 application on our four-tier system configuration. Because the goal is to make sure that Trade3 can utilize the most precious resource, the CPU, in the three boxes (especially the application server box), the first thing we did was capture the initial performance data by performing a User Ramp Up test run.Table 22-1 shows our initial performance data. A corresponding graph for throughput and response time is shown in Figure 22-6. CPU utilization has two numbers (separated by a slash). The first number is the portion for the user and the second is the portion for the system. We used the sar tool to get this information. A heuristic that we commonly use is that the system usage should not be more than 20% of the total CPU utilization. We want to minimize system calls so that user code can do more work and thus increase the throughput. Too much time spent on the system can indicate inefficiencies or problems in the kernel. The Little's Law column is a convenient way to verify the validity of our response time and throughput on each run. The law says that the product of the throughput and response time is equal to the number of concurrent users. The numbers are rounded off to the nearest whole number. In some instances, when the number of concurrent users is large, Little's Law might not give the exact number, but it should be very close. Figure 22-6. Initial User Ramp Up Graph The Web ServerAlthough not shown in our table, we noticed that the CPU utilization on the web server tier is very lowabout 3%. There are times when the system usage is even higher than the user usage, but this is really not much of an issue because the numbers are very low to begin with. So, we are hypothesizing that the web server itself is the bottleneck and that it goes on a wait state too long or too many times (which explains the low CPU utilization). Thus, this could cause a lower rate of requests being passed on to the application server.This assumption is easy to verify. The plug-in passes the original HTTP request "as-is" to the application server. The application server has an HTTP transport layer that can accept full-fledged HTTP requests through port 9080. This means that we could skip the web server altogether and make the workload driver send the request to tethys:9080.The test was performed starting with 10 users to see if we could see some improvements. The second set of performance data is shown in Table 22-2. Unfortunately, there is no significant difference from the original results. The throughput and response time are still within the range. The same is true with the application server CPU utilization. Thus, we can eliminate the web server as the source of the problem.
The Database Back End: A Bottleneck?One way to view this problem is to investigate what causes the application server to "wait." From our knowledge of the application, there is a significant portion of the code that calls the database back end. The application uses container-managed entity beans extensively. It is possible that the application server is being blocked somehow due to some contentions in the database, physical disk, or buffers, which then cause the application server to spend too much time waiting on the database.There are two approaches to determine if this is the case. The first is to get to the database server and collect some traces, either through the database debugging facilities or with Linux tools. We recommend the database debugging tools because they immediately provide information about locks and waits on the database. Linux tools require the correlation of system-level information to the database, which might not be that straightforward. The second, and easier, approach is to look into the application server to check if many or some of its threads are waiting on the database to respond back. If so, we know that the database is the bottleneck. Thread DumpExamining the thread dump from the Java Virtual Machine is a useful way to detect where possible bottlenecks might be. The dump provides a snapshot of information on the current state of each thread in the JVM and the history of the threads' execution through stack traces. Possible states are as follows:R for running, which means that the thread is actively executing (it does not necessarily mean that it is actually running, but at least it is in the runqueue of the kernel scheduler).CW for conditional wait, which means that the Java thread has explicitly invoked the wait() method on an object and that it is waiting to be notified. Thus, if the notification does not arrive soon, the thread will wait a long time and might cause a bottleneck if other threads are depending on this thread to run before they can proceed. Otherwise, it is not that harmful.MW for monitor wait, which means that the thread is queued in a monitor of a Java object. Remember that all Java objects have their own monitor. A thread would get in this state if it enters a synchronized method or block and another thread has already gotten the "right" to execute the synchronized method or block. Unless that executing thread leaves the synchronized method or block, no other thread waiting on it can proceed. Thus, this is the highest possible cause of bottlenecks.We will perform one test run with 10 concurrent users. During that time, we will take note of the process ID of the Java process (which corresponds to the application server). During the run, we will issue the following command: You can check the process ID of the application server by looking for Java processes using the ps command. Some application servers log their process ID when they are started. Although the command says "kill," it does not terminate the process. Instead, it produces a thread dump and saves it on a file of the form javacorexxxxxxx.txt. Note that thread dumps are not part of the JVM or Java Specifications, so different JVM implementations might use different filenaming schemes. We typically issue this command in succession (three to five times) to get more than one thread dump and to monitor the progress of the threads by checking to see if they ever change their state at all.
Figure 22-7. A snippet of a thread stack trace taken from a thread dump.[View full size image]  Database I/O ActivityThe next step is to check the I/O activity at the database server. We ran the same test and used iostat on the database server: This command monitors the disk activity every 2 seconds. Following is a segment of the trace: We removed the other devices from this trace because there were no activities on them. The trace shows not a lot of I/O activity (write operations) is going on, which leads to one possibility. Because the database is actually doing some work, and none of the application server threads is blocked on the database, it is apparent that communication or transfer of data is occurring. Even when the workload is increased, the I/O activity remains the same, and yet none of the threads is blocked. This might mean that the communication between the application server and the database server is the bottleneck. The rate of data transfer over the wire is the same regardless of the load. The threads are taking a long time to receive and send data. Note that sending and receiving data is a task and that is why the threads are in the R state. Network Activity Between the Application Server and Database ServerWe suspect that the network speed between the application server and the database server is slow. The quickest way we can tell this is to use the ping command. When we ping the workload driver, which is the host atlas, the following is returned: When we ping the database server, which is the host telesto, we get the following output: Comparing the turnaround time for both outputs, it is clear that something is not right with the connection to the database server. To investigate the cause of this problem, the recommended approach is to try first from the lowest levelhardware, connections, NIC, and network driversbefore even going into tuning the TCP/IP stack. An important question to ask is "What speed and duplex are being used by the database server's network card?"To determine the speed and duplex of the database server's network card, we log on to the database server and issue the command mii-tool. Unfortunately, the mii-tool on the database server fails with the following message: A tool that is useful when the mii-tool does not work is ethtool. We issue ethtool on the database server (we know that eth0 is the interface that we are using on this server) as follows: Output from ethtool follows: The output indicates that the current setting for eth0 is 10Mbps using half-duplex. This is a very big discovery that could help improve performance. To change the setting, issue the following command: Note that you could also set the speed to "auto" instead of 100. We ran the same tests again after fixing the network card settings. Table 22-3 summarizes the new results. The improvements gained were phenomenal. We raised the maximum CPU utilization on the application server from ~59% to ~94%. Also, we added 56% more to the maximum throughput. More importantly, we now can support up to 200 users and still get a response time of less than 2 seconds.
The Java Virtual MachineThe next step is to tune the application server and the JVM. Because IBM WebSphere Application Server does not use the hotspot technology in its JVM, there is only one thing that we need to investigatethe heap size settings. The default values used are a minimum of 50MB and a maximum of 256MB. To determine if the defaults are the right values or if we have enough heap, we need to examine the garbage collection (GC) activities in the JVM. We know that garbage collection is an overhead that must be minimized to get more throughput from the application. So we ran one test with 20 concurrent users and enabled the verbosegc flag in the JVM parameters. The garbage collection outputs are emitted on the standard error, so we need to redirect them to a file.Analyzing the raw output of verbosegc from the file is very tedious and time consuming. We developed a tool called GCAnalyzer to capture this information and summarize the results. The tool graphs each garbage collection event and shows how much memory was collected and how much is still live. We used the tool with gc.output as the name of the file where the raw GC outputs were saved using the following command: This command specifies that all units be expressed in megabytes and that, in addition to the summary, a text file, gc.txt, be produced that can be imported from Microsoft Excel to create a graph. The results of the summary are shown here: Looking at these numbers, we cannot see a major problem with garbage collection. In fact, the amount of time spent on garbage collection is only 4% of the 5-minute test run. In our experience, 5% is the upper limit that can be imposed for a good garbage collection time. Performing garbage collection is very quick83ms for marking and 12ms for sweeping, and the compaction time is 1ms! We believe that there was a great improvement with the garbage collector in JVM 1.4.1, which is the version that comes with IBM WebSphere Application Server 5.1. Figure 22-8 further shows there is no problem with garbage collection. Figure 22-8. The resulting graph taken from the GCAnalyzer output. The Application ServerA layer in the software performance stack that might need tuning is the application server itself. Different application servers might have different tuning knobs to offer, but they are very similar in many respects. This is partly due to the fact that they are all J2EE-based. Thread PoolsThe first thing to address within the application server is the thread pool. Pooling threads is typical in all application servers. This is a good approach because providing threads is the responsibility of the application serverenterprise applications are not supposed to create threads. Thread pooling is a performance-enhancing mechanism for avoiding the cost of creating and destroying threads whenever they are needed.Linux kernels 2.4 and earlier are very sensitive to threads, especially where a large number of active threads are in the system. This is because these kernels are still using the original threading model and scheduler. Linux was initially designed for small desktops with one processor, so the original design of the scheduler was not very concerned with scaling issues. Linux uses a single runqueue for all threads. Thus, a number of processors that are ready to take a queued runnable process must synchronize their access to the queue. Worse, the scheduler evaluates each process in the queue to choose which of them must be scheduled next. The time complexity is a function of the number of threads in the queue. This explains why a larger thread pool may perform worse. Somehow, it is faster to make some other requests wait for an available thread than to assign a thread to it and contribute to the cost of scheduling. Linux kernel 2.6, however, introduced the O(1) scheduler and a new threading model that conforms to Native POSIX. In the O(1) scheduler, each processor has its own runqueue instead of the single-queue model.For now, we need to look at our thread pool size because we are using kernel 2.4. The default thread pool size of the web container is 50. By setting it at 20, we get the results shown in Table 22-4.
Pass-by-ReferenceThe next parameter to set is the pass-by-reference in the ORB component of the application server. The J2EE specification mandates that all EJB method calls have to pass parameters by value, not by reference. This is needed because for remote method calls, references cannot be maintained in two remote machines. Therefore, the actual values must be passed.Here is when a knowledge of the application helps. We know that all requests to Trade3 come through the HTTP transport and all EJB calls are done locally in the same JVM. It happens that Trade3 does not change the values of the parameters being passed to EJB method calls. Pass-by-reference is a performance enhancement mechanism where, instead of deeply copying objects, only the references to the objects are passed. Data Source Prepared Statements and Connection PoolThe prepared statement size and connection pool size of the data source are two knobs that you can tune if you are having database bottlenecks. However, based on our thread dump analysis, we are not experiencing a performance problem on the database side. Thus, it appears that the default value of 60 prepared statements and 50 connection pools works well for the Trade3 workload.Finally, we reran our tests with pass-by-reference enabled and set the ORB Thread Pool size down to 10. The results are shown in Table 22-5. The performance graph is also shown in Figure 22-9. Compare these results with the initial performance data, and you will see that a big difference has been made. At this point, we have used ~99% of the application server's CPU and raised the number of users to 250.
Figure 22-9. Performance graph of a tuned system running Trade3. HyperthreadingOur final attempt to improve performance was to take advantage of the hyperthreading feature of the Intel Xeon processor. Fortunately, our Linux kernel supports hyperthreading. Using the same system in its tuned state, we enabled hyperthreading through the hardware system configuration and recycled the application server box. The results of rerunning the tests are shown in Table 22-6.
|
لطفا منتظر باشید ...
