You'll never be able to perform a decent test with one mouse and one keyboard. Use the Application Center Test tool to simulate legions of users.
The focus of this hack is to use ACT from within Visual Studio. In many cases, it's preferable to use the ACT tool in its standalone version, so that will be discussed toward the end of this hack.
Like all good web developers, we want to know whether our applications perform well. Does our design and implementation hold up to the expected load levels? Are we seeing the right amount of throughput? How many concurrent users can we handle? How many requests per second can the web application deliver? Will we see memory leaks after a certain amount of time?
These are all valid questions when it comes to performance testing. Of course, there are many other questions, but most of them come when the performance metrics start to get collected and analyzed. In many cases, performance testing occurs too late in the development process. This is in large part due to time constraints; performance testing is put off until toward the end of an application life cycle, when it's too late to do anything about it.
To do some performance testing on web applications early in the development cycle, developers can use the Microsoft Application Center Test (ACT) tool. This tool comes with Visual Studio Enterprise Edition and is a valuable resource in any developer's toolbox; it can be used from within Visual Studio or as a standalone tool. Although not a replacement for something like Mercury's LoadRunner suite, it is a powerful asset because it can provide developers a clear indication of how their web application will perform in the early stages of an application cycle, thus letting them iron out any major performance issues long before the application is deployed to production.
If you do not have the Enterprise Edition of Visual Studio, you can still use a free utility called the Web Application Stress Tool. This tool, while not as feature rich as ACT, will perform a lot of the same actions. The Web Application Stress Tool is a predecessor to ACT, so they have certain similarities as well. You can download the tool from http://www.microsoft.com/downloads/details.aspx?FamilyID=e2c0585a-062a-439e-a67d-75a89aa36495&DisplayLang=en.
The ACT tool generates load by using a web browser. In all configurations, the web browser is the client. And because the client is just a web browser, it can invoke HTTP calls to any web server, local or remote. This creates a few different configurations for setting up your stress tests. A typical web application will consist of several layers that are (possibly) deployed across multiple tiers. This means that your stress test configuration will likely fall into one of the three configurations shown in Figure 10-4.
Here is an explanation of Figure 10-4:
Configuration A shows that the client, the web server, and the application logic all run on the same machine. This will likely be a developer's workstation.
Configuration B shows the client is now physically separated from the web server and application logic, both of which are still deployed together on the same box. This is the case in which the web application is running on a separate server and the developer is using her machine as the client.
Configuration C shows that the client, web server, and application logic are all running on separate machines.
Also note that products such as Virtual PC or VMWare could be used to emulate any of these configurations on a single machine. In fact, it's highly suggested you use these products to create a virtual distributed environment to match the environment where your web application will live. Doing so tends to provide more realistic expectations, basic as they may be, during development.
Before creating your tests, you must first identify your test cases. These could range from simple web page requests to more complex tests that have user workflow with several steps, tasks, and web pages. The key thing is to identify the start and end points of the test. This will allow you to determine the scope of your test and know when to stop recording your test, as you'll see later.
To use ACT for performance testing a web application, you'll first need to create an ASP.NET application in Visual Studio and run it to make sure it does what you want. You can then create an ACT Project by following these steps (assuming you chose to install ACT during the installation of Visual Studio .NET Enterprise Edition; it's selected by default):
In the Visual Studio Solution Explorer, right-click the solution, click Add, and select New Project.
In the Add New Project dialog box, under Project Types, expand Other Projects and click Application Center Test Projects.
In the Templates box, select the lone choice of ACT Project.
Type a name for the project, select the location, and click OK.
You will now see that your ACT Project has been added to your solution.
Now that you've identified your test cases and created an ACT Project, you can create and add individual tests to your ACT Project by performing the following actions:
In the Solution Explorer, right-click your ACT Project, click Add, and select Add New Item.
Under Templates, select Browser Recorded Test.
Type a meaningful name for the test and click OK.
The Browser Record dialog will appear. Click the Start button.
A new instance of Internet Explorer will open. Note that ACT uses IE even if it is not your default browser.
In the browser, type the URL where your web application is running. Finish "clicking-through" your test case and then close the browser.
In the Browser Record dialog box, notice the Request Details text box has a list of the requests that were made. Click the Stop button, then click OK.
You will now see that the test script file (e.g., TestName.vbs) has been opened up in the editor and has been added under your ACT Project in the Solution Explorer.
Now that you've created your ACT Project and added some tests, you need to set the properties of those tests. The Properties window is where you set your load levels, test duration, and test iterations. To view these settings, right-click on a test and select Properties. The Properties window appears, as shown in Figure 10-5.
By default, a brand-new test is set up to run one browser connection for five minutes (Connections = 1, RunTime = 300; RunTime is in seconds). This default configuration is actually a good "smoke test" to get started.
To gather a wide range of metrics, it's best to perform tests with varying load based on duration and iterations. A test based on duration will tell you how many requests were served in the specified time. A test based on iterations will tell you how long it took to run the test the specified number of times.
To configure a test to run 10 connections for one hour, enter 10 for Connections, 3,600 for the RunTime, and blank out Iterations.
To set up a test to run 1,000 iterations using five connections, enter 1,000 for Interations, 5 for Connections, and blank out RunTime.
The last thing to do at this point is run your test. To do this:
Right-click the test in the Solution Explorer and select Start Test.
The Output window now appears and displays statistics for the test as it is running. A sample is shown in Figure 10-6.
You now have two options: let the test run its course or stop it manually, which can be done by right-clicking on the test in the Solution Explorer and selecting Stop Test from the context menu that appears.
After your test has finished running, you can view the results by right-clicking on the test and selecting View Results. A new window will appear that resembles the one shown in Figure 10-7.
The test results window provides a quick view of the statistics for the test run. You can see items such as total run time, total number of iterations, and average requests per second. ACT autogenerates a report name using the test name and a timestamp. Every time a test is performed, ACT generates a report, even if the test was stopped before it completed. And if you rerun the same test with the same configuration over and over, ACT will still generate separate reports for each test run.
To delete a test report, right-click on a report in the test results window and select Delete.
If you do not like ACT's report naming convention, you can rename a report by right-clicking on it, selecting Rename, and typing in a new name. For instance, you might want to name a report based on the test name and its configuration, such as Report-GetOrdersDataSet-1000Iterations or Report-GetOrdersDataSet-10Connections-1Hour.
As mentioned earlier, browser recorded test scripts are autogenerated using VBScript. This has its advantages, because you can customize and/or modify the generated VBScript to better suit the specific needs of your tests. In most cases, the autogenerated script will be all you need, but it's nice to know that if you need to change it, you can very easily. You can see the generated VBScript by simply double-clicking a test in your ACT Project. A sample is shown in Figure 10-8.
The format of the script is simple and modular. For every single request in the recorded test, a SendRequest() method is created, each one being autonumbered starting at 1. It's important to understand that a SendRequest() method does not necessarily correlate to a page request. A single web page can contain several images or references to JavaScript files and CSS stylesheets. Each of these types of items is an individual request, thus ACT will generate a SendRequest() method for each one. Depending on how your web pages are designed, the generated VBScript could become quite large.
The good thing is everything can be found in a given SendRequest() method. Each SendRequest() method is a simple subroutine call that must be invoked by the main program. To see this, look at the bottom of your generated script. You will see something like this:
Sub Main( ) call SendRequest1( ) call SendRequest2( ) End Sub
The Main() subroutine is nothing more than a series of calls to the SendRequest() methods. To demonstrate how you can customize your script, let's presume your test case is a single web page that contains 10 images, and all you want to do is test the load time of that page. The ACT tool will generate 11 SendRequest() methods: SendRequest1() for the page request and SendRequest2() through SendRequest11() for each image request. Your Main() subroutine then looks like this:
Sub Main( ) call SendRequest1( ) call SendRequest2( ) call SendRequest3( ) call SendRequest4( ) call SendRequest5( ) call SendRequest6( ) call SendRequest7( ) call SendRequest8( ) call SendRequest9( ) call SendRequest10( ) call SendRequest11( ) End Sub
Simple enough, so you run your performance tests, analyze the data, and realize the page is not loading within acceptable limits. You can begin to streamline the performance by simply commenting out the calls to some of the SendRequest() methods for the images. Then rerun your tests and analyze the results. This type of easy modification of the script allows you to make quick changes to your test scripts without having to redesign your web page and record a new test.
You've seen how to create an ACT Project in Visual Studio, add and configure test scripts, run tests, and view the results. Another cool thing is that ACT creates and manages all its project files in XML format. You can see a sample of these files in Figure 10-9.
With the exception of the .vbs file, every file shown in Figure 10-9 is an XML file, even the .act and .actproj files. As you configure and make changes to your ACT Project and tests, the ACT tool is modifying these files behind the scenes as needed. A quick look at the testProperties-GetOrdersDataSet.xml file shows the following XML:
<?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE Settings> <DefaultValues > <TestProperties > <ControllerReportTimeout type="long" value="5"/> <Timeout type="long" value="120"/> <Duration type="long" value="300"/> <Warmup type="long" value="0"/> <UseIterations type="bool" value="False"/> <Iterations type="long" value="200"/> <UseRateControl type="bool" value="False"/> <TargetRPS type="long" value="40"/> <PerfCollectionInterval type="long" value="10"/> <TestType type="long" value="1"/> <GenerateUsers type="bool" value="True"/> <UsersToGenerate type="long" value="5000"/> <CollectPerfCounters type="bool" value="False"/> <CollectRuntimeHistory type="bool" value="true"/> <Locale type="long" value="1033"/> <CollectPerPageData type="bool" value="true"/> </TestProperties> <DynamicTest > <Language type="string" value="VBScript"/> <NumberOfThreads type="long" value="5"/> <FollowRedirects type="bool" value="False"/> <RedirectDepth type="long" value="15"/> </DynamicTest> <Network > <Enable type="bool" value="False"/> <NATRange type="long" value="0"/> <Distribution.9.6K type="long" value="1"/> <Distribution.14.4K type="long" value="4"/> <Distribution.28.8K type="long" value="5"/> <Distribution.56K type="long" value="20"/> <Distribution.128K type="long" value="15"/> <Distribution.512K type="long" value="15"/> <Distribution.T1 type="long" value="15"/> <Distribution.Unlimited type="long" value="15"/> <PropagationDelay type="long" value="200"/> <NATStartAddr type="string" value="10.10.1.1"/> <NATStopAddr type="string" value="10.10.1.255"/> <ClientPortStart type="long" value="1000"/> <ClientPortEnd type="long" value="5000"/> </Network> <StaticTest > <DefaultServer type="string" value="localhost"/> <DefaultMethod type="string" value="GET"/> <DefaultHTTPVer type="string" value="HTTP/1.1"/> <DefaultPort type="long" value="80"/> <SessionCount type="long" value="5"/> <UseRandomDelay type="bool" value="True"/> <MinDelay type="long" value="10"/> <MaxDelay type="long" value="1500"/> <FollowRedirects type="bool" value="True"/> <RedirectDepth type="long" value="15"/> </StaticTest> </DefaultValues>
Up to this point, you've seen how to use ACT from within Visual Studio. However, you cannot get to all of ACT's functionality from within Visual Studio itself. Other features available in the standalone tool are:
Graphical reporting
Report breakdown
Additional statistics
Defining users
Performance counter monitoring
To see all that ACT has to offer, you must use the tool on its own. This section will highlight ACT in that way.
To use ACT for performance testing, you'll first need to create and deploy your web application in one of the configurations mentioned earlier. Once you've done that, you can create an ACT Project by following these steps:
Click Start
Click File
In the New Project dialog box, type a name for the project and select a location (by default, the location is in the C:\Documents and Settings\<username>\My Documents\ACT Projects directory). Then click OK.
You can save the project any time by clicking File
You can create and add individual tests to your ACT Project by performing the following actions:
In the left pane, right-click Tests and select New Test. A wizard will start; just follow the steps.
When prompted for a test source, select Record New Test.
When prompted for browser record, click the Start Recording button. This will open a new instance of Internet Explorer (note that ACT uses IE even if it is not the default browser).
In the browser, type the URL where your web application is running. Finish "clicking-through" your test case and then close the browser.
In the wizard, click the Stop Recording button and then click Next.
Give the test a name and click Next. Then click Finish.
You'll now see the test you just recorded listed in the right pane.
To demonstrate the ACT tool, I've created a sample web application named Northwind that contains a single web page, GetOrders.aspx. This page is used to retrieve all the orders from the Northwind database on my local SQL Server. In ACT, I've created a project named NorthwindStress and recorded a test named GetOrdersDataSet, as shown in Figure 10-10.
The GetOrdersDataSet test is quite simple: open the web page, click the Get Orders button, and display the results. Figure 10-11 shows the page when you first open it, and Figure 10-12 shows the page with the results.
In fact, these two views are what was recorded when I created the GetOrdersDataSet test in ACT. The end of the test case is the results page.
Each ACT test has a set of properties where you set the load levels, test duration, test iterations, users, and performance counters to capture. To view these settings, right-click on a test and select Properties. The Properties window appears and displays the General tab, as shown in Figure 10-13.
The General tab is where you set the load level and duration. By default, a brand-new test is set up to run one browser connection for five minutes. This default configuration is actually a good "smoke test" to get started. The General tab is where you'll spend most of your time when setting the properties of a test.
The Advanced button simply displays a checkbox to generate detailed test results, which is on by default. It's best to always leave this box checked.
The next tab is the Users tab, as shown in Figure 10-14.
For tests that do not need to store the HTTP cookies created and used during the test run, users can be automatically generated by ACT during the test on an as-needed basis. Automatically generating users will help to avoid potential problems that are caused when the test's population of users is too small to match the number required to create the specified load level for the test run.
For tests in which authentication is required or you want to view or reuse the cookies that were created, users and user groups must be created and then selected in the Users tab. You will be able to set the name, domain, and password for each of these users.
The last tab of the Properties window is the Counters tab. This is where you add any performance counters you wish to capture, which could be for the test client and/or the web server. For this sample, I've added a few performance counters and changed the collection interval to two seconds, as shown in Figure 10-15.
You can add performance counters in this tab by clicking the Add button. A dialog then appears for you to make your selections, shown in Figure 10-16. If you've used the Performance Monitor tool at all, this dialog will look very familiar.
As you can see, using the ACT tool in standalone mode offers many more test settings than when using ACT from within Visual Studio.
Another advantage of using ACT outside of Visual Studio is the ability to define users and groups of users for your tests. Provided for you is a group of users named Default Users Group, containing 200 users (User0-User199) with generic passwords, as shown in Figure 10-17.
To create your own groups of users, you have three options: type them in by hand, let ACT generate them for you, or import them from a comma-delimited list. Also note that all users must belong to a group.
First, to create your own users group, click on Users in the left pane, right-click and select Add. A group with the name of New Users Group will be added. If you'd like to rename it to something more meaningful, right-click, select Rename, and type in the name.
To manually add your own users, select the users group to place them in, scroll to the bottom of that group's list of users, and enter the username and password. Figure 10-18 shows an example.
|
To let ACT generate a set of users for you, follow these steps:
Select the target users group. Click Actions
The Generate Users dialog appears. Enter the number of users, the name prefix, and password (see Figure 10-19). Then click OK.
The list of generated users will now appear in the right pane.
The only thing of note with generating users is the start number. ACT always generates users beginning with the number 1 (as you can see, the ability to change the start number is disabled). In the sample shown in the figure, the users created for me were DOMAIN\NorthwindUser1 through DOMAIN\NorthwindUser5.
The last way of creating users in ACT is by importing them from a comma-delimited file; ACT will not import users if the file is in other formats. The file should contain usernames and corresponding passwords. To import users from a file, do the following:
Select the target users group. Click Actions
Browse for your file, select it, and click Open.
The list of imported users will now appear in the right pane.
This shows the contents of a sample file (Figure 10-20 shows the list of users after being imported from the file):
DOMAIN\NorthwindUserA,myPasswordA DOMAIN\NorthwindUserB,myPasswordB DOMAIN\NorthwindUserC,myPasswordC DOMAIN\NorthwindUserD,myPasswordD DOMAIN\NorthwindUserE,myPasswordE
Running a test in ACT outside of Visual Studio is much the same as running it within Visual Studioright-click the test to run and select Start Test from the context menu that appears. Both methods display the statistics as the test is running, but using ACT in standalone mode allows you to see a nice graphic instead of a line of text. The graphic is shown in Figure 10-21.
Once your tests have completed, you can check out the results by clicking Results in the left pane and then choosing the appropriate test report under Test Runs (top pane, left). A snapshot is shown in Figure 10-22, with more statistics shown in Figure 10-23.
Recall that I added a few performance counters to monitor for the GetOrdersDataSet test. You can view the statistics for those performance counters by selecting Performance Counters under Report (top pane, right). Figure 10-24 shows the performance counter statistics for the GetOrdersDataSet sample.
And that should about do it. You've seen how you can easily create and manage performance tests for your web applications using ACT, from both inside and outside the Visual Studio IDE. The primary reason for using ACT within Visual Studio is for the tight integration with your ASP.NET Projects, but you don't get ACT in its full capacity. The main reason for using ACT outside of Visual Studio is the better user experience, plus you can take advantage of everything ACT has to offer, such as graphical results, user groups, and performance counter monitoring.
Dave Donaldson