Software Load Test Scripting: Best Practices

1.0 Plan your Business Process scripting effort
Successful performance testing relies on in-depth understanding of the application, the business requirements of the customer and careful planning of the script development phase.

1.1 Gather relevant data
Make sure you get the document that describes the business objectives of the application, maps the customer environment and provides the most appropriate solution at each stage of deployment. This document provides the foundation for the scripting effort and must be created before the script planning and development by the business analysts/Solution Architects.

• Understand the organization’s goals
• Identify the key IT-enabled business processes to achieve these goals
• Meet the stakeholders
• Identify pain points and current state of environment
• Map and analyze the tested applications
• Define the project deliverables and objectives
• Plan the test environment
• For further details refer to the Performance Testing Product Best Practices document.

1.2 Map the application
Get familiar with the application you’re about to test. Understanding the real user workflows that map together and form a business process is the most important step to creating the script design. With a thorough script design, the emulation of real user interaction is legitimate, traceable, and logical. Technical errors are more easily discovered through good test design, and logical errors do not become an issue. Additionally such step might be helpful to detect certain actions that may need special attention (e.g. a web page that often returns an error in a web based application). Interview the system administrator and the application developers (if available) and learn how the application works and its internal architecture. Focus on gathering the following data for each application:

• How the applications is being used by its regular users
• Application architecture (2/3 tier client/server, ASP/Java, etc)
• The internal process of the application and how they work
• Protocols used (HTTP, JavaScript/JSP, Tuxedo/JOLT, etc)
• Development Source: Vendor or in-house, or combination?
• Communication paths to the application (WAN/LAN, link speed, known congestion issues)
• Security issues (firewalls, username/passwords etc.)
• Data required to interact with the application.

1.3 Create the Performance Testing requirements document
The Performance Testing Requirements document is the cornerstone of any Performance Testing project. It must include precise definitions of what would be achieved during the project implementation and set the Performance Testing strategy. Within the Performance Testing Requirements document should reside Use Case scenarios that define the business processes necessary to emulate the load. Thus, this document becomes the basis for the script design, and should contain the following aspects:

• Business processes that need to be tested.
• The flow of each business process.
• Key Performance Indicators (KPI)s per business process. KPIs map what the expected results of the script execution should indicate. For example, a KPI for a script that emulates a login-logout business process would be the number of seconds that is appropriate for the transaction to be considered acceptable.

Make sure you get the stakeholders approval for the test plan before moving forward.

1.4 Plan the Performance Testing
Performance testing in a pre-production environment should be the final step before deploying applications into production. One of the significant contributors to the high cost of application testing is duplication of script development efforts. In order to reduce such inefficiencies, the planning phase should begin weeks before the actual script development and can be performed during other functional testing efforts. In this stage create the detailed Performance testing plan. The plan should cover the following aspects of performance testing:
Decide which functions should be tested. These functions represent the business processes of real users interacting with the system under test
Identify the required Performance test types/Performance test cases.
Align the application lifecycle with the available tests. Make sure that the performance tests you have chosen are suitable for the application lifecycle (development, pre-production or production).
Define in detail the performance test scenarios.

1.5 Validate the test environment

1.5.1 Align with the production environment

• It is important to run Performance tests against an environment that is as similar as possible to the production environment; otherwise this could lead to inaccurate test results.
• The performance test environment requires at least two sets of testing environments:
• Production infrastructure
• Test environment

Note: Normally most of the performance testing would be done in the test environment; however there are situations where the business requirement would be to run certain performance tests on the production environment to validate its readiness for the next business day.

1.5.2 Decide what to script
Analyze the Use Case scenarios and understand the impact they have on the system under test. Oftentimes, several Use Case scenarios have the same effect when executed, making the scripting of them redundant. Therefore, it is efficient to identify these redundancies and decide to script only a small subset of the larger set of business processes. For example, the “login” use case and the “logout” use case both impact the same middleware code and the same database tables, so it may not be necessary to script both processes and execute them.

There are three criteria for deciding what to script:
Volume: the most common business processes that generate the highest total number of interaction.
Dynamic Content: The business processes that cause the most workload for the system, especially workload that involves dynamic information to be produced.
Mission Critical: Not necessarily those business processes which fulfill the above two requirements, but are use cases that are considered the most important aspects of the application usage.

Normally you will find that the Pareto 80-20 rule applies in such situations: 20 percent of the transactions cause 80 percent of the system load. Identifying those 20 percent in the script design phase (and deciding they are the most relevant to your script) ensures that every performance test accurately mimics the load generated by users.

1.6 Scripting design document
Prepare a script design document which explains the Use Case Scenarios and fulfills the framework of the overall Performance Test Plan.

• The script design document should contain, at minimum, the following details:
• The Business Process name (and the technical script name that represents its emulation).
• The Performance Test to which the script is being executed for.
• The specific transactions within the script that are being measured for performance (the script may itself be measured, as well as individual steps within it).
• The KPIs that will contrast the results of the transactions against expectations.
• Technical issues (such as protocol considerations, network considerations, etc).
• Environmental configuration (mapped to the Run-Time Settings of the Performance test scenario).
• Data considerations.

Business process        

Script main purpose          

 Script technical name

Search Friends in Facebook             

Web transaction: Login, Search Friends, Logout.

 Facebook_SearchFriends_01

Creating dll files and using it in LR scripts

Creating DLLs have tons of advantage, especially when you want to take your scripts away from the beaten path. I won’t blabber much in this post and would like to take you people right into the topic I want to cover.

In the previous post I mentioned about Search And Replace function. To serve the purpose of an example, I have recreated the same Search And Replace function in VC++(2010) expresss edition and built a DLL file.

Below is the code that I created in VC++

#include "stdafx.h"
#include "C:\Program Files\HP\LoadRunner\include\lrun.h"

extern "C" __declspec( dllexport ) int lr_searchReplace(char* inputStr, char* outputStr, char lookupChar, char repChar);

int lr_searchReplace(char* inputStr, char* outputStr, char lookupChar, char repChar)
{

char *ptr =inputStr;
char xchar;
int len=0;
int i=0;

lr_output_message("%s",inputStr);
xchar = *ptr;//Copy initial
len=strlen(inputStr);
while (len>0)
{

len--;
xchar = *ptr;
if(xchar==' ')
{
inputStr[i]= repChar;

}

ptr++;
i++;

}

lr_save_string(inputStr,outputStr);
lr_output_message("%s",inputStr);

return 0;
}

Before attempting to build this dll project in VC++, ensure that you have included lrun50.lib in the aditional linker file settings which is in Project Options, else the project won't successfully build/compile. lrun.lib should be present inside \lib folder. Also make sure you include the lrun.h header file as shown in the above code snippet.

After the project is built, dig into the debug folder to find the dll file and then copy it inside the LoadRunner script folder and use it as shown below:

Action()
{
lr_load_dll("SearchNReplace.dll");
lr_save_string("this is a dummy text", "cProjectName");
r_searchReplace(lr_eval_string("{cProjectName}"), "convProjName", ' ', '+');
lr_output_message("%s",lr_eval_string("{convProjName}"));
return 0;
}

How to efficiently utilize the resources of Load Generators?

“How do I calibrate the total number of Load generators required to generate a load of 300 Virtual user if I have each load generators equipped with 3 GB of RAM and 3.3 GHz of CPU clock speed?”

Solution:
Memory foot print which HP as a vendor provides is not very accurate in a general sense. The impact of Memory and CPU footprint of any protocol on a Load generator also depends greatly on the length of the business flow and on the workload profile which the client expects the system to achieve. 

Example: Hypothetically, if in a SAP GUI application if you have 100 virtual users just entering employee name and number in a field and submitting it to the server and are iterating it only 2 times in a half hour period (which if is your test duration) then I would say you may not need more than 1 load generator which is sized with 3 GB of RAM and 3.3 GHz dual core Intel processor.

 Adding to the above, HP does not provide any kind of CPU footprint report. Hence I generally go about telling people to calculate the heaviness/payload of load runner process on their own. This can be done by identifying metrics of resource consumption by 1 user and theoretically determining total resource needed by multiplying the observed consumption for single virtual user with the expected number of concurrent users.

Using Windows Task manager(Process tab) while executing a single Vu will help you determine the memory consumed by a single ‘MDRV’ process which if multiplied with expected concurrent virtual user count will help you in calibrating total memory required to run a load test. Similarly to estimate the CPU resource for a certain protocol, I would recommend you to monitor and record few CPU counters of your Load Generator through ‘perfmon’ while deriving baseline result and multiply the metrics with the target Vu count which is mentioned in the workload profile.

It is ideally recommended to have as many boxes as possible to match the theoretical requirement, but as we all know, one of the many challenges a performance tester faces is lack of sufficient load generator machines. Hence for the benefit of time and to avoid further pestering from your boss to somehow resolve the issue – try utilizing your load generator boxes as efficiently as possible by:

1. Log only when error occurs.
2. Precisely calculate think-time between transactions. This can help
in exerting less load on the load generators. Strictly follow real user screen pause patterns.(note this point as
very important!).
3. Uninstall any daemon applications running on Load generators
which is not a part of AUT.
4. Avoid excessively using the “Show Vuser” option during test
execution.
5. Declaration of variables with large size should be avoided, which
directly helps in conserving overall memory. Consult the development
team if your unable to estimate the right size(This is a major concern
in Java Vuser and VB Vuser scripts).
6. Never over-iterate a script.

Explicitly Dealing with HTTP Redirect Requests in Load runner

By default, Load Runner does not show Redirect requests in the script, though it take care of redirects during replay. This becomes an issue during transaction analysis, because all redirect requests and corresonding responses are combined into one step and shows up as one transaction in the results report.

The web_url/web_custom_request/web_submit_data functions automatically detects a 302 response from the server and redirects to the location specified in the response header.

The problem with automatic redirection in LoadRunner is that you will NOT be able to figure out the amount of time it takes for each redirected request. Redirects are invisible to the user in LoadRunner. But in few test requirements you would want to know the break up of response timings between primary hit and the redirected server hits. The bigger problem comes in when you are dealing with recursive redirects.

To make an explicit redirect request you will first have to turn off automatic following of redirect instructions. This can be accomplished by setting MaxRedirectionDepth to ’0′. The key thing to remember is to identify redirected location by analyzing “Location: http://” line item in the HTTP response header of the request in question.

Example:
//In your action file
Place this in the beginning of the action file

web_set_option("MaxRedirectionDepth", "0", LAST ); //This is the key

recursiveRedirect_open();
web_url("url_which_gets_redirected","http://sampleredirect.com/",LAST);
recursiveRedirect_close();


Here is how you do it:

//place this in global.h

int HttpRetCode;
int i=0;
char depthVal[10];
char cTransactName[20000];

recursiveRedirect_open()
{
web_set_max_html_param_len("10000");

web_reg_save_param("cRedirectUrl",
"LB=Location: ",
"RB=\r\n",
"notfound=warning",
"Search=Headers",
LAST);

web_reg_save_param("cTransactionName",
"LB=https://Domain.com/",
"RB=\r\n",
"Search=Headers",
"notfound=warning",
LAST);

web_reg_save_param("httpCode",
"LB=HTTP/1.1 ",
"RB= ",
"Search=Headers",
"ORD=1",
"notfound=warning",
LAST);
}

recursiveRedirect_close()
{

HttpRetCode = atoi(lr_eval_string("{httpCode}"));
lr_output_message("xReturnCode=%d", HttpRetCode);

if(HttpRetCode == 302)//If redirect
{
i++;
web_reg_save_param("cRedirectUrl",
"LB=Location: ",
"RB=\r\n",
"Search=Headers",
"notfound=warning",
LAST);

web_reg_save_param("cTransactionName",
"LB=https://https://Domain.com/",
"RB=\r\n",
"Search=Headers",
"notfound=warning",
LAST);
web_reg_save_param("httpCode",
"LB=HTTP/1.1 ",
"RB= ",
"ORD=1",
"notfound=warning",
LAST);
sprintf(cTransactName, "Redirect_depth_%d_%s", i,lr_eval_string("{cTransactionName}"));
lr_start_transaction(cTransactName);
web_url(cTransactName, "URL={cRedirectUrl}", "Mode=HTTP", LAST);
lr_end_transaction(cTransactName, LR_AUTO);
HttpRetCode = web_get_int_property(HTTP_INFO_RETURN_CODE);
recursiveRedirect_close();

}
else
{
return;
}

}
The above code automatically generates new transactions for each unique follow redirect request’s made by the script. It keeps a track of the HTTP response code of each request and exits from recursion state when the response code is not 302.

The first challenge is to find the requests in the script which has URL redirections. If you find it hard to identify the locations where redirects are performed in the script, run the script after including web_set_option(“MaxRedirectionDepth”, “0″, LAST ); with standard log enabled. After the script completes executing, check the logs for “redirection depth exceeded” message, because whenever a web server prompts for a redirection of the resource, it sends back a 302 HTTP code with the new location, but since MaxRedirectionDepth is set to 0 in LoadRunner it will not explicitly hit the new URL. This is where the above script comes into play. Now, double click on that line, and Vugen will highlight the step where the redirection was attempted. Now place the recursiveRedirect_open() and recursiveRedirect_close() function call at the start and end of the concerned step/request and you should be good to go!

converting a C function to a floating number to string in Load Runner

I would like to call this function as ftoa

here is the code…Note: To adjust the radix/floating point position...you will have to further customize this function. 

ftoa(float floatNum, char *convFloatString) 

{

 char new[10];

 float number,dTemp,temp_val;

 int base, floatVal, radxFlag;

 char *token;

 char *temp; char cfloatVal[10],

 cBase[10]; 

 char cfloatValx[10] = "0";

 int DEBUG = 1; 

//Turn DEBUG OFF/ON by switch 0/1 

 radxFlag = 0;

 //Separate the number before and after the "." 

 number = floatNum; 

 base=number; 

 dTemp = number-base; if(DEBUG == 1) 

 { 

 lr_output_message("Base Value = %f\n", number);

 } 

 sprintf(cBase, "%d", base);

 if(DEBUG == 1) 

 { 

 lr_output_message("Floating Value = %.2f\n", dTemp);

 }

 if(dTemp == 0) //If number is a whole number then return!

 {

 lr_save_string(cBase, convFloatString); 

 return 0; 

 }

 sprintf(cfloatVal, "%.2f", dTemp);

 //Place the decimal point to suit your requirement. Default is 2

 temp = (char *)strtok(cfloatVal, "0."); 

 temp_val = atoi(temp);

 if((dTemp - 0.1) < 0) 

 radxFlag=1; 

 else

 radxFlag=0;

 if(temp_val == 0)

//If decimal values equals to 0 then return!

 { 

 strcat(cfloatVal, ".00");

 //increase the number of zero to suit your requirement. lr_save_string(cfloatVal, convFloatString); return;

 }

 if (radxFlag ==1) 

 { 

 strcat(cfloatValx,temp); strcpy(temp,cfloatValx); 

 } 

 if(DEBUG == 1) 

 {

 lr_output_message("Final decimal value = %s\n", temp); 

 } 

 if(strlen(temp) == 1 && radxFlag == 0) 

 { strcat(temp,cfloatValx); //strcpy(temp,cfloatValx); if(DEBUG == 1) 

 {

 lr_output_message("Appending a 0 %s", temp); 

 }

 } strcat(cBase, "."); strcat(cBase, temp); 

 if(DEBUG == 1) 

 {

 lr_output_message("Final decimal value = %s\n", cfloatVal); 

 }

 if(DEBUG == 1) 

 {

 lr_output_message("Final coverted floating number = %s", cBase);

 } 

 lr_save_string(cBase, convFloatString); }

Sample usage:
float floatNum; 

 floatNum = 34.102; 

 ftoa(floatNum, "convFloatStr");

 lr_output_message("Converted String = %s", lr_eval_string("{convFloatStr}"));

 return 0;

RPT Troubleshooting help

Problem:-> Test run aborted

Execution failure. No status received from location computername in interval seconds.
Explanation
The workbench has lost communication with one or more agent computers.
System action
The test run stops.
Solution :->
For each agent computer:
Check that the playback Java process is running. If it is still running, the problem might be on the workbench computer where RPT is installed. Stop the Java process and all related typeperf and vmstat processes.
Examine the problem determination log for error messages or exception messages.
If the playback Java process is not running, search for javacore.* files. The contents of these files might help you determine the cause of the problem.
Run tests again and monitor the memory size of the playback Java process. If the playback Java process is consistently running at its maximum heap size, the process might not have enough memory.
If the problem is not on an agent computer, check that the workbench computer has sufficient memory. To increase the available memory, either increase the workbench heap size or reduce the level and amount of execution history.

How to increase memory allocation for a localhost location

It is possible to see an error like: "Test Run aborted due to error"

The Driver: has become unresponsive, possibly due to an out-of-memory condition.
At last notification this driver was using 90+ percent of its allocated memory.
Please refer to the "Increasing memory allocation" Help topic for more information on how to increase memory allocation.

during playback of a Rational(R) Performance Tester (RPT) Test with many virtual users running on the test controller machine.

Cause
There are actually two heap sizes that RPT utilizes. The first one is the workbench heap, which is related to space available for controlling and monitoring the playback and reviewing results.
The second is the playback heap used by the virtual testers. This is the heap size you need to change to avoid out-of-memory conditions. When you're running the virtual testers on the same system as the workbench, there are several steps necessary to change this heap.

Solution
First you will need to create an "Agent" on the local host:

1) Decide upon a place to store the Deployment Location (either the project's root directory or a folder in the project under Test Navigator)
2) Right click at the folder from step 1 and select New and Other, then browse to Test Elements -> Location.
3) Name the Location Local and press Next and then Finish.
4) In the General Information section of the Local Location, make sure that 'Host name' is "localhost"

Problem: "Test run aborted due to error" 

Virtual users have exited prior to stage completion.
Explanation
In a schedule with multiple stages, at the end of a stage that is configured with n virtual users, there were fewer than n virtual users running. The schedule did not assign enough work to the virtual users to keep them active for the duration of the stage, or the virtual users stopped because of an a different error. The workload problem can occur if you use multiple stages but do not enclose the workload in an infinite loop.
System action
The test run stops.

Solution:
On the workbench computer, examine the workload for each user group in the schedule. Check that the entire workload is inside an infinite loop, so that the virtual users always remain active regardless of the stage duration.
If the tests already use infinite loops, then the agent computers might have encountered errors while running tests. To determine why virtual users stopped before the stage completed:
Examine the test log for error or exception messages.
Increase the test log level and decrease the number of users. Try running the schedule again. Examine the test log for error or exception messages.
Simplify the workload by running one user in one stage. Check to see whether the single user takes the expected amount of time to complete all actions. If not, examine the test log and problem determination log for error or exception messages.

Problem:" No local agent controller"

Local computer is not running the Agent Controller.
Explanation
The agent controller coordinates schedule playback between the workbench and agent computers. The agent controller is not running on the local computer.
System action
No tests run.

Solution :
Make sure that the agent controller is installed and running on the workbench computer. On Windows computers, the agent controller is a process called ACWinService. On Linux computers, the agent controller is a process called RAServer.
To start the agent controller, do one of the following steps:

1. At a command prompt, change to the directory AgentController\bin in the product installation directory.

Start->run then type cmd

>cd “C:\Program Files\IBM\SDP70Shared\AgentController\bin”

Then Typenet start "IBM Rational Agent Controller"
On Linux, change to the directory AgentController/bin in the product installation directory; then enter this command:./RAStart.sh

Problem:"Connection refused"

Connection refused on host computername
Explanation
The agent controller can be configured to restrict playback to the local computer only. If the agent controller on an agent computer is configured for local playback only, it will not accept connections from other agent controllers.
System action
The test run stops.

Solution :

Make sure that the agent controller is configured correctly.
To configure the agent controller on Windows:
2. At a command prompt, change to the directory AgentController\bin in the product installation directory.

Start->run then type cmd

>cd “C:\Program Files\IBM\SDP70Shared\AgentController\bin”

3. Type net stop "IBM Rational Agent Controller" to stop the agent controller.

4. Type SetConfig.bat to run the configuration utility.

5. Type net start "IBM Rational Agent Controller" to start the agent controller.
To configure the agent controller on Linux:
1. At a command prompt, change to the directory AgentController/bin in the product installation directory.
2. Type ./RAStop.sh to stop the agent controller.
3. Type ./SetConfig.sh to run the configuration utility.
4. Accept all of the default values except for network access mode. Set network access mode to ALL.
5. Type ./RAStart.sh to start the agent controller.

Problem:-> Unable to connect to license server
Solution :->
1) Check if "FLEXlm License Manager" service is running in Services [Control Panel>Administrator>Services: WinXP/Win2000].
Or
Start->run then type services.msc press

Restart this service "FLEXlm License Manager"

2) Launch Rational License Administrator and check if all the floating licenses are served properly [Menu Item: Setting >Client Server Config >check if Use License Server Option is checked. Check if correct server name is entered and then click on show licenses]

3) Check if Installation manager is configured for the same server.

Once above steps are verified, launch RPT go to help>license Status and check the expiration date [Should be same as seen in license server] and then run you're schedule.

Problem:-> Start recording aborted, The Agent Controller on localhost: 10002 is unavailable

Symptom

When you start a recording in RPT, the following error message can occur:

Start recording aborted.

Reason:
IWAT3034E The Agent Controller on localhost:10002 is unavailable.

IWAT3035E Confirm the Agent Controller is still running, and verify the location in Window > Preferences > Test.

Solution :->

This problem typically indicates that the installation of the Remote Agent Controller (RAC) is not correct. If it is and if the IBM_Agent_Controller directory is present, RAC does not start because of other issues. The following suggestions could help to overcome the problem:
Check the IBM Rational Agent Controller service.
In the Tab Log on, it should read:
Local System Account
Allow service to interact with desktop

Check RPT Windows > Preferences > Test.
The Agent Controller local port should be 10002.

Check the serviceconfig.xml file
Path :-> C:\Program Files\IBM\SDP70Shared\AgentController\config

Open the serviceconfig.xml and verify the followings
Make sure that the Agent Controller uses port 10002
One of these two lines must exist:


or alternatively

Check the C:\Windows\System32\drivers\etc\host file.
Make sure it contains the exact line:127.0.0.1 localhost
Bring up RPT

Windows > Preferences > Profile and Logging > Hosts.
Select the localhost: 10002 selections in the Default Hosts list.
Click the Test Connection button.

Open a command prompt,

Run the command
netstat -a -o
Look for a line like:
TCP hostname:10002 hostname:0 LISTENING 3528.
Hostname is the actual name of the computer. The right-most value in this line (3528 above) should match the process ID (PID) for the RAService.exe process.
From a command prompt, run the command
telnet localhost 10002
Note any error message that appears.

Make sure, that no firewall or other security application prevents it to start.

Common paths in Loadrunner

C:\Program Files\HP\LoadRunner\bin\certs
you can make changes in index file for Certificate entry

C:\Program Files\HP\LoadRunner\config

open wlrun7 and you will see all default setting of loadrunner.

Like default script path, result path, timeout, etc...

AgentMaxThreadsPerDriver :It is where you can specify number of threads per process(mdrv).

HostConnectTO,HostDisconnectTO etc.

you can make changes and restart Loadrunner.You can see recent scripts on the path
C:\Program Files\HP\LoadRunner\config

For Error file
C:\..Script-Name\logfile

Execution detail
C:\..Script-Name\output.txt

C:\..Script-Name\data\
Recordinglog
CorrelationLog
codeGenerationLog
also..downloaded image will be saved under data

 Result Set
C:\..Script-Name\Result

 Recorded Script
C:\..saved-Script-Name\

e.g script name is dataEntry.usr. can be launched script by clicking on it.

LR Controller online Graphs Directory to modify

1. By default, the Controller online monitor shows a maximum of 20 measurements for each graph. To increase it, go to the LoadRunner\dat\online_graphs directory to modify the value ofMaxDispMeasurments= in the file controlling each type of graph:

Description------------------------------------------------------------File Name
All---------------------------------------------------------------------- generalsettings.ini
System Resource Graphs -------------------------------------------online_resource_graphs.rmd
Runtime Graphs ------------------------------------------------------online_runtime_graphs.def
Transaction Graphs--------------------------------------------------- online_transaction_graphs.def
Web Resource Graphs----------------------------------------------- online_web_graphs.def
Streaming Media ------------------------------------------------------online_web_graphs_mms.def

2. Default counters for the System Resource, Microsoft IIS, Microsoft ASP, or SQL Server monitors are defined in the res_mon.dft file within the LoadRunner/dat folder. Its values can be pasted from the [MonItemPlus] section within scenario .lrs files.

Performance goal be in terms of concurrent users?

We often come across of  a question of how many users would require for performance testing.

1. First Approach:

Assume that we have 5000 average visitors per day and working window is 12 hours
So calculate like this ....

Average visits per hour = (5000 average visitors/day) / 12 hours = 417

Average Concurrent Users = Visits per hour / (60 min/hour / average visit.

2. Second Approach

This approach is going to tell us how many users we have on average.

U = V / (60/D)

Where:
U is the number of load test virtual users (that's what we are trying to figure out)
V is the average number of visitors per hour
D is the average duration of a visitor
60 is the number of minutes in an hour

Also , we can use

Users = (Pages views per hours * Avg. duration of script execution)/60 * Total No. of pages in the script

Web Services Performance Testing

Web Services Performance Testing using SOAPUI

Recently i had a chance to recommend a performance testing tool to carryout performance testing for web services system. I evaluated a series of open source and licensed tools and finally concluded SOAPUI would be meet the requirement for our system.

soapUI provides extensive load testing functionality allowing you to do the following:

Functional LoadTesting : validate functionality under load using standard TestCase methods.
Behavioral LoadTesting : analyze performance behaviour under varying load with different load strategies.
Performance LoadTesting : find maximal performance available using thread strategies and Command Line LoadTest execution.

Requirements Driven LoadTesting : define performance requirements and continuously validate using Load Test assertions.

SOAPUI helps in the following.

> Validate a Web Services performance under different Load scenarios.
> Maintain Functional validations to see that they don't "break" under load.
> Run several load tests simultaneously to see how they affect each other.

Its a freeware tool which best suits web services load testing. Its commercial version is called SOAPUI Pro includes productivity enahancements.

In Overall SOAPUI meets most the requirements for a Web services performance testing and it is a apt tool to meet most of the web services.

Web Log Analysis

A typical issue which i faced during web log analysis...

While conducting Performance testing for applications which are falling into the category of Post-Production case, the most important activity that needs to be done is web log analysis. The web server logs are the gold mines which hold all the information about the server during the load condition.

During the web log analysis, look for the peak hour traffic time and identify what is the server load in terms of hits/sec and in terms of users. Based on this statistics, derive performance test goals by considering the business forecasts.

I would like to share a typical issue which I faced while doing web log analysis for a typical post production category of an application. The application under test uses a shared web-app server (IIS) in which there was 2 other applications deployed. Now, the problem is the web server log files have the traces of all 3 applications together as these files are common for a server and not specific for each of the applications running on the server.

The web log analyzer tool which I was using doesn’t have any option to set filters. In that case, I tried using a web log analyzer tool (123LogAnalyzer tool) which provides a filter option to filter out the traces pertaining to a specific application and export the filtered log file into a text file. For example, if the application under test has a context root as ‘methodologies’ (http://132.23.34.21/methodologies/......), then try setting up this URL in the filter and create the text files for your application of interest.

Now you could use the newly generated text file to feed it into any of the log analyzer tool which you are confident enough and proceed with your performance goal setting. My kind advice is that before choosing the log analysis tool of your choice, analyze the accuracy of log analysis thoroughly.

A Comprehensive Report on Web Log Analysis

In today’s dynamic world, the IT sector of all the businesses want to use more sophisticated techniques to support their application scalability. All the web site owners are interested to know the usage pattern of their site in order to make any business decisions. The necessity to know the visitor profile is very high in any web site as most of time the visitors are anonymous & unpredictable for any open systems (which uses internet).

The Web analytics activity deals with measuring the end user behaviors on a web site. Of the two major web analytics techniques (Web server log analysis & Page Tagging) available, this paper deals with the Web Server Log Analysis concepts. This paper provides an overview of various log analyzer tools available in the market & their comparative ratings.

The Web log analysis is used by various types of users with different backgrounds. The different sets of users are interested to know about specific data from the web server log. This has lead to the development of lot of log analyzer tools with specific extra features, though the basic features are available in most of the tools.

Is the system Response time directly / inversely proportional to the server utilization?(Response Time Vs Queue Length Vs Server Utilization)

 Will there be queues(for CPU/Memory/Disk) if Utilization is less than 100% 

Actually speaking, there is no correlation between system response time and the server utilization. The Response time increase might happen even when the server is less utililised. 

Most of us dont understand this relationship. Increase in response time (during high load) is caused only because of queueing of the requests. The user request arrival pattern contributes to the high response time. Due to the adhoc user request arrival pattern, long queues are formed in various service centers which leads to high response time.

Response Time - What are the various components of Response time?

Waiting time (Queuing time) + Processing time = Response Time.

For a server, the processing time is always the same irrespective of load. For example, the server might take 2 seconds of processing during 1 user load and during 1000 users load for a transaction. But the waiting time would be high during 1000 users load, which drastically increases the overall response time. So, response time increases, if the waiting time increases which is caused by long queue.

Also, dont think that there will be queues only if Utilization is more than 100%. There can be queues formed even when the Utilization is less. So, everything boils down to knowing the user arrival pattern of the system. This is going to determine, when the system is going to be loaded and break.

Hence, knowing the user arrival pattern of a system is very important rather than setting mere goals to perform load/stress test. Analysing the user arrival pattern of an application would help in setting realistic goals which becomes more of organization logistics issue and most organizations bypass and pay for it later.

WCF Services Performance Testing

Most of the market available performance testing tools available for web services performance testing does not provide support to WCF (Windows Communication Foundation) web services performance testing.

The following are the tools which supports WCF web servcies performance testing.

•  WCFStorm
• WCF Load Test
•  SOAtest

SOAPUI doesnt support WCF web services performance testing. Out of above 3 tools, we have finlized on the SOAtest 5.5 tool to carryout WCF web services performance testing.

Learn about Think Time in Load runner

I got a chance to interact with lot of performance testers in various organizations , not only in India, but also in abroad & found that most of us are interested in spending time for setting realistic performance test goals & identifying the frequently used business scenarios (business flows) , but very few analyze the think time of various business flows.

It becomes very important that the average think time of the end users of the application needs to be analyzed rather than estimating specific time limit by counting the total fields to be filled by the end user. For the same application(lets take Yahoo new account creation), a new user might take 4-5 minutes or more to fill the required details to create a new account. But at the same time, a frequent old user, might take 1-2 minutes to create a new account. These think times (4-5 minutes for new user & 1-2 minutes for old user) might impact the server load.

Lets assume 100 new users are accessing the yahoo application for the time interval 13.00 to 14.00 hours & 100 old users are accessing the application for the time interval 14.00 to 15.00 hours continuously. Though there are 100 users (100 active sessions created) load on the server during both 13.00 to 14.00 hours and during 14.00 to 15.00 hours, the actual server load(requests) is going to be different during these 2 hours.

When you have a close look at the server load in requests/sec (Hits/sec) , you will find that the server load during 13.00 to 14.00 hours might be lesser than 14.00 to 15.00 hours, because of high think time used by the new users. If you look at the quantified numbers, you will be shocked to know that during the first hour, the average load could be apprx 12 hits/sec whereas during the second hour, the server load could be around 25 hits/sec.

Having performance test goals in mere user load alone doesnt make the test more realistic. Detailed analysis on the end user's think time makes it more realistic.

Fundamental laws of Performance Testing?

 Little's Law:

Little's law is quite simple and intuitively appealing.

The law states that the average number of customers in a system (over some time interval), N, is equal to their average arrival rate, X, multiplied by their average time in the system, R.

N = X . R (or) for easy remembrance use L = A . W

This law is very important to check whether the load testing tool is not a bottleneck.
For Example, in a shop , if there are always 2 customers available at the counter queue , wherein the customers are entering the shop at the rate of 4 per second , then time taken for the customers to leave from the shop can be calculated as

N = X. R
R = 2/4 = 0.5 seconds

A Simple example of how to use this law to know how many virtual users licenses are required:

Web system that has peak user load per hour = 2000 users
Expected Response time per transaction = 4 seconds
The peak page hits/sec = 80 hits/sec

For carrying out Performance tests for the above web system, we need to calculate how many number of Virtual user licenses we need to purchase.

N = X . R
N = 80 . 4 = 320

Therefore 320 virtual user licenses are enough to carry out the Load Test.

Fundemental law of performaance testing:

PEA was formed by professionals who have had the opportunity to lead assignments that brought about several radical changes by bridging the gap between theory and practice in PE.

Fundamental Laws:

1. Utilization law
2. Little's law
3. Response Time law
4. Forced Flow law
5. Service Demand law

DataBase Monitoring

Useful Info about Oracle STATSPACK Analysis
From Perfmon (Windows platform) / SAR , vmstat & other commands (UNIX platforms), you can get the insight of the server resource utilization information. But in case of the database servers, it’s very important that you need to know what is happening in the database server.

Oracle (versions later than 8i) comes with a performance diagnosis tool called STATSPACK. (In case of SQL server, try using SQL Profiler tool).

Statspack is an Oracle performance diagnosis tool (perfstat) available from Oracle 8i. It has the features of Oracle’s utlbstat.sql and utlestat.sql utilities (used in previous versions of Oracle). It has about 25 tables which help to maintain the performance information of the server.

There are good urls available in the net which details the working of Statspack. Some of them include

http://www.pafumi.net/Statspack_Analysis.html
http://www.dba-oracle.com/art_statspack.htm
http://www.akadia.com/services/ora_statspack_survival_guide.html
http://jonathanlewis.wordpress.com/2006/11/29/analysing-statspack-pt1/

There are certain things which a performance tester needs to know about the database server. I feel it’s very much required that a performance tester knows what the various terminologies (atleast) though DBA’s knowledge is not possible. From my experience, I strongly feel that Statspack analysis helps in figuring out whether something is really going wrong in the database. It would help the performance tester to answer the question - Is it worth to spend some time in tuning my database to improve system performance?

At a high-level, it works like this. Before running the test, take a snapshot (by running a simple command) of the database (snapshot collects the current value of all possible database statistics and stores it in the database table. After running the test, take another snapshot of the database and now compare these 2 snapshots (by running a simple command) and get the text report. By analysing this report, you can identify how the database was performing between these 2 snapshots (i.e... during the test run, which is what we are interested in looking at).

I am sure that if we are aware of following terms, it will be easy to figure out certain major database issues. Ofcourse, Statspack analysis performed by DBA is recommended, this is not practically achieved in many projects. In such cases, being a performance tester, you can atleast visualize bottlenecks in the database without DBA’s help.

1. Parses (hard parses vs. soft parses)
2. Transactions per second
3. User calls
4. % Library Hit & % Buffer Hit
5. Top Wait Events
6. Specific Wait events related to Indexing issues/ high table scans
7. Rollbacks per transaction
8. Top 10 SQLs sorted by Query executions, Buffer gets (CPU time) and Parse Calls.

There are lot of Statspack analysis tools available which will analyse the Statspack report and provide the analysis report. But you need to atleast know the above stated information in order to understand the report better. Try using online tools like oraperf.com or statspackanalyzer.com. If you are looking at deep insights of the database server to provide recommendations to the development team, try using tools like spAnalyzer, Wise, etc. (http://www.spviewer.com/index.html , http://www.wise-oracle.com )

Code & Application server related performance issues in J2EE

In J2EE environment, there are some common Code related or Application server related problems. It include :

Code related problems :

1. Slow Methods
a. Consistently Slow Methods
b. Intermittently Slow Methods
2. Synchronization Problems
3. Memory Problems
4. Coding Practices, such as using exceptions as a means to transfer control in the applications

Application server configuration problems :
1. JDBC Connection Pool size
2. JVM Heap size
3. Thread Pool size

Bottlenecks in performance Testing

Network Bottleneck Symptoms:

The simplest way to measure effective bandwidth is to determine the rate at which your server sends and receives data. Network bandwidth availability is a function of the organization's network infrastructure. Network capacity is a function of the network cards and interfaces configured on the servers.

Network Interface: Bytes Total/sec : To determine if your network connection is creating a bottleneck, compare the Network Interface: Bytes Total/sec counter to the total bandwidth of your network adapter card. To allow headroom for spikes in traffic, you should usually be using no more than 50 percent of capacity. If this number is very close to the capacity of the connection, and processor and memory use are moderate, then the connection may well be a problem.

Web Service: Maximum Connections and Web Service: Total Connection Attempts : If you are running other services on the computer that also use the network connection, you should monitor the Web Service: Maximum Connections and Web Service: Total Connection Attempts counters to see if your Web server can use as much of the connection as it needs. Remember to compare these numbers to memory and processor usage figures so that you can be sure that the connection is the problem, not one of the other components.

To determine the throughput and current activity on a server's network cards, you can check the following counters:

· Network\Bytes Received/sec
· Network\Bytes Sent/sec
· Network\Bytes Total/sec
· Network Current Bandwidth

If the total bytes per second value is more than 50 percent of the total capacity under average load conditions, your server might have problems under peak load conditions. You might want to ensure that operations that take a lot of network bandwidth, such as network backups, are performed on a separate interface card. Keep in mind that you should compare these values in conjunction with Physical Disk\% Disk Time and Processor\% Processor Time. If the disk time and processor time values are low but the network values are very high, there might be a capacity problem. Solve the problem by optimizing the network card settings or by adding an additional network card. Remember, planning is everything—it isn't always as simply as inserting a card and plugging it into the network.

Disk Bottleneck Symptoms:

A bottleneck from a disk can significantly impact response time for applications running on your system.
Physical Disk (instance)\Disk Transfers/sec counter for each physical disk and if it goes above 25 disk I/Os per second then you've got poor response time for your disk.
By tracking Physical Disk(instance)\% Idle Time, which measures the percent time that your hard disk is idle during the measurement interval, and if you see this counter fall below 20% then you've likely got read/write requests queuing up for your disk which is unable to service these requests in a timely fashion. In this case it's time to upgrade your hardware to use faster disks or scale out your application to better handle the load.

Look for the Physical Disk (instance)\Average Disk Queue length &Physical Disk (instance)\Current Disk Queue length parameters to get more details on the queued up requests.

CPU Bottleneck Symptoms:
Symptoms for CPU bottlenecks include the following :

The Processor(_Total)\% Processor Time(measures the total utilization of your processor by all running processes) will be high. If the server typically runs at around 70% or 80% processor utilization then this is normally a good sign and means your machine is handling its load effectively and not under utilized. Average processor utilization of around 20% or 30% on the other hand suggests that your machine is under utilized and may be a good candidate for server consolidation using Virtual Server or VMWare.

Further to breakdown this %processor Time , monitor the counters -Processor(_Total)\% Privileged Time and Processor(_Total)\% User Time, which respectively show processor utilization for kernel- and user-mode processes on your machine. If kernel mode utilization is high, your machine is likely underpowered as it's too busy handling basic OS housekeeping functions to be able to effectively run other applications. And if user mode utilization is high, it may be you have your server running too many specific roles and you should either beef hardware up by adding another processor or migrate an application or role to another box.

The System\Processor Queue Length(indication of how many threads are waiting for execution) consistently greater than 2 or more for a single processor CPU is a clear indication of processor bottleneck . Also look at other counters like ASP\Requests Queued or ASP.NET\Requests Queued as well.

Symptoms for Application & Web Server Bottlenecks

Symptoms for Application server bottleneck
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Increased 'Server Time' breakup
2. One or more page components of transaction takes more time where in the DB query is having less execution time.
3. The Static files are having less response time whereas the dynamic contents (servlets,jsp,etc) takes more time.
4. Network delay is negligible.
5. Home Page gets displayed in few seconds even during the stress period(as it is fetched from the web server).
6. Hits / sec & Throughput remains less.
7. If the CPU/ Memory/Disk of the App server has any bottleneck symptoms.
8. If the HTTP / HTTPS connections established doesn’t increase proportionally with the load.
9. If the new connections established is very higher & the reused connections are very less.

Symptoms for Web server bottleneck
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Increased 'Server Time' breakup
2. One or more page components of transaction takes more time where in the DB query is having less execution time.
3. The static files are having high response time than the dynamic contents (servlets,jsp,etc).
4. Network delay is negligible.
5. Home Page takes more time for display.
6. Hits /sec in the web server is very less.
7. If the CPU/ Memory/Disk of the web server has any bottleneck symptoms.

Hardware Malfunctioning Symptoms:

System\Context Switches/sec (measures how frequently the processor has to switch from user- to kernel-mode to handle a request from a thread running in user mode.). If this counter suddenly starts increasing however, it may be an indicating of a malfunctioning device, especially if you are seeing a similar jump in the Processor(_Total)\Interrupts/sec counter on your machine.
You may also want to check Processor(_Total)\% Privileged TimeCounter and see if this counter shows a similar unexplained increase, as this may indicate problems with a device driver that is causing an additional hit on kernel mode processor utilization.
If Processor(_Total)\Interrupts/sec does not correlate well withSystem\Context Switches/sec however, your sudden jump in context switches may instead mean that your application is hitting its scalability limit on your particular machine and you may need to scale out your application (for example by clustering) or possibly redesign how it handles user mode requests. In any case, it's a good idea to monitor System\Context Switches/sec over a period of time to establish a baseline for this counter, and once you've done this then create a perfmon alert that will trigger when this counter deviates significantly from its observed mean value.

Memory Bottleneck Symptoms:

When it comes to the System memory , there are 3 things to monitor :

1.Monitor Cache (Hits/Misses),
2.Monitor Memory (Memory Available/sec, Process/Working Set),
3.Monitor Paging (Pages Read/Sec, Pages Input/Sec, Page Faults/Sec, % Disk Processing)

Memory\Available Bytes, if this counter is greater than 10% of the actual RAM in your machine then you probably have more than enough RAM and don't need to worry.

The Memory\Pages/sec counter indicates the number of paging operations to disk during the measuring interval, and this is the primary counter to watch for indication of possible insufficient RAM to meet your server's needs.

You can monitor Process(instance)\Working Set for each process instance to determine which process is consuming larger and larger amounts of RAM. Process(instance)\Working Set measures the size of the working set for each process, which indicates the number of allocated pages the process can address without generating a page fault. A related counter isMemory\Cache Bytes, which measures the working set for the system i.e. the number of allocated pages kernel threads can address without generating a page fault.
Finally, another corroborating indicator of insufficient RAM isMemory\Transition Faults/sec, which measures how often recently trimmed page on the standby list are re-referenced. If this counter slowly starts to rise over time then it could also indicating you're reaching a point where you no longer have enough RAM for your server to function well.

What is the difference between Simultaneous & Concurrent users ?

These are the 2 terms which is often used in Performance Testing.

Simultaneous users are the users who have a valid session in the server. Each of the users would be performing different actions like one doing login , other viewing the reports, etc.

Concurrent users are the users who have a valid session in the server & they are performing the same operation at any point of time. All the concurrent users would perform login, reports download, etc at the same point of time.

Hence for a web site, the number of simultaneous users would be always greater than the concurrent users.For example , a banking web site has the workload of about 30000 simultaneous users & 5000 concurrent users.

Knowing the correct figures of simultaneous & concurrent users is very vital in order to perform performance testing of an application.

Detection of Memory Leaks in load runner visual explanation

It needs perform testing of Web server to discover memory leaks.

The solution:
Create LoadRunner VuGen script for your application (Web server in my script)
Let's suppose, we have done

Create LoadRunner Controller scenario for your application using the VuGen scripts.

The next step is to add measurement monitors which are quantitative indicators of resources being monitored (for example, memory usage, CPU usage, handle and thread count, and so on).

For that, on "Run" tab in the LoadRunner Controller, drag "Windows Resources" item from the "Available Graphs" tree and drop it to graphs area.
See the screenshot:

Then right-click on the just added graph ("Windows Resources") and select "Add Measurements" from the pop-up menu."Windows Resources" dialog will be shown:

The following actions are easy. I will add the name of server where the application (Web server in my case) will be run. This server can be a remote one. In this case, you have to make sure that LoadRunner will have an access to get info from the remote computer. Usually, I provide administrator rights for user which executes LR scripts.

After that I select counters I want to measure during the load testing:

So, "Windows Resources" dialog looks like

I selected the following counters:

% User Time for two processes (RService & tomcat5) and the whole system (_Total)
Available bytes in the system
Handle counts
Thread counts

Working SetsAlso you may wish to add "Private Bytes" counter. Description for each counter is available on "Windows Resources" dialog. Obviously that the set of counters depends on the requirements and software to be tested.

Pay attention... If you select the "Windows Resources" graph, the list of counters will be shown in the lower part of the LoadRunner Controller:

We have to run the scenario against a number of concurrent users (I user 30 users) and observe the graph displaying info on memory, CPU, handles and so on...
I will show the result graph after 4 hrs of execution:

Pay attention to the yellow dotted line (memory usage of "RService" process) and the brown dotted line (habdle count counsumed by "RService" process) trends. They grow constantly during ~ 4.5 hours. I marked them out with red lines.
So, this graph provides proofs that the application, I tested, contains memory leaks in the "RService" process.

Several words about useful tricks and hints, connected to measurements ans graphs:

It is possible to export graphs to HTML from the LR. For that, right-click on the graph and select "Export to HTML...". It will generate HTML report containing the graph and a legenda for it.

I recommend to set the range of time to be shown on the graph, to "Relative to scenario start". To perform this, right-click on the graph and select "Configure...". That allow to display statistics for the whole execution, but the latest N minutes.

Sometimes, it is convenient to show more detailed graph. For that, double click on the graph - one big graph will be shown. To revert, double-click again. Also, you can change the number of graphs displayed - open menu View/View Graphs and set the required number.


So, now Dev team has to fix the problem in the "RService" process.. The last graph shows the results of load testing perfomed on the fixed application:



No any memory leaks found during 10 hrs.

Vugen Not Recording — Internet Explorer (iexplore.exe) “Hangs” when launched during recording

Vugen Not Recording — Internet Explorer (iexplore.exe) "Hangs" when launched during recording
When recording WEB (HTTP/HTML) script Vugen launches iexplore.exe but Internet Explorer window is not displayed.

This may occur due to COM object permissions.
Change the following Recording Options setting:
"Recording Options" -> General ->"Script": UN-CHECK "Track processes created as COM local servers"
Then record the WEB (HTTP/HTML) script.

Can QTP and LR be installed on the same machine

Can QTP and LR be installed on the same machine

QTP and LR share some files. Therefore, if QTP is installed and then LR is installed, this will cause problems. QTP and LR can be installed on the same machine, but special attention has to be given to the install squence.

LR needs to be installed first, then QTP can be installed.

If you plan to execute QTP scripts, QTP should be installed on Load Generator computer.

How Unique with Each iteration and Abort Vusers works in Parameter file

When a File parameter is unique, LoadRunner will divide the data evenly into blocks and assign each block to a virtual user when it starts running. Each vuser will use the data in their assigned block sequentially.

This means that the used values aren’t all at the start of the file; they appear in chunks, spread evenly throughout the file (unless your scenario has a single virtual user).

Size of block depends on the "Automatically allocate block size" and "Allocate ____values for each Vusers."If we select first "Automatically allocate block size" it allocates data in 1 block= no of iterations.

Load Runner with VMWare

HP recommendation about LoadRunner on VMWare

LoadRunner 11 supports VMware ESX 4.0 on Windows XP, 2003 and Vista.
It is not recommended to run Load Generator for load testing purposes on Virtual Environment, however it shouldn’t be a problem.

Don’t need to install any patches to be able to run it on VM but always recommend installing the latest patches (currently patch 4).

How to download LoadRunner tool through HP site

This article is about to download LoadRunner11.0 from HP site when you have proper SAID

To download the installation you go to the following link:
http://support.openview.hp.com/ (make sure u are logged in)

1. After you load the link go to >>Downloads
2. >>Software updates window button
3. My Updates
4. To make sure you see the same options like me we will go the other way around and enter your SAID directly
5. Fill the checkbox where you accept the terms and conditions
6. Click View Available Products >>
7. Under Performance Center you can find HP LoadRunner 11.00 Eng SW E-Mediacheck the box and click the Get Software button on the bottom of the page.
8. Now you should see it on top of the table so under Deliverables click again Get Software.
9. Now you can see in the second window one .zip and one .z01 of LoadRunner.
10. Pick one of them and Download it (Directly or via HP Download Manager)

Best practices in performance testing

Here are some of the most common things projects do that make performance testing difficult or less productive…

Stick to the requirements, even when they don’t make sense
You should know by now that projects aren’t very good at defining their non-functional requirements. This means that it is necessary to use a certain amount of common sense when applying them to the performance test results. As an example, imagine that a response time requirement specified that average response times for login should be less than 5 seconds. During performance testing, it is found that 90% of login transactions take 1 second, but 10% take 40 seconds. On average, the response time for the login transaction is 4.9 seconds, so someone who interprets requirements very strictly would consider the response time acceptable, but anyone with good critical thinking skills would see that something is wrong and think to get the intent behind the requirement clarified.

Use the wrong people to do your Performance Testing
A very common mistake is to assume that someone who does functional test automation is necessarily suited to performance testing because they know how to use another tool that is made by the same company. Another mistake is to assume that just because the project has purchased the very best tool money can buy, and it is very easy to use, this will compensate for testers who don’t know anything about performance testing (“fools with tools”). Performance testing is a highly technical activity, and is not a suitable job for anyone who cannot write code, and who does not understand how the system under test fits together.

Don’t provide enough technical support to investigate and fix problems
A good way to ensure that it takes a long time to fix defects is to fail to provide someone who is capable of fixing the problem, or to provide someone who is too busy to work on the problem. Load and performance-related defects are difficult problems, which are not suitable to assign to a junior developer. It is best to make code-related performance problems the responsibility of a single senior developer, so that they have a chance to focus, and are not distracted by all the other (much easier to fix) problems in the buglist.

Don’t let performance testers do any investigation themselves
Having a rigidly enforced line between testers (who find problems), and a technical team (who determine the root cause of a problem, and fix it) doesn’t work so well with performance testing. Performance testers find problems that are impossible for other teams to reproduce themselves (and it’s pretty hard to fix a problem you can’t reproduce). This means that performance testers and technical teams need to work together to determine the root cause of problems. Performance testers can do a lot of this by themselves if they have access to the right information. This means setting up infrastructure monitoring, and providing logons to servers in the test environment and access to any application logs.

Wishful extrapolation
Imagine that the test system is two 2-CPU servers, and performance testing shows that it can handle 250 orders per hour. The Production system is two 8-CPU servers, so it should be able to handle 1000 orders per hour, right? Well, not necessarily; this assumes that the system scales linearly, and that CPU is the only bottleneck. Both are bad assumptions. It is best to test in a Production-like environment, or to have a very solid (experimentally proven) knowledge of how your system scales.


Hide problems
One of the main reasons for software testing is so that the Business stakeholders can make an informed decision about whether a new system is ready to “go live”. Often performance testers are put under pressure to downplay the severity or likelihood of any problems in their Test Summary Report. This is usually due to a conflict of interest; perhaps performance testing is the responsibility of the vendor (who is keen to hit a payment milestone), or the maybe the project manager is rewarded for hitting a go-live date more than for deploying a stable system. In either case, it is a bad outcome for the Business.

Following are some of these performance stats and business losses examples.

• In September 2010, Virgin Blue's airline's check-in and online booking systems went down. Virgin Blue suffered a hardware failure, on September 26, and subsequent outage of the airline's internet booking, reservations, check-in and boarding systems. The outage severely interrupted the Virgin Blue business for a period of 11 days, affecting around 50,000 passengers and 400 flights, and was restored to normal on October 6. .There is a loss of $20million.
• Average user clicks away after 8 seconds of delay
• $45 billion business revenue loss due to poor web applications performance
• In November 2009, a computerized system used by US based airlines to maintain flight plans failed for several hours causing havoc amongst all major airports. This caused huge delays to flight schedules causing inconvenience for thousands of frustrated passengers. Identified as a ‘serious efficiency problem’ by the Federal Aviation Authority, this was one of the biggest system failures in US Aviation History!
• Aberdeen found that inadequate performance could impact revenue by up to 9%
• Business performance begins to suffer at 5.1 seconds of delay in response times of web applications and 3.9 for critical applications and an additional second of waiting on a website significantly impact customer satisfaction and visitor conversions. Page views, conversions rate and customer satisfaction drops 11%, 7% and 16% respectively!
• A/c to Amazon: Every 100ms delay costs 1% of sales

Load Runner 11.5

These are the major changes in LR11.5

New and Improved VuGen – The whole look and feel of VuGen has been revamped here to include much more modular panes and viewing ability. While there are a handful of default views available, these are easily modified to taste. Tree view has been replaced with the new step navigator and the new solutions explorer gives access to all actions, run time and parameter panels.

1. Ajax TruClient Firefox Enhancements – Changes here include a move to Firefox 8 and support for HTML5. Functions can be created for easy reuse and event handlers are available to support asynchronous behaviors. I think the HTML5 is a significant advancement. We’ve actually seen a demo from HP of a Vugen script playing back a game Angry Birds.
2. Ajax Truclient Internet Explorer – A new addition to the Ajax TruClient family, HP is establishing TruClient as a cross browser solution for capturing heavy AJAX front went applications. Now companies who created their application with Internet Explorer support only can find bugs under load that they would not have simply because Firefox may run different code. Applications must work in IE9 standard mode to be supported.
3. Web Protocol Asynchronous Support – the Web (HTTP/HTML) as well as Flex, Silverlight and Web Services have been modified to support advanced applications that make use of asynchronous communications such as Poll, Long Poll and Push. This means applications such as chat, messaging, stock tickers, and news updates that use various asynchronous mechanisms should be easier to script against.
4. Improved Correlations – Correlations can now be found based on server responses during recording, so you don’t always have to play it back at least one to start correlation. Coupled with the new Correlation Studio interface, introduced in LR 11 which gives us regular expressions and XPATH, scripting becomes easier and faster.
5. Flex Enhancements – Improvements here include use of Web Correlation mechanisms, support for Web requests and the bundling of Adobe Flex platform jars into the application. In addition, there is now support for RTMP FMS streaming, RTMPT, RTMPS and GraniteDS.
6. Mobile Protocols– Initially introduced in FP4 of LoadRunner 11, these protocols enabe script development for http-based and native mobile applications. Traffic based analysis is used for native applications and Ajax TruClient technology is used for browser based mobile applications.
7. Data Format Extension (DFE) for Google Web Toolkit (GWT) – In addition to the already supported formats of Base64, JSON, URLEncoding, XML and Prefix-Postfix, support is now included for Google Web Toolkit (GWT). This includes: Formatting the GWT RPC data into readable and correlate-able content, addition of more data such as object field names and simple parameterization.
8. Additional important updates for support include 64-bit recording in Vugen (very important), .NET 4.0 support, WCF application support enhancements, and IPV6 support.