On occasions, the Load Tester has the luxury of Load Testing in the production environment. This is rare - normally this would happen before an application goes into production for the first time. In other circumstances, the system administrators can take down the production instance of an application, and start in its place an environment that can be used for Load Tests. Again, this is rare as it normally requires Load Testing to take place during unsociable hours and it causes an outage to users or customers. There is also a risk that on bringing the production instance back up, problems will occur.
The best a Load Tester should realistically hope for is a production like disaster recovery environment that can be used for Load Testing. While there will always be differences with production, the fact that it could one day be asked to support production gives the Load Tester a high degree of confidence.
The norm for a Load Test environment is a halfway house between a small sized functional test environment and the full sized production environment. Often the Load Test environment is on a reasonably sized hardware platform which is shared with a number of other test environments. In many cases one too many test environments is squeezed onto a configuration than should really be the case. A limitation of memory means that paging can and will occur. This can cause response times to increase hugely. It can also cause CPU utilisation to increase if it is not already at 100%.
TP recommend that in this circumstance, the Load Test environment is looked at very carefully. While Load Test preparations can take place in an environment like this, execution of a Load Test would be fraught with problems. There are a number of solutions that the Load Tester can recommend to the project, including:
1. Purchase hardware and relocate the Load Test environment. This option is not realistic. The cost of an environment would likely be so high that any self respecting infrastructure manager would rule it out without a second thought. Lead times for purchase and installation of hardware is normally weeks, if not months. A project manager is more likely to cancel Load Testing than go for this option.
2. Wait till some of the other testing phases are complete, shut those test environments down thus freeing up system resource. This is a more sensible option, however, project timescales and implementation dates are often set well in advance. Delay can cause many problems increasing the cost of implementation. Again, this is not something a project manager would be likely to accept.
3. Plan for Load Test execution to take place out of hours. Functional testing normally takes place during normal working hours, i.e. from 08:00 till 18:00 though this is not always the case. Testers could be located in other time zones, extending the period which functional testing takes place. Testing often runs late, overtime is used to decrease the elapsed time for testing a code drop, again extending the test window. Even considering these exceptions, the Load Tester should always be able to find a window of opportunity when there is no functional testing taking place. This may be at the weekend or possibly weekdays between 00:00 & 04:00. These unsociable hours are not much fun, but they can provide a cost effective opportunity to Load Test on halfway decent kit without the interference of other users. Request that the system administrators maintain two configurations of the Load Test environment:
a. The first configuration would be defined to use a minimum of resources that could co-exist with other test environments during the day. This would enable the Load Tester to develop scripts, create and extract data, set up monitoring and carry out all those other preparations necessary for building a Load Test pack.
b. The 2nd configuration would be large sized and used for Load Test execution. Before Load Test execution begins, the system administrators would take down all other test environments that share hardware with the Load Test and bring up the large sized Load Test environment. This should enable a much better Load Test to be run, with a greatly increased chance of meeting Load Testing objectives and understanding the performance profile of the application
The best a Load Tester should realistically hope for is a production like disaster recovery environment that can be used for Load Testing. While there will always be differences with production, the fact that it could one day be asked to support production gives the Load Tester a high degree of confidence.
The norm for a Load Test environment is a halfway house between a small sized functional test environment and the full sized production environment. Often the Load Test environment is on a reasonably sized hardware platform which is shared with a number of other test environments. In many cases one too many test environments is squeezed onto a configuration than should really be the case. A limitation of memory means that paging can and will occur. This can cause response times to increase hugely. It can also cause CPU utilisation to increase if it is not already at 100%.
TP recommend that in this circumstance, the Load Test environment is looked at very carefully. While Load Test preparations can take place in an environment like this, execution of a Load Test would be fraught with problems. There are a number of solutions that the Load Tester can recommend to the project, including:
1. Purchase hardware and relocate the Load Test environment. This option is not realistic. The cost of an environment would likely be so high that any self respecting infrastructure manager would rule it out without a second thought. Lead times for purchase and installation of hardware is normally weeks, if not months. A project manager is more likely to cancel Load Testing than go for this option.
2. Wait till some of the other testing phases are complete, shut those test environments down thus freeing up system resource. This is a more sensible option, however, project timescales and implementation dates are often set well in advance. Delay can cause many problems increasing the cost of implementation. Again, this is not something a project manager would be likely to accept.
3. Plan for Load Test execution to take place out of hours. Functional testing normally takes place during normal working hours, i.e. from 08:00 till 18:00 though this is not always the case. Testers could be located in other time zones, extending the period which functional testing takes place. Testing often runs late, overtime is used to decrease the elapsed time for testing a code drop, again extending the test window. Even considering these exceptions, the Load Tester should always be able to find a window of opportunity when there is no functional testing taking place. This may be at the weekend or possibly weekdays between 00:00 & 04:00. These unsociable hours are not much fun, but they can provide a cost effective opportunity to Load Test on halfway decent kit without the interference of other users. Request that the system administrators maintain two configurations of the Load Test environment:
a. The first configuration would be defined to use a minimum of resources that could co-exist with other test environments during the day. This would enable the Load Tester to develop scripts, create and extract data, set up monitoring and carry out all those other preparations necessary for building a Load Test pack.
b. The 2nd configuration would be large sized and used for Load Test execution. Before Load Test execution begins, the system administrators would take down all other test environments that share hardware with the Load Test and bring up the large sized Load Test environment. This should enable a much better Load Test to be run, with a greatly increased chance of meeting Load Testing objectives and understanding the performance profile of the application
No comments:
Post a Comment