Volume Tests are often most appropriate to Messaging, Batch and Conversion processing type situations. In a Volume Test, there is often no such measure as Response time. Instead, there is usually a concept of Throughput.
A key to effective volume testing is the identification of the relevant capacity drivers. A capacity driver is something that directly impacts on the total processing capacity. For a messaging system, a capacity driver may well be the size of messages being processed.
Volume Testing of Messaging Systems
Most messaging systems do not interrogate the body of the messages they are processing, so varying the content of the test messages may not impact the total message throughput capacity, but significantly changing the size of the messages may have a significant effect. However, the message header may include indicators that have a very significant impact on processing efficiency. For example, a flag saying that the message need not be delivered under certain circumstances is much easier to deal with than a message with a flag saying that it must be held for delivery for as long as necessary to deliver the message, and the message must not be lost. In the former example, the message may be held in memory, but in the later example, the message must be physically written to disk multiple times (normal disk write and another write to a journal mechanism of some sort plus possible mirroring writes and remote failover system writes!)
Before conducting a meaningful test on a messaging system, the following must be known:
The capacity drivers for the messages (as discussed above).
The peak rate of messages that need to be processed, grouped by capacity driver.
The duration of peak message activity that needs to be replicated.
The required message processing rates.
A test can then be designed to measure the throughput of a messaging system as well as the internal messaging system metrics while that throughput rate is being processed. Such measures would typically include CPU utilization and disk activity.
It is important that a test be run, at peak load, for a period of time equal to or greater than the expected production duration of peak load. To run the test for less time would be like trying to test a freeway system with peak hour vehicular traffic, but limiting the test to five minutes. The traffic would be absorbed into the system easily, and you would not be able to determine a realistic forecast of the peak hour capacity of the freeway. You would intuitively know that a reasonable test of a freeway system must include entire 'morning peak' and 'evening peak' of traffic profiles, as both peaks are very different. (Morning traffic generally converges on a city, whereas evening traffic is dispersed into the suburbs.)
Volume Testing of Batch Processing Systems
Capacity drivers in batch processing systems are also critical as certain record types may require significant CPU processing, while other record types may invoke substantial database and disk activity. Some batch processes also contain substantial aggregation processing, and the mix of transactions can significantly impact the processing requirements of the aggregation phase.
In addition to the contents of any batch file, the total amount of processing effort may also depend on the size and makeup of the database that the batch process interacts with. Also, some details in the database may be used to validate batch records, so the test database must 'match' test batch files.
Before conducting a meaningful test on a batch system, the following must be known:
The capacity drivers for the batch records (as discussed above).
The mix of batch records to be processed, grouped by capacity driver.
Peak expected batch sizes (check end of month, quarter & year batch sizes).
Similarity of production database and test database.
Performance Requirements (eg. records per second)
Batch runs can be analysed and the capacity drivers can be identified, so that large batches can be generated for validation of processing within batch windows. Volume tests are also executed to ensure that the anticipated numbers of transactions are able to be processed and that they satisfy the stated performance requirements.
A key to effective volume testing is the identification of the relevant capacity drivers. A capacity driver is something that directly impacts on the total processing capacity. For a messaging system, a capacity driver may well be the size of messages being processed.
Volume Testing of Messaging Systems
Most messaging systems do not interrogate the body of the messages they are processing, so varying the content of the test messages may not impact the total message throughput capacity, but significantly changing the size of the messages may have a significant effect. However, the message header may include indicators that have a very significant impact on processing efficiency. For example, a flag saying that the message need not be delivered under certain circumstances is much easier to deal with than a message with a flag saying that it must be held for delivery for as long as necessary to deliver the message, and the message must not be lost. In the former example, the message may be held in memory, but in the later example, the message must be physically written to disk multiple times (normal disk write and another write to a journal mechanism of some sort plus possible mirroring writes and remote failover system writes!)
Before conducting a meaningful test on a messaging system, the following must be known:
The capacity drivers for the messages (as discussed above). The peak rate of messages that need to be processed, grouped by capacity driver. The duration of peak message activity that needs to be replicated. The required message processing rates.A test can then be designed to measure the throughput of a messaging system as well as the internal messaging system metrics while that throughput rate is being processed. Such measures would typically include CPU utilization and disk activity.
It is important that a test be run, at peak load, for a period of time equal to or greater than the expected production duration of peak load. To run the test for less time would be like trying to test a freeway system with peak hour vehicular traffic, but limiting the test to five minutes. The traffic would be absorbed into the system easily, and you would not be able to determine a realistic forecast of the peak hour capacity of the freeway. You would intuitively know that a reasonable test of a freeway system must include entire 'morning peak' and 'evening peak' of traffic profiles, as both peaks are very different. (Morning traffic generally converges on a city, whereas evening traffic is dispersed into the suburbs.)
Volume Testing of Batch Processing Systems
Capacity drivers in batch processing systems are also critical as certain record types may require significant CPU processing, while other record types may invoke substantial database and disk activity. Some batch processes also contain substantial aggregation processing, and the mix of transactions can significantly impact the processing requirements of the aggregation phase.
In addition to the contents of any batch file, the total amount of processing effort may also depend on the size and makeup of the database that the batch process interacts with. Also, some details in the database may be used to validate batch records, so the test database must 'match' test batch files.
Before conducting a meaningful test on a batch system, the following must be known:
The capacity drivers for the batch records (as discussed above). The mix of batch records to be processed, grouped by capacity driver. Peak expected batch sizes (check end of month, quarter & year batch sizes). Similarity of production database and test database. Performance Requirements (eg. records per second)Batch runs can be analysed and the capacity drivers can be identified, so that large batches can be generated for validation of processing within batch windows. Volume tests are also executed to ensure that the anticipated numbers of transactions are able to be processed and that they satisfy the stated performance requirements.
No comments:
Post a Comment