The most important points to consider when writing an NFR are:
1. Qualified – It should describe in words what is required. This is important, spreadsheets with boxes that have numbers in them are only useful if everyone agrees on the same assumptions – this is rarely the case as a box with 1 second in it can mean a lot of things to a lot of people.
2. Defined – For example, a response can be measured at various points, there are lots of layers that a request can pass through and the very last point in the stack is when it hits the user’s eyeball. It is important to define which layer we are working in. Typically, in load testing you want to remove the client and the network and only focus on server response times (different people have different bandwidths and different PCs so we remove those variables).
3. Quantified – We need actual numbers to work to. The best number to use is the 90th percentile as this prevent the minority skewing the majority.
4. Flexibility – NFRs are not written in stone and can be changed. But we must have a line in the sand to start with. This line can be, and usually is, moved but having it defined prevents either needless tuning or, worse, insufficient tuning.
5. Volumetrics – Now that we know how fast we want the system to respond we have to decide under what conditions we want it to do this. This is known as defining the ‘load’ to be tested to and is most often expressed in terms of business requests per second/minute. For end to end performance testing it should be complete and describe traffic from all sources. For component testing it should set the scope for what is to be tested. If the system is complicated then this data can be defined in a separate document and referenced.
An example:
“As a business owner, I want 85% of Login requests to be served within 650ms, 95% within 850ms and none over 1200ms where a response is defined as a HTTP 200 return code from the server to Apache and does not include any client side or external network (internet) processing. Requests must be made against a fully populated database that is a replica of Production and this level of performance should be maintained for 1 hour without degradation during a simulation of the peak load as defined in the ‘Project Volumetrics’ document which includes 2.6 Login requests per second via 220 concurrent threads.”
1. Qualified – It should describe in words what is required. This is important, spreadsheets with boxes that have numbers in them are only useful if everyone agrees on the same assumptions – this is rarely the case as a box with 1 second in it can mean a lot of things to a lot of people.
2. Defined – For example, a response can be measured at various points, there are lots of layers that a request can pass through and the very last point in the stack is when it hits the user’s eyeball. It is important to define which layer we are working in. Typically, in load testing you want to remove the client and the network and only focus on server response times (different people have different bandwidths and different PCs so we remove those variables).
3. Quantified – We need actual numbers to work to. The best number to use is the 90th percentile as this prevent the minority skewing the majority.
4. Flexibility – NFRs are not written in stone and can be changed. But we must have a line in the sand to start with. This line can be, and usually is, moved but having it defined prevents either needless tuning or, worse, insufficient tuning.
5. Volumetrics – Now that we know how fast we want the system to respond we have to decide under what conditions we want it to do this. This is known as defining the ‘load’ to be tested to and is most often expressed in terms of business requests per second/minute. For end to end performance testing it should be complete and describe traffic from all sources. For component testing it should set the scope for what is to be tested. If the system is complicated then this data can be defined in a separate document and referenced.
An example:
“As a business owner, I want 85% of Login requests to be served within 650ms, 95% within 850ms and none over 1200ms where a response is defined as a HTTP 200 return code from the server to Apache and does not include any client side or external network (internet) processing. Requests must be made against a fully populated database that is a replica of Production and this level of performance should be maintained for 1 hour without degradation during a simulation of the peak load as defined in the ‘Project Volumetrics’ document which includes 2.6 Login requests per second via 220 concurrent threads.”