Book review: Performance Testing

Title: Performance Testing

Subtitle: An ISTQB Certified Tester Foundation Level Specialist Certification Review

Author: Keith Yorkston

Review

The book is oriented towards obtaining the ISTQB Certified Performance Tester certification and uses its syllabus as a guide to organize the contents.

The author does a good job of explaining the certification syllabus, expanding it with information from other resources, adding his own experiences, and including summary boxes to favour studying. It does not feel like other certification books, where understanding and learning are disregarded in favour of an extreme focus on passing the exam.

On the other hand, it only scratches the technical surface of systems performance, sometimes feels repetitive, and assumes you have a testing background (which seems reasonable if you're trying to get the certificate).

It's a good read, even if you're not planning on taking the exam (like me), but I'd recommend skimming over the chapters related to formal processes and to testing tools.

Key Insights

• The problem with performance is where the cut-off between “good” and “bad” performance exists.
• Performance is a component of a user’s “good experience” and forms part of an acceptable quality level.
• Quality attributes of "Performance Efficiency" as per ISO 25010 (which superseded ISO 9126): Time behavior, Resource utilisation, and Capacity.
• Performance testing is any kind of testing that focuses on the performance (responsiveness) of a system.
• Performance testing can be performed in all phases of the software development lifecycle.
• Components of a typical load test: Test controller, Load generators, Scripts (virtual users), SUT, and Monitoring.
• There are multiple causes for bad performance. For example:
  • Resource saturation (cpu, memory, disk i/o, network i/o)
  • Differences between testing and production (e.g. missing a load balancer)
  • Graceless error handling (e.g. resource pools, queues, timeouts)
  • Database design
  • Networking issues
  • Unexpected background loads
• Performance test goals are defined and achieved by the result metrics gathered during the test. Metrics are defined with measurements.
• Metrics are aggregated or derived data that provide insights and summarize the performance (e.g. peak CPU usage, p95 response time).
• Measurements are raw, quantitative data points collected directly during performance tests (e.g. individual response times, CPU usage readings).
• Metrics and measurements should be defined to be meaningful in the context of the requirements/user stories.
• Measurements should be accurate and captured with a level of precision defined by the context.
  • Accuracy refers to how close the measurement is to the real value.
  • Precision refers to how exact the measurement is (i.e. how close are the measurements to each other).
• Typical measurements and metrics are similar to those seen in systems performance analysis.
• Consider the Goal-Question-Metric (GQM) approach to select what metrics to use.
• A deductive approach conducts multiple test iterations with changes to single items.
• A diagnostic approach prefers to capture more information earlier and analyze the results data if there is a problem.
• Avoid the probe effect by considering the precision required and the measurement load on the system.
• Metrics can be categorized depending on the context where they are needed: Technical, Business, and Operational.
• Key sources of metrics: Testing tools, Monitoring tools, and Log analysis tools.
• Consider adding a tolerance to time-related requirements to avoid white or black results.
• Failing to plan is planning to fail... Have a test plan.
• Test planning involves business and technical stakeholders, reviews requirements and user stories, performs a volumetric analysis, and defines the testing environment.
• Performance test cases are created in modular form to be used as the building blocks of larger performance tests.
• Categories of test data: Master data (base data), User-defined data (test input data), Transactional data (created when executing the test).
• Different types of system architectures have different types of performance risks.
• Transactions describe a set of activities from initiation to process completion, to be measured as part of the performance test to identify issues.
• Think time is used to simulate a real user performing actions. The response time plus the think time is the elapsed time.
• Operational profiles describe a user’s interaction with the system. Multiple operational profiles are combined to form a load profile (scenario) to fulfill a test objective.
• A load profile combines operational profiles with groups of virtual users ramping up, running for a duration, and ramping down to replicate production load.
• Consider throughput and concurrency when modeling operational and load profiles.
• System throughput is the amount of transactions processed in a given time. It defines the load on the system. Concurrency represents parallel sessions consuming resources.
• Performance scripts should simulate the real system user, in the same order and at the same speed.
• Data typically analyzed after a performance test: Status of (virtual) users, Transaction response time, Transactions per second, Transaction failures, Hits (request) per second, Network throughput (amount of data received by users), HTTP responses.
• The goal of analyzing the test results is twofold:
  • Develop cause-effect relationships chains to uncover root causes of performance issues.
  • Prove the SUT has achieved the stated performance requirements.
• In performance testing, minimum outliers are better removed as they were probably gathered when the load was still low.
• Time, relative or absolute, is the easiest way to correlate metrics.