What is the difference between load and performance testing?
A short definition and overview of strategies and tools.
What is a load and performance test
A load test checks and measures the performance of a system under several workloads. This may be crucial, for example, prior to a release deployment or going-live as well as during the software development process and run (operations) of an application.
If you expect an uncommonly higher load (e.g. before an Black Friday event, university enrollment, etc.) or intensive use of the services or a website you perform at least a basic load and performance test scenario to ensure the stability, load-robustness and stable run of your application.
To figure out where weaknesses and limits are hidden in a system, it is subjected to a continously increasing load until it can no longer withstand the stress (load test type: stress test/ overload scenario). This works by simulating high payload, either in form of many concurrent users (concurrent VUs) or fast repeating actions (high throughput through small think time). This allows you to spot any errors that occur only under high or higher system load. The load test focuses on measuring the time and consumption behaviour.
The load test or performance test deals with this. It measures how the response and processing speed behaves at different loads on the system.
In the SLC/ ALC (software/ application life cycle) load and performance tests are usually found in the software development phase in the area of quality assurance activities (software test), in integration or system test. As a rule, a load test always involves a kind of test automation, since normally one would not want to apply the load with tens of thousands of manual testers, but rather leaves this to virtual users (VUs) in form of test scripts which run on load generators via dedicated load test software.
Performance tests are generally sorted into non-functional tests inbetween software testing.
You may have encountered load tests under abbreviations like LuP or LPT (both: Load & Performance Test).
Goals of Performance Testing
Time is money
This is more true than ever in today’s business world and more so on the World Wide Web than in real world. That is why it is important to ensure that systems work smoothly, regardless of their duration and usage. Problem-free, that means fast and reliable in this context, because long waiting times cost a company a lot of money – internally and externally.
A slow ecommerce site (as well as e-banking and many other web applications and sites) means unhappy clients. Big players cannot risk clients leaving their site. A slow website response has a significant negative impact on sales. So that a smooth work with a system is guaranteed, it is necessary to limit waiting times (searching, checkout process, browsing, …) for users to a minimum. That is where performance test comes in – checking speed with which a system works. If the result is not satisfactory, it is up to the company to make performance improvements.
The tested system should run optimally and efficiently with all its components. This is exactly what a load and performance test should prove or find out.
Types of load tests and load profiles/ workloads
To find out which load profile or workload should be used, we first define what goal is being sought. Depending on the requirement profile, the test can check the system for different scenarios.
Terms are used differently depending on the company or literature. At the core, however, often the same thing is meant.
So there is the stress test, in this, the load, starting from a small starting value, steadily increased until the system has errors and the load is no longer able to withstand. While the load increases linearly, the performance usually flattens out at some point. We are looking for the moment when the response power drops exponentially as the workload increases. At this point, the maximum capacity of the system is reached.
The continuous load test, on the other hand, checks stability, resource requirements and response time by running for a longer period of time (12 hours to several days). Memory leaks (memory-leaking memory leaks) are also easily identifiable here.
The scalability test is used to find out how intensive a system can be used and whether it „scales“ well with more hardware. Say, if with more hardware also more work load with still good response times is processed.
The performance test is often used as a generic term for test types that deal with time behavior. According to ISTQB the correct term would be performance test or efficiency test, but this is rarely used in practice. According to ISTQB, the load test is a kind of performance test.
In the real world of software testing, independent of curriculum, „load and performance testing“ is by far the most widely used generic term for performance test types. In companies often quite pragmatic with the above mentioned abbreviations „LPT“ or „LuP“.
According to ISO/ IEC 25010:2011, efficiency is also one of the six superior quality features, including the performance test.
In addition, a load test can be set up at various interfaces, via GUI (eg. TruClient), via API (HTTP, web services, REST) or, for example, via SQL directly in the database.
Practical tips and checklist for successful performance tests
Accurate and careful preparation is just as important as professional execution with experienced experts
Performance test checklist
Before you can do a load and performance test, you should do the following.
- What do I want to find out by using the test?
- How fast should my system respond and is it dependent on different conditions?
- Are there SLAs (Service Level Agreements) or NFR / NFA (non-functional-requirements)?
- Which load types should I choose to test my system and what are the goals of the different test types?
- What results do I expect from this software test?
- Is it all about profit maximization, server resource savings or workload and customer satisfaction?
Set Timeline and Appointments
- When – in which step of the system development does it make sense to take the test?
- Should the test be carried out once or continuously – if applicable, included in a CI (Continuous Integration) procedure?
Determine load test tool, personnel, effort and techniques
- Which resources am I willing to invest?
- Which load or throughput is needed?
- Which interface would I like to use for the test?
- Which test data do I need or how should my basic data of the database look like and where do they come from?
- Which technologies and architectures are used?
- Flex, Flash, Push and Pull, Cloud, Dynamic HTML5, AJAX, Web 2.0, Industry 4.0, the Internet of Things, Wearables, Mobile Apps and Mobile Devices, … some of them are partly buzzwords , However, they all have enormous influence on the required know-how and requirements for the load test tool and the load test concept, or the necessary effort for the load and performance test.
- How much should the test environment be similar to the production environment? It’s usually a budget question. Basically, the test environment should be as close as possible to the production environment during performance testing.
All these points should be recorded in a load test concept.
Challenges of performance test execution
Performance testing is not test automation
In order to make a load test representative, the major challenge is to make the load on the system as realistic as possible. Here are some aspects of particular importance: The number of concurrent users (concurrent VUs), their activity and how they use the system (scenarios, business processes or use cases). The quantity structure is also important for this.
Also the choice of a specific browser to be used for performance test execution can be relevant. For example, browsers use different parallelization of connections when accessing a web page, which in turn leads to different numbers of open server connections per user.
To assess this, it makes sense to gather data while performance test execution: analyze log data, evaluate monitoring data, communicate with project staff, users and managers, and extract information from tools, concepts, and production data.
Furthermore, it is relevant to start a test with a low load and then increase it to the desired amount. Since errors at startup could be overlooked with too much user interaction and thus the test loses its meaning. It is also important to scale the load generators to the expected load to avoid false results or test problems.
As you can already guess, performing load and performance tests is not easy. Therefore, it is important to leave this work to people who are aware of these challenges as well as have the necessary technical background.
Special challenges in performance testing of mobile apps and devices
The use of mobile apps has increased enormous due to smartphones and tablets. Therefore, there is also an urgent need of load test to include the resulting factors as:
- Emulation of different bandwidths
- Different connection qualities
- Potential disconnects
- Packet loss
- Changing radio masts with fast movement
It must also be taken into account that different volumes of data are delivered, depending on the browser used. Therefore, it makes sense to assemble test groups with typical different browser preferences in order to simulate and factor in these fluctuations in the load and performance test.
Furthermore, mobile devices greatly increase the variety of display resolutions (viewports). Responsive designs and implementations for different devices can make a big difference in which device or resolution a web application is viewed at. Due to responsive design the webserver response can be quite different. Possibly. Certain images are only loaded in small resolutions. Or maybe certain Java scripts will not run at all, so certain asynchronous JSON requests (eg. reloading widgets) will not be sent at low resolutions or on certain devices. At the beginning of the iphones Apple refused to let Flash run on their devices. So all Flash based applications did not run on Apple iphones at all.
If you want to evaluate end-to-end performance and thus also include client’s (here it is meant the client’s device) time, for example the time it takes for the client to run all scripts of a website and render the website, it gets even more complicated. Because of this and similar reasons, time and behavior of various devices must be considered. This represents a great challenge due to the high fragmentation of clients‘ devices.
Performance Test Tools
Commercial and open source tools and software
Due to the large number of load testing tools, it can be difficult to decide the right one in your specific need or case.
Basically, open source tools are cheaper, but usually require more effort and work to succeed. They are mainly supporting load and performance test on HTTP protocoll level. However, not all AUT can be tested well with them and the load test scripts are usually only feasible with more advanced programming skills. Reporting is often only possible with great manual effort, or it is possible to set up your own automated reporting system.
By contrast, commercial test tools bring their own reporting capabilities and usually also provide support for monitoring your servers and infrastructure. They often provide interfaces to APM (application performance monitoring) tools and support a variety of protocols and interfaces.
In addition to a load test tool, there are usually more like test management software and / or ALM (application lifecycle management) tools available in the tool landscape of the software test. The load testing tools, especially the commercial test tools, usually provide interfaces to specific test management and ALM tools.
Rocketlab Load & Performance Test Tool Portfolio
Rocketlab provides load and performance testing services since more than 2 decades now. Various software solutions and tools have come across and been tested by our experts in different environments and for different challenges and needs.
We now focus on a set of commercial and open source tools where we had best results and optimal experience from. Here is a list of the most important tools we use:
- LoadRunner** by micro focus (former HPE – Hewlett Packard Enterprise)
- Storm Runner Load by micro focus
- Neoload** by Neotys
* open source
** commercial tools, community version available