In order to make a load test representative, the major challenge is to make the load on the system as realistic as possible. Here are some aspects of particular importance: The number of concurrent users (concurrent VUs), their activity and how they use the system (scenarios, business processes or use cases). The quantity structure is also important for this.
Also the choice of a specific browser to be used for performance test execution can be relevant. For example, browsers use different parallelization of connections when accessing a web page, which in turn leads to different numbers of open server connections per user.
To assess this, it makes sense to gather data while performance test execution: analyze log data, evaluate monitoring data, communicate with project staff, users and managers, and extract information from tools, concepts, and production data.
Furthermore, it is relevant to start a test with a low load and then increase it to the desired amount. Since errors at startup could be overlooked with too much user interaction and thus the test loses its meaning. It is also important to scale the load generators to the expected load to avoid false results or test problems.
As you can already guess, performing load and performance tests is not easy. Therefore, it is important to leave this work to people who are aware of these challenges as well as have the necessary technical background.
Special challenges in performance testing of mobile apps and devices
The use of mobile apps has increased enormous due to smartphones and tablets. Therefore, there is also an urgent need of load test to include the resulting factors as:
- Emulation of different bandwidths
- Different connection qualities
- Potential disconnects
- Packet loss
- Changing radio masts with fast movement
It must also be taken into account that different volumes of data are delivered, depending on the browser used. Therefore, it makes sense to assemble test groups with typical different browser preferences in order to simulate and factor in these fluctuations in the load and performance test.
Furthermore, mobile devices greatly increase the variety of display resolutions (viewports). Responsive designs and implementations for different devices can make a big difference in which device or resolution a web application is viewed at. Due to responsive design the webserver response can be quite different. Possibly. Certain images are only loaded in small resolutions. Or maybe certain Java scripts will not run at all, so certain asynchronous JSON requests (eg. reloading widgets) will not be sent at low resolutions or on certain devices. At the beginning of the iphones Apple refused to let Flash run on their devices. So all Flash based applications did not run on Apple iphones at all.
If you want to evaluate end-to-end performance and thus also include client’s (here it is meant the client’s device) time, for example the time it takes for the client to run all scripts of a website and render the website, it gets even more complicated. Because of this and similar reasons, time and behavior of various devices must be considered. This represents a great challenge due to the high fragmentation of clients’ devices.