Apache Benchmark stress test
Apache Bench is a good tool to measure raw HTTP performance. It is not indicative for application performance, but rather aimed at testing your infrastructure.
This document describes the configuration to run ab
to make meaningful
comparisons and see which changes have which effect.
Running ab
ab -n 1000 -c 100 https://test.openzaak.nl/zaken/api/v1/
The most interesting statistic is the number of requests per second, which you want to be as high as possible.
Philosophy
We run a total of 1000 requests to measure performance, using up to 100 concurrent users. Playing around with the concurrency level can reveal hot spots or ideal numbers.
We test and API endpoint that has no IO and is inexpensive to generate. We don’t want to stress the application with queries or complicated template rendering, which is why we pick an API root that generates very little JSON.
Note that if you are running the tests directly against other services, that
you may have to specify the Host
header: -H 'Host: localhost'
, otherwise
Django will generate an HTTP 400 because of the InvalidHostHeader
, which
is very fast but not representative.
Also note that you want to run ab
from a fast client machine. Aim for a
server in a different data center. We’ve seen cases where our client is limited
in requests per second because of a slow home network.
Parameters to vary
Traefik/Ingress: SSL termination happens here. We’ve seen Traefik CPU spikes with large private keys (4096 bits), and performance did improve a bit by changing to EC 256 bits at the cost of a worse SSL rating (A+ to A).
Number of nginx replicas - more replicas can handle more traffic from Traefik
Number of Open Zaak replicas - by default each container can handle 4 concurrent requests (2 process * 2 threads per process)
Number of Open Zaak processes/threads in a container
CPU/memory resource requests/limits in Kubernetes. Assigning more CPU/memory power will probably give better results. This applies for the Traefik, nginx and Open Zaak containers.