Notifications performance
Open Zaak sends notifications using Celery as an asynchronous task queue. Notification delivery failure to the Notifications API/service may happen, which is why we leverage “autoretry” for these failed tasks.
The reliability and performance of sending notifications was analyzed and the results are reported below. In this test case, we observed the behaviour of notifications triggered by “case create” events.
Sending notifications under burst load
We measure the notifications median time and failure rate under an increasing amount of concurrent users. The goal is to observe the reliability of notification delivery and delay between notification scheduling and it actually arriving at the Notifications API.
Test specifications:
1 celery worker
3 processes for the worker
no waiting time between requests for 1 user
all users send requests to create zaak (
POST /api/v1/zaken
)time under load is 1 minute
Users |
Request Count |
Requests/s |
Failure Count |
Median Response Time (ms) |
Notifications failure |
Median Notification Time (ms) |
---|---|---|---|---|---|---|
10 |
1430 |
24.183 |
0 |
390 |
0 |
88.572 |
20 |
1479 |
25.034 |
0 |
740 |
0 |
83.922 |
40 |
1429 |
24.189 |
0 |
1200 |
0 |
86.557 |
The data suggests no reliability issues, even with only a single Celery worker, as the failure rates are 0 for both the API operation and notification delivery.
Note that the worker containers can scale horizontally and we recommend deploying at least two worker containers, ideally distributed over different hardware instances for high-available set-ups.
Automatic retry for notifications
By default, sending notifications has automatic retry behaviour, i.e. if the notification publishing task has failed, it will automatically be scheduled/tried again until the maximum retry limit has been reached.
Autoretry explanation and configuration
In Configuratie > Notificatiescomponentconfiguratie admin page the autoretry settings can be configured:
Notification delivery max retries: the maximum number of retries the task queue will do if sending a notification failed. Default is
5
.Notification delivery retry backoff: a boolean or a number. If this option is set to
True
, autoretries will be delayed following the rules of exponential backoff. If this option is set to a number, it is used as a delay factor. Default is3
.Notification delivery retry backoff max: an integer, specifying number of seconds. If
Notification delivery retry backoff
is enabled, this option will set a maximum delay in seconds between task autoretries. Default is48
seconds.
With the assumption that the requests are done immediately we can model the notification tasks schedule with the default configurations:
At 0s the request to create a zaak is made, the notification task is scheduled, picked up by worker and failed
At 3s with 3s delay the first retry happens (1s *
Notification delivery retry backoff
)At 9s with 6s delay - the second retry
At 21s with 12s delay - the third retry
At 45s with 24s delay - the fourth retry
At 1m33s with 48s delay - the fifth retry, which is the last one.
So if the Notification API is up after 1 min of downtime the default configuration can handle it automatically.
Autoretry test with default configuration
In this scenario, we measure how autoretry behaves with increasing downtime durations of the notification service and with the default autoretry configuration.
Test specifications:
1 celery worker
3 processes for the worker
10 concurrent users
no waiting time between requests for 1 user
all users send requests to create zaak (
POST /api/v1/zaken
)requests are sent for a total duration of 1 min
auto retry settings: default (5 maximum attempts per task)
Notification downtime (s) |
Request Count |
Failure Count |
Notifications failed |
Notifications retried |
Notifications processed |
---|---|---|---|---|---|
10 |
1486 |
0 |
0 |
919 |
2114 |
60 |
1478 |
0 |
0 |
3729 |
5195 |
300 |
1509 |
0 |
1276 |
6380 |
7908 |
Notifications retried
includes all notifications that were automatically retried. The minimum number equals the number of failed notifications. If all notifications are eventually failedNotifications retried
=Notifications failed
*maximum retries per task
.Notifications processed
includes failed, retried and succeeded notifications.
According to the result if the notification server is down for less than 1 min, then all notifications would be eventually send. If it’s expected that the notification service is down for longer time the autoretry configuration should be adjusted in the admin.
Autoretry test with adjusted configuration
In the last case in the test run, when the notification server was down for 300 seconds, all notifications have failed. Let’s change the autoretry configuration, so the notifications will be retrying during 5 minutes:
Notification delivery max retries is changed to
7
.Notification delivery retry backoff max is changed to
192
.
With this configuration we expect the following task schedule:
At 189s with 96s delay - the 6th retry
At 381s with 192s delay - the 7th retry, which is now the last one.
Test specifications:
1 celery worker
3 processes for the worker
10 concurrent users
no waiting time between requests for 1 user
all users send requests to create zaak (
POST /api/v1/zaken
)requests are sent for a total duration of 1 min
auto retry settings:
7
Notification downtime (s) |
Request Count |
Failure Count |
Notifications failed |
Notifications retried |
Notifications processed |
---|---|---|---|---|---|
10 |
1328 |
0 |
0 |
639 |
2002 |
60 |
1335 |
0 |
0 |
3846 |
5209 |
300 |
1262 |
0 |
0 |
7427 |
8717 |
600 |
1393 |
0 |
1181 |
8267 |
9687 |
The adjusted autoretry configuration resulted in 0 failed notifications for 5 min downtime with the tradeoff of the increased amount of the retried ones. However the adjusted settings were not efficient for the 10 min downtime. Therefore we advice to take into account the statistics of server downtimes before adjusting autoretry settings.