Job Priority Management

Purpose

The current architecture of the charon daemon uses some synchronous blocking operations while working on IKE_SAs. Two examples of such operations are communications with a RADIUS server via EAP-RADIUS or fetching CRL/OCSP information during certificate chain verification. Under high load conditions the thread pool may run out of threads and some more important jobs such as liveness checking may not get executed in time.

To prevent thread starvation in such situations job priorities were introduced. The job processor will reserve some threads for higher priority jobs. These threads are not available for lower priority jobs that might go into temporary lock.

Implementation

Currently 4 priorities have been defined. They are used by the the charon daemon as follows:

Priority Description

CRITICAL

Priority for long-running dispatcher jobs

HIGH

INFORMATIONAL exchanges, as used by liveness checking (DPD)

MEDIUM

Everything not HIGH/LOW, including IKE_SA_INIT processing

LOW

IKE_AUTH message processing. RADIUS and CRL fetching block here, hence the low priority

Although IKE_SA_INIT processing is computationally expensive, it is explicitly kept in the MEDIUM class. This allows charon to do the DH exchange while other threads are blocked in IKE_AUTH. To prevent the daemon from accepting more IKE_SA_INIT messages than it can handle, use IKE_SA_INIT dropping (see below).

The thread pool processes jobs strictly by priority, meaning that it will consume all higher priority jobs before looking for ones with lower priority. Further it reserves threads for certain priorities. A priority class having reserved n threads will always have n threads available for this class (either currently processing a job, or waiting for one).

Configuration

To always have enough threads available for higher priority tasks, they must be reserved for each priority class. This is done in the charon.processor subsection of strongswan.conf:

Key Default Description

priority_threads.critical

0

Threads reserved for CRITICAL priority class jobs

priority_threads.high

0

Threads reserved for HIGH priority class jobs

priority_threads.medium

0

Threads reserved for MEDIUM priority class jobs

priority_threads.low

0

Threads reserved for LOW priority class jobs

Let’s consider the following configuration:

charon {
  processor {
    priority_threads {
	  high = 1
	  medium = 4
	}
  }
}

  • With this configuration one thread is reserved for HIGH priority tasks. As currently only liveness checking and `vici message processing is done with high priority, one or two threads should be sufficient.

  • The MEDIUM class mostly processes non-blocking jobs. Unless your setup is experiencing many blocks in locks while accessing shared resources, threads for one or two times the number of CPU cores is fine.

  • It is usually not required to reserve threads for CRITICAL jobs. Jobs in this class rarely return and do not release their thread to the pool.

  • The remaining threads are available for LOW priority jobs. Reserving threads does not make sense (until we have an even lower priority).

Monitoring Thread Usage and Job Load

To see what the threads are actually doing, invoke swanctl --stats. Under high load something like this will show up:

worker threads: 32 total, 2 idle, working: 5/1/2/22
job queues: 0/0/1/149
jobs scheduled: 198

From 32 worker threads,

  • 2 are currently idle.

  • 5 are running CRITICAL priority jobs (dispatching from sockets, etc.).

  • 1 is currently handling a HIGH priority job. This is actually the thread currently providing this information via vici.

  • 2 are handling MEDIUM priority jobs, likely IKE_SA_INIT or CREATE_CHILD_SA messages.

  • 22 are handling LOW priority tasks, probably waiting for an EAP-RADIUS response while processing IKE_AUTH messages.

The job queue load shows how many jobs are queued for each priority, ready for execution. The single MEDIUM priority job will get executed immediately, as we have two spare threads reserved for MEDIUM class jobs.

IKE_SA_INIT Dropping

If a responder receives more connection requests per second than it can handle, it does not make sense to accept more IKE_SA_INIT messages. If they are queued but can’t get processed in time, an answer might be sent after the client has already given up and restarted its connection setup. This additionally increases the load on the responder.

To limit the responder load resulting from new connection attempts, the charon daemon can drop IKE_SA_INIT messages just after reception. There are two mechanisms to decide if this should happen that can be configured with strongswan.conf options:

Key Default Criteria Description*

charon.init_limit_half_open

0

Number of half open IKE_SAs

Half open IKE_SAs are SAs in connecting state, but not yet established

charon.init_limit_job_load

0

Job load on the processor

Number of jobs currently queued for processing, sum over all job priorities

The second limit includes load from other jobs, such as rekeying. Choosing a good value is difficult and depends on the hardware and generated load.

The first limit is simpler to calculate, but includes the load from new connections only. If your responder is capable of negotiating 100 tunnels/s, you might set this limit to 1000. It will then drop new connection attempts if generating a response would require more than 10 seconds. If you’re allowing for a maximum response time of more than 30 seconds, consider adjusting the timeout for connecting IKE_SAs in strongswan.conf:

Key Default Description

charon.half_open_timeout

30

Timeout, in seconds, of connecting IKE_SA

A responder, by default, deletes an IKE_SA if the initiator does not establish it within 30 seconds. Under high load a higher value might be required.