Wt

Problem Description¶

We observed a scenario in which a single blocked worker thread can eventually cause all worker threads to become blocked.

Initial Situation¶

A request for session S1 is being processed by worker thread T1.

Due to a bug in the backend, the request enters a deadlock (or remains blocked indefinitely). As a result, worker thread T1 is blocked while still holding WebSession::mutex_.

Although this specific bug will be fixed, we cannot guarantee that similar situations will never occur again. Therefore, the session cleanup logic should be robust against worker threads that unexpectedly hold a session lock for an extended period.

Follow-up Problem¶

While T1 remains blocked, WebSession::mutex_ for session S1 is never released.

Eventually, session S1 expires.

Meanwhile, another worker thread T2 successfully finishes processing a request for a different session (S2) and calls:

WebController::expireSessions();

During session cleanup, expireSessions() attempts to lock every expired session:

WebSession::Handler handler(session, WebSession::Handler::LockOption::TakeLock);

which eventually executes:

WebSession::Handler::Handler(...)
{
    ...
    lock_.lock();
}

Since WebSession::mutex_ is still held by T1, thread T2 blocks while trying to acquire the lock.

The same sequence repeats for every worker thread that later executes WebController::expireSessions(). Each thread blocks while attempting to lock the expired session S1.

As a result, all worker threads eventually become blocked, effectively preventing the server from processing any further requests.

Proposed Solution¶

Instead of unconditionally acquiring the session lock, expireSessions() should first attempt a non-blocking lock.

WebSession::Handler handler(session, WebSession::Handler::LockOption::TryLock);
if (!handler.haveLock())
{
    continue;
}

If the lock cannot be acquired immediately, the expired session is simply skipped during the current cleanup cycle.

If T1 was only blocked temporarily (rather than permanently), it will eventually release WebSession::mutex_. A subsequent invocation of WebController::expireSessions() will then successfully acquire the lock and remove the expired session.

This approach prevents a single blocked session from propagating into a system-wide worker thread stall while still ensuring that expired sessions are eventually cleaned up once they become accessible.