Keepalive in the connection pool

Both pool clients can probe their idle connections at a fixed interval using the server job krn.CheckServerConnection. Server-side that job is nothing but a counter running over the existing session — the cheapest sign of life available. It keeps NAT, firewall and idle timeouts from expiring and exposes silently discarded sockets, because the write or read then fails.

Without keepalive a dead connection only surfaces on the next business call: that call fails, the pool discards the connection, and only the call after it succeeds again. Keepalive is what prevents this.

1. Configuration

from ecmind_blue_client.pool import SyncPoolClient

pool = SyncPoolClient(
    servers="server1:4000:1",
    username="<user>",
    password="<pass>",
    keepalive_interval=300,   # seconds
)

None (default)

keepalive disabled, no background worker is started

⇐ 0

disabled as well

> 0

seconds between two sweeps

AsyncPoolClient takes the same parameter and uses an asyncio.Task instead of a thread.

Pick the interval from the shortest idle timeout on the way to the server (firewall, load balancer, NAT gateway) and stay well below it. 300 seconds is a reasonable starting point.

2. What one sweep does

  1. The worker starts lazily: only when the first connection is created. A pool that is never used spawns neither a thread nor a task.

  2. The interval is measured between the end of one sweep and the start of the next. A slow sweep therefore causes neither catch-up bursts nor overlapping sweeps.

  3. Each sweep takes a snapshot of the number of currently idle connections and probes exactly that many. A connection that was probed and put back cannot be probed twice in the same sweep.

  4. Every connection taken out is probed with krn.CheckServerConnection (Flags=0):

    success

    the connection goes back into the pool, keepalive_count and last_keepalive_at are updated

    failure (return code != 0 or a network error)

    the connection is closed and removed from the bookkeeping

3. Properties

Idle connections only

A checked-out connection is out of the queue and invisible to the sweep. Running jobs are never disturbed.

No lock held across the RPC call

Only the queue bookkeeping runs under the pool lock, the server job does not. A hanging server therefore never blocks the pool for the duration of the sweep.

No reconnect inside the sweep

A broken connection is only closed, never replaced. The pool shrinks instead of blocking slots with dead connections; the next call creates a fresh one on demand.

No error surfaced to the application

Failures are logged at logging.DEBUG on the ecmind_blue_client.pool._sync_pool_client / …​_async_pool_client logger only. There is no event, no callback and no exception in the calling thread.

Separate from the application statistics

Keepalive traffic counts towards keepalive_count / last_keepalive_at, never towards call_count / last_call_at.

4. What keepalive does not do

  • It does not check configured servers that currently have no connection. Use ecm.check_connections() for that, which opens a throwaway connection per server.

  • It does not touch the shared server session. If the server-side session is gone (server restart, session timeout, krn.SessionDrop), only the next connection setup notices: krn.SessionAttach then returns a different SessionGUID, the stale entry is discarded and a full login is performed. Keepalive and session recovery are two independent mechanisms that complement each other: keepalive removes the dead connection, the next execute() builds a new one with a fresh session.

5. Reading the statistics

for stats in pool.connection_stats():
    print(
        f"{stats.hostname}:{stats.port} "
        f"calls={stats.call_count} "
        f"keepalives={stats.keepalive_count} "
        f"last_keepalive={stats.last_keepalive_at}"
    )

6. Shutting the pool down cleanly

While a keepalive is running, close the pool explicitly. close() (sync) and aclose() (async) stop the worker and close every idle connection; both are safe to call more than once.

# Sync
pool = SyncPoolClient(..., keepalive_interval=300)
try:
    ...
finally:
    pool.close()

# or as a context manager
with SyncPoolClient(..., keepalive_interval=300) as pool:
    ...
# Async
async with AsyncPoolClient(..., keepalive_interval=300) as pool:
    ...

The worker holds the pool only through a weak reference. A pool that is simply let go of without close() can therefore still be garbage collected — the worker then exits on its own. That is the fallback, not the recommended path: timing and ordering are up to the garbage collector.

7. FastAPI

Build the pool once in the lifespan and close it explicitly on shutdown:

@asynccontextmanager
async def lifespan(app: FastAPI):
    client = AsyncPoolClient(
        servers=os.environ["ECM_SERVERS"],
        username=os.environ["ECM_USERNAME"],
        password=os.environ["ECM_PASSWORD"],
        keepalive_interval=300,
    )
    app.state.ecm = ECM(client)
    yield
    await client.aclose()
    app.state.ecm = None

If only the ECM instance is passed around, ecm.client gets you back to the pool: await ecm.client.aclose().

Two things that are easy to miss in a web service:

  • The asyncio.Task is created in the running event loop when the first connection is created, not in the constructor. A pool built in the lifespan therefore always binds to the loop that actually uses it.

  • If the service runs with several worker processes (uvicorn --workers, Gunicorn), each process has its own pool and therefore its own keepalive worker. The interval applies per process.

The complete example is in FastAPI integration.