Keepalive in the connection pool

Both pool clients can probe their idle connections at a fixed interval using the server job krn.CheckServerConnection. Server-side that job is nothing but a counter running over the existing session — the cheapest sign of life available. It keeps NAT, firewall and idle timeouts from expiring and exposes silently discarded sockets, because the write or read then fails.

Without keepalive, a dead connection in the sync pool only surfaces on the next business call: that call fails, the pool discards the connection, and only the call after it succeeds again. In the async pool, every loss the kernel can observe (the server or a device in between closing the socket) is detected instantly through the transport’s connection_lost event, even without keepalive. Keepalive is the remaining safety net: it catches the losses the kernel cannot see — a firewall rule removed, a NAT entry expired — and it is the only detection available to the sync pool.

1. Configuration

from ecmind_blue_client.pool import SyncPoolClient

pool = SyncPoolClient(
    servers="server1:4000:1",
    username="<user>",
    password="<pass>",
    keepalive_interval=300,   # seconds
)

None (default)

keepalive disabled, no background worker is started

⇐ 0

disabled as well

> 0

seconds between two sweeps

AsyncPoolClient takes the same parameter and uses an asyncio.Task instead of a thread.

Pick the interval from the shortest idle timeout on the way to the server (firewall, load balancer, NAT gateway) and stay well below it. 300 seconds is a reasonable starting point.

2. What one sweep does

The worker starts lazily: only when the first connection is created. A pool that is never used spawns neither a thread nor a task.
The interval is measured between the end of one sweep and the start of the next. A slow sweep therefore causes neither catch-up bursts nor overlapping sweeps.
Each sweep takes a snapshot of the number of currently idle connections and probes exactly that many. A connection that was probed and put back cannot be probed twice in the same sweep.

Every connection taken out is probed with krn.CheckServerConnection (Flags=0):

success

the connection goes back into the pool, keepalive_count and last_keepalive_at are updated

failure (return code != 0 or a network error)

the connection is closed and removed from the bookkeeping

3. Instant detection of lost connections (async pool)

The async pool hooks every pooled connection’s transport connection_lost event. When the server — or a device in between — closes the socket while it is idle, the pool notices at the very moment of loss, without waiting for the next borrow or sweep:

the connection is removed from the bookkeeping and from the idle queue, so the next call transparently creates a fresh one;
an INFO line is logged on ecmind_blue_client.pool._async_pool_client (with a one-time hint to enable keepalive if it is disabled).

A connection that is checked out when it dies is logged at DEBUG instead: the calling code already receives the connection error and disposes of the connection through its usual exception path.

The hook covers peer-initiated closes — server idle timeouts, server restarts, an explicit close. It cannot see a connection that was dropped silently (firewall rule removed, NAT entry expired), because then no close ever reaches the socket; that case is what keepalive is for. The sync pool has no event-loop equivalent and relies on the keepalive sweep.

The two mechanisms divide the failure space: the hook reacts instantly to every loss the kernel can observe, keepalive polls for the losses the kernel cannot see.

4. Properties

Idle connections only: A checked-out connection is out of the queue and invisible to the sweep. Running jobs are never disturbed.
No lock held across the RPC call: Only the queue bookkeeping runs under the pool lock, the server job does not. A hanging server therefore never blocks the pool for the duration of the sweep.
No reconnect inside the sweep: A broken connection is only closed, never replaced. The pool shrinks instead of blocking slots with dead connections; the next call creates a fresh one on demand.
No error surfaced to the application: Failures are logged at logging.DEBUG on the ecmind_blue_client.pool._sync_pool_client / …_async_pool_client logger only. There is no event, no callback and no exception in the calling thread.
Separate from the application statistics: Keepalive traffic counts towards keepalive_count / last_keepalive_at, never towards call_count / last_call_at.

5. What keepalive does not do

It does not check configured servers that currently have no connection. Use ecm.check_connections() for that, which opens a throwaway connection per server.
It does not touch the shared server session. If the server-side session is gone (server restart, session timeout, krn.SessionDrop), only the next connection setup notices: krn.SessionAttach then returns a different SessionGUID, the stale entry is discarded and a full login is performed. Keepalive and session recovery are two independent mechanisms that complement each other: keepalive removes the dead connection, the next execute() builds a new one with a fresh session.

6. Reading the statistics

for stats in pool.connection_stats():
    print(
        f"{stats.hostname}:{stats.port} "
        f"calls={stats.call_count} "
        f"keepalives={stats.keepalive_count} "
        f"last_keepalive={stats.last_keepalive_at}"
    )

7. Shutting the pool down cleanly

While a keepalive is running, close the pool explicitly. close() (sync) and aclose() (async) stop the worker and close every idle connection; both are safe to call more than once.

# Sync
pool = SyncPoolClient(..., keepalive_interval=300)
try:
    ...
finally:
    pool.close()

# or as a context manager
with SyncPoolClient(..., keepalive_interval=300) as pool:
    ...

# Async
async with AsyncPoolClient(..., keepalive_interval=300) as pool:
    ...

The worker holds the pool only through a weak reference. A pool that is simply let go of without close() can therefore still be garbage collected — the worker then exits on its own. That is the fallback, not the recommended path: timing and ordering are up to the garbage collector.

8. FastAPI

Build the pool once in the lifespan and close it explicitly on shutdown:

@asynccontextmanager
async def lifespan(app: FastAPI):
    client = AsyncPoolClient(
        servers=os.environ["ECM_SERVERS"],
        username=os.environ["ECM_USERNAME"],
        password=os.environ["ECM_PASSWORD"],
        keepalive_interval=300,
    )
    app.state.ecm = ECM(client)
    yield
    await client.aclose()
    app.state.ecm = None

If only the ECM instance is passed around, ecm.client gets you back to the pool: await ecm.client.aclose().

Two things that are easy to miss in a web service:

The asyncio.Task is created in the running event loop when the first connection is created, not in the constructor. A pool built in the lifespan therefore always binds to the loop that actually uses it.
If the service runs with several worker processes (uvicorn --workers, Gunicorn), each process has its own pool and therefore its own keepalive worker. The interval applies per process.

The complete example is in FastAPI integration.