Engineering ausdata.io: the per-thread _client cache pattern

How a cross-thread asyncio loop crash brought down ausdata-api under uvicorn worker pressure, and the threading.local() fix that solved it.

In the second week of May we shipped a Fly.io deployment of ausdata-api that started 500-ing roughly one in fifty requests under modest concurrent load. The error in the logs was always the same:

RuntimeError: Task <Task pending name='Task-12' coro=<...>> got Future <Future pending> attached to a different loop

If you've run FastAPI under uvicorn with httpx and a process-global async client, you've probably seen this one. This is the write-up.

The setup

ausdata-api is a thin FastAPI service that fans out to ten sister MCP libraries. Each sister exposes an async function that internally holds an httpx.AsyncClient. The natural pattern is to cache the client at module level:

# DON'T DO THIS
_client: httpx.AsyncClient | None = None

async def get_client() -> httpx.AsyncClient:
    global _client
    if _client is None:
        _client = httpx.AsyncClient(timeout=30.0)
    return _client

This is the pattern every httpx tutorial shows. It works on a single-threaded asyncio event loop. It fails the moment uvicorn spawns more than one worker thread, because each worker thread runs its own event loop, and httpx.AsyncClient binds its connection pool to the loop alive at construction time.

When request A on thread 1 constructs the client, then request B on thread 2 tries to reuse it, B's event loop sees a Future that was attached to A's loop. RuntimeError.

Why we didn't see it locally

Local dev was uvicorn --workers 1 --reload. One process, one loop, one thread. The bug is invisible until you scale horizontally.

Fly.io's default config for our shared-cpu-1x machine spins up uvicorn with --workers 2. The crash rate scaled almost linearly with concurrent requests above ~5 RPS, which we hit briefly during a Discord launch.

The wrong fixes we tried first

Attempt 1: re-construct the client per request. Works, but blows the connection pool every call. Latency jumped from 80ms p50 to 340ms p50. Untenable.

Attempt 2: use lifespan context to construct one client. This is the FastAPI-canonical pattern. It would work if FastAPI's lifespan ran per-worker, but it runs per-process, and uvicorn worker threads share the process. Same bug.

Attempt 3: lock around client creation. Doesn't help. The bug isn't a race on construction, it's that the loop the client was constructed on isn't the loop the consumer is running.

The fix: `threading.local()`

The pattern that actually works is a per-thread client, scoped by thread identity:

import threading
import httpx

_thread_local = threading.local()

async def get_client() -> httpx.AsyncClient:
    client = getattr(_thread_local, "client", None)
    if client is None:
        client = httpx.AsyncClient(
            timeout=httpx.Timeout(30.0, connect=10.0),
            limits=httpx.Limits(max_keepalive_connections=20, max_connections=50),
        )
        _thread_local.client = client
    return client

Each worker thread gets its own _thread_local.client. The client is constructed lazily on the first request that lands on that thread, then reused for the lifetime of the thread. No cross-loop contamination, full keep-alive pooling, no per-request setup cost.

For the FastAPI-side dependency:

from fastapi import Depends

async def http_client() -> httpx.AsyncClient:
    return await get_client()

@app.get("/v1/real-rate-regime")
async def real_cash_rate(client: httpx.AsyncClient = Depends(http_client)):
    response = await client.get("https://...")
    ...

The long-lived loop, separately

There's a second piece, which we shipped a few days later (0.6.38): a single persistent asyncio event loop per worker, used by all the sister-library calls. The sisters internally use asyncio.run(...) in places, which constructs a new loop each call. Fine in a script, catastrophic in a server.

We patched the sisters to discover an existing loop via asyncio.get_event_loop() first, and only fall back to asyncio.new_event_loop() if none exists. Combined with the threading.local() client cache, the 500 rate went from ~2% to zero across a 24-hour window.

How we verified the fix

# tests/test_concurrent_client_isolation.py
import asyncio
from concurrent.futures import ThreadPoolExecutor
from ausdata_api.http import get_client

async def hit_endpoint(i: int) -> int:
    client = await get_client()
    r = await client.get("https://api.ausdata.io/v1/health")
    return r.status_code

def thread_worker(i: int) -> int:
    return asyncio.run(hit_endpoint(i))

def test_concurrent_threads_no_loop_crash():
    with ThreadPoolExecutor(max_workers=8) as pool:
        results = list(pool.map(thread_worker, range(200)))
    assert all(s == 200 for s in results), f"got {results.count(200)}/200"

200 calls across 8 worker threads, zero loop errors. Ran it ten times in a row before tagging the release.

What we'd do differently

The single biggest lesson: --workers 1 is a lie about your prod config. It hides every threading-shaped bug. We now run local dev with --workers 2 by default, which surfaces these issues in the dev loop instead of in Fly logs at 02:00 Sydney.

The second lesson: don't trust framework tutorials' default patterns when you scale beyond a single worker. The httpx docs' "one client per process" pattern is correct for a script, wrong for a multi-threaded ASGI server. The fix isn't novel, it's in httpx issue #1990 and dotted through the asyncio docs, but it's not in any "FastAPI quickstart" I could find.

What this isn't

This pattern is overkill for:

A single-worker dev server.
A serverless / per-invocation Lambda (each invocation is fresh; no thread reuse).
A pure synchronous Flask app (no event loops involved).

It's the right pattern for: ASGI server + multi-worker + cached HTTP client + httpx.

Pricing

ausdata.io is a public free-tier-first API. Free tier is 500 calls/month, $29 Analyst tier is 10k, $99 Pro tier is 100k with webhooks. The engineering you read about above is what keeps the free tier reliable under bursty load, you're not on a less-stable code path because you're not paying.

Grab a free key at ausdata.io.