Async HTTP Requests
Deep dive · part of Python Asyncio
Pure asyncio doesn't ship with an HTTP client, but aiohttp and httpx do. Fetching many URLs concurrently is one of asyncio's killer use cases: hundreds of concurrent requests on a single thread.
Fetching hundreds of URLs sequentially wastes wall-clock time waiting on network latency. asyncio with httpx or aiohttp overlaps waits on a single thread: while one request awaits bytes, others progress. This is the canonical asyncio win for I/O-bound workloads.
You still need timeouts, connection limits, and backoff because concurrency amplifies failure modes—rate limits, file descriptor exhaustion, and thundering herds on flaky APIs all appear faster when you parallelize.
asyncio.gather schedules multiple coroutines and collects results (or exceptions with return_exceptions=True).
AsyncClient context manager pools connections; reuse one client per batch, not per URL.
Semaphore limits in-flight requests (bounded concurrency) protecting remote servers and local fds.
stream() and aiter_bytes() process large downloads without loading entire bodies into RAM.
Timeouts belong on each request (timeout=10) and optionally on the whole gather via asyncio.wait_for.
SSL, redirects, and auth mirror sync httpx/requests APIs with async method names.
Choose httpx for API symmetry with requests and HTTP/2 support; aiohttp is mature for server and client with slightly different session APIs. Both require asyncio.run(main()) or an existing loop in Jupyter (nest_asyncio in notebooks).
For CPU-bound post-processing of responses, offload to asyncio.to_thread or ProcessPoolExecutor—async only helps waiting, not parsing huge JSON on one core.
Retry policies combine Semaphore with tenacity or custom loops; jitter backoff avoids synchronized retries after outages.
Per-host semaphores keyed by netloc improve crawl throughput when batches mix many domains.
Separate semaphores per netloc when crawling mixed domains to avoid one slow host blocking all.
Creating a new AsyncClient per URL, destroying connection pooling benefits.
Unbounded gather on millions of URLs without semaphores, hitting EMFILE or bans.
Calling blocking requests.get inside async def, freezing the event loop.
Ignoring status codes and treating 500 responses as success without raise_for_status.
Reuse one AsyncClient per logical batch and set limits=Limits(max_connections=20).
Wrap gather in asyncio.wait_for for global deadlines on scrape jobs.
Log per-host failure rates and circuit-break hot domains.
Test with respx or pytest-httpx mocking async transports.
Re-read the examples below with these ideas in mind; change variable names and inputs to match your own project.
The program below demonstrates fetch many urls. Read the comments on each line, run the code, then change names or values to see how the output shifts.
# Example: Fetch many URLs
# Run in the REPL or save as a .py file and execute with python.
import asyncio, httpx
async def fetch(client, url):
r = await client.get(url, timeout=10)
return url, r.status_code, len(r.content)
async def main(urls):
async with httpx.AsyncClient() as c:
return await asyncio.gather(*(fetch(c, u) for u in urls))
urls = ["https://example.com", "https://httpbin.org/get"] * 5
print(asyncio.run(main(urls)))This sample walks through bounded concurrency in a small, runnable script. Paste it into the REPL or save it as a .py file before you continue to the next block.
# Example: Bounded concurrency
# Run in the REPL or save as a .py file and execute with python.
import asyncio, httpx
async def worker(sem, client, url):
async with sem:
r = await client.get(url, timeout=10)
return url, r.status_code
async def main(urls, concurrency=5):
sem = asyncio.Semaphore(concurrency)
async with httpx.AsyncClient() as c:
return await asyncio.gather(*(worker(sem, c, u) for u in urls))
print(asyncio.run(main(["https://example.com"] * 20)))Here is a hands-on illustration of streaming response. Follow the inline comments first; only then execute the snippet and compare the result with what you expected.
# Example: Streaming response
# Run in the REPL or save as a .py file and execute with python.
import asyncio, httpx
async def stream(url):
async with httpx.AsyncClient() as c:
async with c.stream("GET", url) as r:
async for chunk in r.aiter_bytes(chunk_size=1024):
print("got", len(chunk), "bytes")
asyncio.run(stream("https://httpbin.org/bytes/4096"))The program below demonstrates gather urls. Read the comments on each line, run the code, then change names or values to see how the output shifts.
# httpx AsyncClient fetches many URLs concurrently
import asyncio # event loop
import httpx # async HTTP (pip install httpx)
async def fetch(client, url): # single GET
r = await client.get(url, timeout=10) # await response
return url, r.status_code # summary tuple
async def main(): # runner
urls = ["https://example.com", "https://httpbin.org/get"] # targets
async with httpx.AsyncClient() as client: # shared client
out = await asyncio.gather(*(fetch(client, u) for u in urls)) # parallel
print(out) # list of tuples
asyncio.run(main()) # executeThis sample walks through semaphore limit in a small, runnable script. Paste it into the REPL or save it as a .py file before you continue to the next block.
# Semaphore caps concurrent in-flight requests
import asyncio, httpx # deps
async def worker(sem, client, url): # bounded worker
async with sem: # acquire slot
r = await client.get(url, timeout=10) # fetch
return url, r.status_code # result
async def main(): # orchestrator
sem = asyncio.Semaphore(2) # max two concurrent
urls = ["https://example.com"] * 5 # repeated URL
async with httpx.AsyncClient() as client: # client
print(await asyncio.gather(*(worker(sem, client, u) for u in urls))) # run
asyncio.run(main()) # startRelated deep dives on Python Asyncio: