Making Web Requests

The web is made of HTTP requests: a client sends a method (GET, POST, PUT, DELETE) to a URL with optional headers and a body; a server returns a status code, response headers, and a body. Python has two ways to speak HTTP: the standard library's urllib.request and the de-facto third-party library requests. For everything beyond a one-off urlopen, requests (or its async sibling httpx) is dramatically more pleasant.

A minimal GET with requests is a single line: r = requests.get("https://api.example.com", timeout=10). Always pass timeout — requests with no timeout hang forever when the server stalls. r.status_code, r.headers, r.text, r.json() give you the response. r.raise_for_status() raises an exception on 4xx/5xx.

For POSTs, pass json={...} to send a JSON body (it sets the Content-Type header for you) or data={...} for form-urlencoded. Add authentication with auth=(user, pw) for HTTP Basic, or set headers={"Authorization": f"Bearer {token}"} for token auth. requests.Session() reuses the underlying TCP connection across calls, which is both faster and courteous to the server.

Three habits save pain: always set a timeout; always call raise_for_status (or check status_code yourself); and retry transient failures with exponential backoff. requests plays nicely with urllib3.util.Retry, or you can use tenacity for richer retry policies.

GET, POST, and Session

requests.get(url, params={"q": "python"}) URL-encodes the query string for you. requests.post(url, json={"a": 1}) sends JSON. Session persists cookies and headers across calls: s = requests.Session(); s.headers.update({"User-Agent": "demo/1.0"}).

For binary downloads, stream to disk: with requests.get(url, stream=True) as r: r.raise_for_status(); for chunk in r.iter_content(64*1024): f.write(chunk).

Errors, retries, and async

r.raise_for_status() raises HTTPError; catch it specifically, translate to your own error type at the edge. Don't catch Exception and swallow.

For async I/O, use httpx.AsyncClient. It has the same API but async/await-friendly. Useful when hitting dozens of endpoints concurrently.

HTTP tools.

Making Web Requests code example

The script uses the stdlib urllib so it runs anywhere; the sidebar shows the requests version for reference.

# Lesson: Making Web Requests
import json
import urllib.error
import urllib.parse
import urllib.request


def get_json(url: str, timeout: float = 10.0) -> dict:
    req = urllib.request.Request(url, headers={"User-Agent": "pythondeck-demo/1.0"})
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        if resp.status != 200:
            raise urllib.error.HTTPError(url, resp.status, "not OK", resp.headers, None)
        return json.loads(resp.read().decode("utf-8"))


def build_url(base: str, **params) -> str:
    qs = urllib.parse.urlencode(params)
    return f"{base}?{qs}" if qs else base


# In a real environment this would hit the network; here we just demo URL building.
url = build_url(
    "https://api.example.com/search",
    q="python learning",
    limit=5,
)
print("url:", url)

# Pretend response
pretend_body = json.dumps({"items": ["a", "b", "c"]}).encode("utf-8")
parsed = json.loads(pretend_body.decode("utf-8"))
print("items:", parsed["items"])


# The same thing with `requests` would be:
REQUESTS_EXAMPLE = '''
import requests
r = requests.get(
    "https://api.example.com/search",
    params={"q": "python", "limit": 5},
    headers={"User-Agent": "demo/1.0"},
    timeout=10,
)
r.raise_for_status()
data = r.json()
'''
print(REQUESTS_EXAMPLE.strip())

# Robust GET wrapper with retries
import time

def resilient_get(url, *, attempts=3, base_delay=0.1):
    for i in range(1, attempts + 1):
        try:
            return get_json(url, timeout=2)
        except (urllib.error.URLError, urllib.error.HTTPError) as err:
            if i == attempts:
                raise
            time.sleep(base_delay * (2 ** (i - 1)))

Three habits in one script:

1) `urlencode(params)` builds a safe query string — never concatenate manually.
2) Always set a timeout; stalled sockets hang the program otherwise.
3) Retry only specific network errors with exponential backoff.
4) The `requests` version is a few lines shorter but does the same thing.

Post a JSON body with the requests library.

# pip install requests
import requests

def create_user(base, name, email):
    r = requests.post(
        f"{base}/users",
        json={"name": name, "email": email},
        headers={"Authorization": "Bearer TOKEN"},
        timeout=10,
    )
    r.raise_for_status()
    return r.json()

# create_user("https://api.example.com", "ana", "ana@x.com")

URL building is deterministic.

import urllib.parse
qs = urllib.parse.urlencode({"q": "py", "n": 5})
assert qs in {"q=py&n=5", "n=5&q=py"}

Running prints:

url: https://api.example.com/search?q=python+learning&limit=5
items: ['a', 'b', 'c']
import requests
r = requests.get(
    "https://api.example.com/search",
    params={"q": "python", "limit": 5},
    headers={"User-Agent": "demo/1.0"},
    timeout=10,
)
r.raise_for_status()
data = r.json()