Python Threading

Tutorial 43 of 65 · pythondeck.com Python course

The threading module enables concurrent execution of I/O-bound tasks. Due to the GIL, threads are not ideal for CPU-bound work; use multiprocessing instead. Use Lock, Event, Queue for synchronisation.

Threads run multiple flows of control in one process, sharing memory. CPython's GIL limits parallel CPU-bound Python bytecode, but threads still help for I/O-bound work (network, disk) where threads wait on the OS. Use threading.Thread or pools (ThreadPoolExecutor) for concurrent blocking calls.

Synchronisation primitives—Lock, RLock, Semaphore, Event—prevent race conditions on shared mutable state. Prefer immutable data and message passing when possible.

Starting threads: Thread(target=fn, args=...) and start()/join().

The GIL and why CPU-bound threads do not scale in pure Python.

threading.Lock and the with lock: pattern.

concurrent.futures.ThreadPoolExecutor for task pools.

Thread-safe queues: queue.Queue for producer-consumer patterns.

Daemon threads vs non-daemon: process exit waits for non-daemons.

Race conditions are subtle: check-then-act on shared dicts without locks loses updates. Use queue.Queue to hand off work items instead of sharing lists protected by ad-hoc flags.

For CPU parallelism in Python, use multiprocessing, subprocess, NumPy calling C code, or native extensions. asyncio handles many concurrent I/O tasks with one thread when APIs are async.

Debugging threads: log thread name, use faulthandler, and avoid deadlocks by acquiring locks in a consistent global order.

Main-thread rules apply to GUI frameworks and some libraries: schedule UI updates on the main thread even when worker threads compute in the background.

Expecting linear speedup on CPU-bound pure Python with many threads.

Sharing mutable globals without locks or queues.

Deadlocks from nested lock acquisition in different orders.

Calling GUI toolkit methods from non-main threads without platform rules.

Creating unbounded threads instead of a bounded executor pool.

Use threads for I/O-bound concurrency; use processes or native code for CPU-bound.

Pass work via queue.Queue; minimise shared mutable state.

Bound concurrency with ThreadPoolExecutor(max_workers=...).

Consider asyncio when libraries support async I/O end-to-end.

Re-read the examples below with these ideas in mind; change variable names and inputs to match your own project.

The program below demonstrates thread. Read the comments on each line, run the code, then change names or values to see how the output shifts.

# Example: Thread
# Run in the REPL or save as a .py file and execute with python.
import threading, time
def worker(name):
    for i in range(3):
        print(name, i)
        time.sleep(0.1)

ts = [threading.Thread(target=worker, args=(n,)) for n in ("A", "B")]
for t in ts: t.start()
for t in ts: t.join()

This sample walks through lock in a small, runnable script. Paste it into the REPL or save it as a .py file before you continue to the next block.

# Example: Lock
# Run in the REPL or save as a .py file and execute with python.
import threading
lock = threading.Lock()
counter = 0

def bump():
    global counter
    for _ in range(100_000):
        with lock:
            counter += 1

ts = [threading.Thread(target=bump) for _ in range(4)]
for t in ts: t.start()
for t in ts: t.join()
print(counter)

Here is a hands-on illustration of threadpoolexecutor. Follow the inline comments first; only then execute the snippet and compare the result with what you expected.

# Example: ThreadPoolExecutor
# Run in the REPL or save as a .py file and execute with python.
from concurrent.futures import ThreadPoolExecutor
import urllib.request

urls = ["https://example.com"] * 4
with ThreadPoolExecutor(max_workers=4) as ex:
    sizes = list(ex.map(lambda u: len(urllib.request.urlopen(u).read()), urls))
print(sizes)

The program below demonstrates thread pool. Read the comments on each line, run the code, then change names or values to see how the output shifts.

# Threads suit I/O-bound work; GIL limits CPU parallelism
import threading, time  # threading primitives

def fetch(name, delay):  # simulated I/O
    time.sleep(delay)  # block thread
    print(f"{name} done")  # completion log

threads = []  # track thread objects
for label, wait in [("A", 0.2), ("B", 0.1)]:  # two tasks
    t = threading.Thread(target=fetch, args=(label, wait))  # construct
    t.start(); threads.append(t)  # start concurrently
for t in threads:  # join all
    t.join()  # wait for finish
print("all finished")  # after joins

This sample walks through thread lock in a small, runnable script. Paste it into the REPL or save it as a .py file before you continue to the next block.

# Locks protect shared mutable state across threads
import threading  # locks live here
counter = 0  # shared integer
lock = threading.Lock()  # mutual exclusion

def inc():  # increment safely
    global counter  # module global
    with lock:  # acquire/release automatically
        counter += 1  # critical section

workers = [threading.Thread(target=inc) for _ in range(100)]  # many threads
for w in workers: w.start()  # start all
for w in workers: w.join()  # wait
print(counter)  # 100 when lock used correctly

« Python Context Managers All tutorials Python Multiprocessing »