Python Multiprocessing
Tutorial 44 of 65 · pythondeck.com Python course
multiprocessing spawns separate OS processes, bypassing the GIL for CPU-bound work. Communicate through queues, pipes or shared memory. concurrent.futures.ProcessPoolExecutor provides a high-level API.
Multiprocessing spawns separate Python interpreters, sidestepping the GIL for CPU-bound parallelism. Each process has its own memory space—sharing requires multiprocessing.Queue, pipes, or sharedctypes, not plain global variables. Startup cost and pickling overhead matter for small tasks.
The if __name__ == "__main__": guard is required on Windows (and best practice everywhere) when using the spawn start method to avoid recursive process creation.
Process, Pool, and ProcessPoolExecutor.
Start methods: spawn (default on Windows/macOS), fork (Linux), forkserver.
Queue and Pipe for inter-process communication.
Picklability requirement for target functions and arguments.
CPU count: os.cpu_count() and sensible max_workers.
Shared memory and Manager for complex shared structures (use sparingly).
Pool.map distributes chunks to workers—great for embarrassingly parallel numeric loops. For fine-grained tasks, batch work to amortise pickling. Logging from child processes needs QueueHandler patterns or file redirection.
Fork copy-on-write on Linux is fast but unsafe with threads in the parent at fork time—prefer spawn for mixed threading/multiprocessing apps.
Alternative: run calculations in NumPy, Cython, or an external worker service when operational complexity of processes is too high.
Measure serialisation cost: tiny tasks spent mostly pickling arguments will run slower with processes than a single-threaded loop.
Missing __main__ guard causing spawn errors on Windows.
Trying to share complex objects without pickling or using Manager overhead blindly.
Oversubscribing CPUs with more processes than cores without measuring.
Mixing fork with threaded parents leading to deadlocks or corrupted state.
Ignoring that lambdas and nested functions may not pickle on all setups.
Protect entry points with if __name__ == "__main__":.
Use ProcessPoolExecutor with batchable, picklable top-level functions.
Size pools near cpu_count(); profile end-to-end, not micro-benchmarks alone.
Prefer queues for IPC; avoid shared mutable globals across processes.
Re-read the examples below with these ideas in mind; change variable names and inputs to match your own project.
The program below demonstrates process. Read the comments on each line, run the code, then change names or values to see how the output shifts.
# Example: Process
# Run in the REPL or save as a .py file and execute with python.
from multiprocessing import Process
def worker(n):
print("worker", n)
if __name__ == "__main__":
ps = [Process(target=worker, args=(i,)) for i in range(3)]
for p in ps: p.start()
for p in ps: p.join()This sample walks through pool / map in a small, runnable script. Paste it into the REPL or save it as a .py file before you continue to the next block.
# Example: Pool / map
# Run in the REPL or save as a .py file and execute with python.
from multiprocessing import Pool
def heavy(n):
return sum(i*i for i in range(n))
if __name__ == "__main__":
with Pool(4) as pool:
print(pool.map(heavy, [10_000, 20_000, 30_000, 40_000]))Here is a hands-on illustration of processpoolexecutor. Follow the inline comments first; only then execute the snippet and compare the result with what you expected.
# Example: ProcessPoolExecutor
# Run in the REPL or save as a .py file and execute with python.
from concurrent.futures import ProcessPoolExecutor
def sq(x): return x*x
if __name__ == "__main__":
with ProcessPoolExecutor() as ex:
print(list(ex.map(sq, range(8))))The program below demonstrates process pool. Read the comments on each line, run the code, then change names or values to see how the output shifts.
# Processes bypass GIL for CPU-bound tasks
from multiprocessing import Pool # process pool
def square(n): # pure function for workers
return n * n # compute
if __name__ == "__main__": # guard required on Windows spawn
with Pool(4) as pool: # four worker processes
results = pool.map(square, range(8)) # parallel map
print(results) # [0,1,4,9,16,25,36,49]
print(sum(results)) # aggregateThis sample walks through queue ipc in a small, runnable script. Paste it into the REPL or save it as a .py file before you continue to the next block.
# multiprocessing.Queue passes picklable objects between processes
from multiprocessing import Process, Queue # IPC primitives
def worker(q, n): # child process target
q.put(n * n) # send result to parent
if __name__ == "__main__": # spawn guard
q = Queue() # shared queue
ps = [Process(target=worker, args=(q, i)) for i in range(3)] # children
for p in ps: p.start() # start
for p in ps: p.join() # wait
print([q.get() for _ in ps]) # collect squares