Python Generators
Tutorial 39 of 65 · pythondeck.com Python course
A generator function uses yield to produce values lazily, suspending and resuming execution. Generator expressions (x*x for x in ...) are like list comprehensions but evaluated on demand. They are memory efficient for large or infinite streams.
Generators produce a sequence of values lazily, one at a time, using yield. They save memory for large or infinite streams and can pipeline transformation stages. A generator function returns a generator iterator; consuming it drives execution until the next yield or StopIteration.
Generator expressions look like list comprehensions in parentheses: (x*x for x in range(n)). They are single-use—iterate once or store results in a list if you need replay.
yield suspends state; local variables persist between steps.
Generator expressions vs list comprehensions (memory vs eager list).
next(gen), for x in gen:, and exhaustion.
Sending values with gen.send() (advanced coroutine-style generators).
yield from to delegate to sub-generators.
Built-in itertools for combinatorial and infinite iterators.
Compose pipelines: read lines → strip → parse → filter, each stage a generator, without building intermediate lists. For I/O-bound streams, async generators (async def with async for) integrate with asyncio.
Generator cleanup runs finally in the generator body when the iterator is closed or garbage-collected—use for releasing resources tied to iteration.
When you need random access or len(), materialise to a list or use a sequence type instead of a generator.
Reusing a consumed generator expecting fresh values.
Building a huge list comprehension when a generator expression would suffice.
Mixing return values and yields in Python 3 (return with value ends iteration).
Not closing generators that hold files when breaking out of a loop early.
Using generators where simple lists are clearer and small.
Use generators for large or streaming data; profile before optimising micro-scripts.
Close or exhaust generators that manage resources; prefer context managers for files.
Combine with itertools for readable iterator algebra.
Document one-shot behaviour if callers might iterate twice.
Re-read the examples below with these ideas in mind; change variable names and inputs to match your own project.
The program below demonstrates generator function. Read the comments on each line, run the code, then change names or values to see how the output shifts.
# Example: Generator function
# Run in the REPL or save as a .py file and execute with python.
def countdown(n):
while n > 0:
yield n
n -= 1
print(list(countdown(5)))This sample walks through infinite stream in a small, runnable script. Paste it into the REPL or save it as a .py file before you continue to the next block.
# Example: Infinite stream
# Run in the REPL or save as a .py file and execute with python.
def naturals():
n = 1
while True:
yield n
n += 1
from itertools import islice
print(list(islice(naturals(), 10)))Here is a hands-on illustration of yield from. Follow the inline comments first; only then execute the snippet and compare the result with what you expected.
# Example: yield from
# Run in the REPL or save as a .py file and execute with python.
def chain(*iters):
for it in iters:
yield from it
print(list(chain([1,2], (3,4), "AB")))The program below demonstrates square generator. Read the comments on each line, run the code, then change names or values to see how the output shifts.
# yield turns a function into a generator — lazy iteration
def squares(n): # produce first n squares
for i in range(n): # counter
yield i * i # pause and return value
gen = squares(5) # generator object (not a list)
print(next(gen), next(gen)) # 0 1 — manual advance
print(list(gen)) # remaining: [4, 9, 16] after consuming 0,1
total = sum(squares(10)) # consume in aggregation
print(total) # sum of squares 0..9
print(hasattr(squares(1), "__iter__")) # generators are iterableThis sample walks through file line gen in a small, runnable script. Paste it into the REPL or save it as a .py file before you continue to the next block.
# Generators keep memory flat for large files
def lines(path): # stream lines
with open(path, encoding="utf-8") as fh: # open once
for line in fh: # iterate file
yield line.rstrip() # strip newline lazily
from pathlib import Path # ensure file exists for demo
p = Path("data/notes.txt"); p.parent.mkdir(exist_ok=True)
p.write_text("a\nb\n", encoding="utf-8") # tiny sample file
for row in lines(p): # consume generator
print("row:", row) # a then b
print(sum(1 for _ in lines(p))) # count lines lazilyContinue with these focused follow-up lessons on Python Generators: