Generating Values on Demand

A generator is a function that produces values one at a time with the yield keyword. Each call to the function returns a generator object; calling next() on that object runs the function up to the next yield, produces the yielded value, and pauses. The next next() picks up where it left off. Generators are how Python unifies “compute on demand” with regular Python control flow.

The for loop knows how to drive a generator automatically: for value in gen(): calls next() for you and stops when the generator returns. That makes generators interchangeable with lists in almost every consumer: sum, max, "".join, the CSV writer, the JSON streamer — all of them take any iterable.

The big win is memory. A list of a million items owns a million slots in memory; a generator over the same sequence owns a single frame and a few local variables, no matter how many values it will eventually produce. For streaming log files, reading huge CSVs, or walking a file tree, that difference makes programs that simply wouldn't run.

Generators also model infinite streams naturally. while True: yield next_value is a perfectly reasonable generator body as long as the consumer decides when to stop (via islice, a condition, or simply breaking out of a loop). That is a kind of expressiveness plain lists cannot offer.

yield and the generator state machine

Each yield hands a value back to the caller and pauses the function. Local variables, the instruction pointer, and the execution frame stay alive until the next next(). From the generator's point of view, it's ordinary sequential code that occasionally pauses.

return inside a generator raises StopIteration and ends the stream. You rarely write it explicitly — falling off the end of the function does the same thing.

yield from and generator delegation

yield from other_iterable is shorthand for for x in other_iterable: yield x, but it also forwards send, throw, and return values for coroutine-style patterns. Use it whenever one generator wants to produce all the values of another.

For common patterns, the itertools module already ships a dozen generators (count, cycle, repeat, chain). Reach for those before rolling your own.

The generator vocabulary.

Generating Values on Demand code example

The script below builds a small set of generators and composes them like Lego blocks.

# Lesson: Generating Values on Demand
from itertools import islice
from typing import Iterator


def count_up(start: int = 0) -> Iterator[int]:
    n = start
    while True:
        yield n
        n += 1


def take_while_less(limit: int, it: Iterator[int]) -> Iterator[int]:
    for value in it:
        if value >= limit:
            return
        yield value


def windowed(n: int, it: Iterator):
    window = []
    for value in it:
        window.append(value)
        if len(window) > n:
            window.pop(0)
        if len(window) == n:
            yield tuple(window)


# Pipeline: count 0,1,2,... up to (not including) 10
pipeline = take_while_less(10, count_up(0))
print("first 10 from infinite:", list(pipeline))

# Sliding windows of size 3 over the first 7 integers
print("windows:", list(windowed(3, count_up(1))))  # would be infinite!
# Actually we need to bound it:
print("windows (bounded):", list(islice(windowed(3, count_up(1)), 5)))

# yield from to flatten a nested structure one level
def flatten_one(nested: list[list]):
    for inner in nested:
        yield from inner

print("flat:", list(flatten_one([[1, 2], [3, 4], [5]])))

# Generators are one-shot state machines
g = count_up(0)
print("  head:", next(g), next(g), next(g))  # advances the same generator

# Infinite is fine if the consumer stops early
for value in count_up(100):
    if value > 104:
        break
    print("  value:", value)

Key things to absorb:

1) A generator is defined with `def` + `yield`; calling it returns a paused state machine.
2) Generators compose like pipelines: each one consumes the next lazily.
3) `yield from` flattens one level of iteration cleanly.
4) Infinite generators are fine; the consumer is responsible for stopping.

Write a generator and use islice.

from itertools import islice

def triangles():
    n, total = 0, 0
    while True:
        n += 1
        total += n
        yield total

print(list(islice(triangles(), 5)))  # [1, 3, 6, 10, 15]

def batched(it, n):
    batch = []
    for v in it:
        batch.append(v)
        if len(batch) == n:
            yield batch
            batch = []
    if batch:
        yield batch

print(list(batched(range(7), 3)))

Verify the generator contract.

def g():
    yield 1
    yield 2
it = g()
assert next(it) == 1
assert next(it) == 2
try:
    next(it)
except StopIteration:
    pass
else:
    raise AssertionError("should have stopped")

Running the script prints:

first 10 from infinite: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
windows (bounded): [(1, 2, 3), (2, 3, 4), (3, 4, 5), (4, 5, 6), (5, 6, 7)]
flat: [1, 2, 3, 4, 5]
  head: 0 1 2
  value: 100
  value: 101
  value: 102
  value: 103
  value: 104