Generators and Iterators

1The Iterator Protocol

Any object usable in a for loop implements the iterator protocol: __iter__ returns an iterator, and that iterator's __next__ produces one value at a time, raising StopIteration when exhausted.

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        self.current -= 1
        return self.current + 1

for n in Countdown(3):
    print(n)  # 3, 2, 1

A for loop is sugar for calling iter(obj) once, then calling next() repeatedly until StopIteration is raised — that exception is how the loop knows to stop, not an error condition.

2Generator Functions — yield Instead of Hand-Written Classes

Writing __iter__/__next__ by hand is verbose. A generator function — any function containing yield — does the same job automatically:

def countdown(start):
    while start > 0:
        yield start
        start -= 1

g = countdown(3)
print(next(g))  # 3
print(next(g))  # 2
print(next(g))  # 1
print(next(g))  # raises StopIteration

Calling countdown(3) does not run any of the function body. It returns a generator object immediately. The body only starts executing on the first next() call, and each yield pauses execution exactly where it is, handing a value back to the caller.

3Generators Are Iterators (But Not All Iterators Are Generators)

def countdown(start):
    while start > 0:
        yield start
        start -= 1

g = countdown(3)
print(iter(g) is g)  # True — a generator is its own iterator

Every generator object automatically has __iter__ (returning itself) and __next__ — it satisfies the iterator protocol from section 1 for free. This is why for n in countdown(3): works with no extra code: generators are simply the easiest way to write something that satisfies the iterator protocol.

4Generator Expressions

A generator expression is to a generator function what a list comprehension is to a loop — a compact, lazy equivalent:

squares_list = [x * x for x in range(1_000_000)]      # builds the ENTIRE list in memory now
squares_gen = (x * x for x in range(1_000_000))         # builds NOTHING yet; lazy

print(next(squares_gen))  # 0 — computed on demand
print(next(squares_gen))  # 1

Prefer a generator expression whenever the values are only consumed once, in order — e.g. sum(x * x for x in data) never materializes the intermediate list at all.

5send(), close(), and Two-Way Communication

yield is an expression, not just a statement — a generator can receive a value through send(), not only produce one:

def running_total():
    total = 0
    while True:
        amount = yield total
        total += amount

acc = running_total()
next(acc)             # prime the generator (advance to the first yield)
print(acc.send(10))    # 10
print(acc.send(5))     # 15
acc.close()             # raises GeneratorExit inside the generator, stopping it

next(acc) must run first to advance the generator to its first yield before send() has anywhere to deliver a value into. close() requests early termination by throwing GeneratorExit at the suspended yield point.

6yield from — Delegating to a Sub-Generator

def inner():
    yield 1
    yield 2

def outer():
    yield "start"
    yield from inner()   # transparently yields 1, then 2
    yield "end"

print(list(outer()))  # ['start', 1, 2, 'end']

yield from flattens one level of delegation: values yielded by inner() pass through outer() untouched, and outer()'s generator also correctly forwards send()/close()/exceptions down into inner() — something a manual for x in inner(): yield x loop does not do for send/throw.

7itertools — Building Blocks for Lazy Iteration

import itertools

evens = itertools.count(0, 2)             # infinite: 0, 2, 4, 6, ...
first_five = itertools.islice(evens, 5)     # take exactly 5, lazily
print(list(first_five))                      # [0, 2, 4, 6, 8]

print(list(itertools.chain([1, 2], [3, 4])))  # [1, 2, 3, 4] — lazily concatenates
print(list(itertools.zip_longest([1, 2], [1], fillvalue=0)))  # [(1, 1), (2, 0)]

itertools functions all follow the same discipline as generators: nothing is computed until something actually iterates the result, which is what makes itertools.count() (an infinite sequence) safe to create — it's only a problem if you fully consume it without a bound like islice.

⚠️ Common Pitfalls

1. Assuming a generator function's body runs when it's called — g = countdown(3) runs zero lines of countdown's body; nothing happens until the first next(g). A common bug is putting validation logic at the top of a generator function and expecting it to raise immediately — it only raises on first iteration.
2. Iterating a generator twice — a generator is exhausted after one full pass; list(g) a second time on the same object returns [], since there's no frame left to resume (see Under the Hood). If you need to iterate more than once, call the generator function again, or use a class/list instead.
3. Forgetting to next()-prime a generator before the first send() — send(value) on a freshly created, never-advanced generator raises TypeError, because there's no suspended yield expression yet to deliver value into.
4. Building an infinite generator without a limiting step — for x in itertools.count(): with no break/islice never terminates; always pair an unbounded generator with an explicit stopping condition.
5. Using a generator expression when you need to iterate more than once or need len() — a generator expression, like a generator function, is exhausted after one pass and has no __len__; use a list/tuple comprehension instead when either property is needed.

📋 Quick Reference

def gen():
    yield 1
    yield 2                    # generator function — lazy, one value per next()

g = gen()
next(g)                          # advances to the next yield, returns its value
g.send(value)                     # resumes, delivering `value` as the yield expression's result
g.close()                          # raises GeneratorExit at the suspended point
g.throw(SomeError)                  # raises SomeError at the suspended point

(x * x for x in range(10))            # generator expression — lazy equivalent of a list comp

def outer():
    yield from inner()                  # delegate to a sub-generator, forwarding send/throw too

itertools.count(start, step)              # infinite arithmetic sequence
itertools.islice(iterable, n)               # take first n, lazily
itertools.chain(a, b)                         # lazily concatenate iterables

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

Source: py_generators_and_iterators_theory.md · generated by tools/build_lessons.py