PYTHON / ITERATORS, GENERATORS, AND COMPREHENSIONS
Generator expressions and laziness
Write generator expressions and predict exactly when each element is computed, including the one part evaluated eagerly and why a second pass yields nothing.
What you will learn
- Build a generator with parentheses and know its body has not run yet
- Trace when each element is computed by interleaving next() with print side effects
- Recall that only the outermost iterable is evaluated at creation time
- Materialise with tuple() or list() when you need a second pass, len, or indexing
Understanding Generator expressions and laziness
A generator expression looks like a list comprehension with parentheses instead of brackets, but the resemblance stops at the syntax. `[f(x) for x in xs]` runs the whole loop and hands you a list; `(f(x) for x in xs)` compiles the loop into a hidden function, calls it, and hands you the paused generator object it returns. Nothing in the body has executed, no results exist anywhere, and `f` has not been called once. The right mental model is a recipe plus a bookmark, not a container.
Every `next()` on that object resumes the hidden loop, runs it just far enough to produce one value, and freezes again at the point where the value left. That is what laziness means concretely: the filter condition and the output expression run interleaved with whatever the consumer is doing, so a `print` inside the body appears in the middle of the consuming loop rather than all at once beforehand. It also means a consumer that stops early, like `next()`, `any()`, or `itertools.islice`, makes the generator do proportionally less work, which is why a generator expression can safely wrap an infinite source.
One piece is not lazy. The iterable of the outermost `for` clause is evaluated and passed to `iter()` the moment the generator expression is created, so a misspelled name there raises immediately, and later rebinding that name has no effect on the generator, which already holds the iterator. Everything else — inner loops, `if` conditions, the output expression — waits. Finally, a generator is an iterator: it has no length, no indexing, and it is single-pass, so once `sum()` has drained it, a second `sum()` quietly returns 0 instead of complaining. As a convenience, when a generator expression is the only argument to a call you may drop its parentheses: `sum(len(w) for w in words)`.
def costly(n):
print(f" computing {n}")
return n * n
squares = (costly(n) for n in [1, 2, 3])
print("generator created:", type(squares).__name__)
print("first:", next(squares))
print("second:", next(squares))
print("rest:", list(squares))
print("again:", list(squares))A generator expression creates a paused computation rather than a sequence: apart from the outermost iterable, nothing is evaluated until a consumer asks for the next value.
Worked examples
The outermost iterable is captured immediately
Shows that mutating the source list is visible to a generator expression but rebinding its name is not.
nums = [1, 2, 3]
gen = (n * 10 for n in nums)
nums.append(4)
print(list(gen))
nums = [99]
gen2 = (n * 10 for n in nums)
nums = [1, 2, 3]
print(list(gen2))Example explained
Line 1Creating `gen` calls `iter(nums)` at once and stores a list iterator over that exact object.
Line 2`nums.append(4)` mutates the object the stored iterator walks, so 40 is produced too.
Line 3`gen2` captures the iterator over `[99]`; the later `nums = [1, 2, 3]` only rebinds a name.
Line 4The body (`n * 10`) is still lazy — it runs during `list(...)`, not at creation.
Laziness lets you stop early
A generator expression searching a list stops at the first match, and one over an infinite counter pulls only what is asked for.
import itertools
lines = ["ok 1", "ok 2", "ERROR disk full", "ok 3"]
first = next((i for i, ln in enumerate(lines) if ln.startswith("ERROR")), None)
print("first error at index:", first)
counter = itertools.count(1)
multiples = (n for n in counter if n % 7 == 0)
print("first three:", [next(multiples) for _ in range(3)])
print("raw counter now yields:", next(counter))Example explained
Line 1`next(genexp, None)` gives a scan-and-stop search: `lines[3]` is never tested at all.
Line 2Wrapping the infinite `count(1)` is free because building the generator pulls nothing.
Line 3Three pulls advanced `counter` only as far as 21, so the next raw value is 22.
Line 4A list comprehension over the same filter would never finish.
One pass only
Demonstrates the silent failure of iterating an exhausted generator expression twice and the explicit fix.
readings = [3.5, 4.0, 2.5, 6.0]
scaled = (r * 2 for r in readings)
print("total:", sum(scaled))
print("count after exhaustion:", sum(1 for _ in scaled))
scaled = tuple(r * 2 for r in readings)
print("total:", sum(scaled), "count:", len(scaled))Example explained
Line 1`sum(scaled)` drains the generator; the object survives but is permanently exhausted.
Line 2The second pass iterates zero times and returns 0 — no exception marks the bug.
Line 3`tuple(...)` runs the same recipe once and stores the results, so it can be measured and re-scanned.
Line 4Note the dropped inner parentheses: the generator expression is the sole call argument.
Important notes
Because the body runs during consumption, a traceback from a generator expression points at the line that consumed it, not the line that created it; temporarily wrapping the expression in list() moves the failure back next to the code that caused it.
The loop variable lives in the generator's own scope and never leaks into the surrounding namespace, but the outermost iterable expression is evaluated in the enclosing scope at creation time.
Common mistakes
Passing the same generator expression to two consumers, e.g. sum() then len-by-counting: the second sees an exhausted iterator and returns 0 or False with no error, producing a wrong answer instead of a crash.
Assuming creation evaluates nothing: `(x for x in typo_name)` raises NameError on the assignment line, while errors inside the body only surface later at the list()/sum() call site.
Treating the object as a sequence — `len(gen)`, `gen[0]`, or printing it — which gives a TypeError or `<generator object <genexpr> at 0x...>` instead of values.
Try it yourself
Change, predict, then run
Build a generator expression over range(1, 51) that yields n ** 3 only for values of n divisible by 4, print the first two values with next(), then print list() of that same object twice and explain the difference between the two results.
Open the Python workspaceCheck your understanding
What does this print? data = [1, 2, 3] g = (x * 2 for x in data) data = [10, 20] print(sum(g))
- 12
- 60
- 0
- A TypeError, because a generator has no length for sum() to use
Show answer
The outermost iterable is evaluated when the generator expression is created, so `g` holds an iterator over the original [1, 2, 3] object; sum() therefore adds 2 + 4 + 6 = 12. Answer 60 assumes the name `data` is looked up lazily at consumption time, but only the body (`x * 2`) is deferred, not the source iterable.