Iterators & Generators (yield, Lazy Evaluation, Data Pipelines) - Python Tutorial for Beginners #24

Iterators & Generators in Python: Lazy Evaluation and Data Pipelines

Understanding iterators and generators is key to writing efficient Python code that processes data lazily and elegantly. This tutorial covers the iterator protocol, creating custom iterators, generator functions using yield, and building memory-friendly data pipelines with generator chaining.

Code

# Custom iterator class implementing the iterator protocol
class Countdown:
  def __init__(self, start):
    self.current = start

  def __iter__(self):
    return self  # An iterator must return itself from __iter__

  def __next__(self):
    if self.current <= 0:
      raise StopIteration  # Signal iteration end
    else:
      self.current -= 1
      return self.current + 1

# Using the custom iterator
for number in Countdown(5):
  print(number)  # Outputs 5,4,3,2,1

# Generator function using yield for lazy evaluation
def fibonacci(n):
  a, b = 0, 1
  count = 0
  while count < n:
    yield a  # Yield next Fibonacci number
    a, b = b, a + b
    count += 1

# Consuming the generator
for num in fibonacci(7):
  print(num)  # Outputs Fibonacci sequence up to 7 numbers

# Generator pipeline example: filter even numbers and square them
def even_numbers(nums):
  for n in nums:
    if n % 2 == 0:
      yield n

def square(nums):
  for n in nums:
    yield n * n

numbers = range(10)
pipeline = square(even_numbers(numbers))

print(list(pipeline))  # Outputs [0, 4, 16, 36, 64]

Key Points

• Iterators implement __iter__ and __next__ to provide sequential access to data.
• Generators use yield to produce values lazily, saving memory compared to lists.
• Manual iteration with next() requires handling StopIteration to end loops.
• Generator pipelines chain multiple generators to process data step-by-step efficiently.
• yield from can delegate part of a generator’s operations to a sub-generator for cleaner code.