Python 101 - 10: Generators

Memory Problem

Normally, when you create a list, every single element is stored in your RAM (memory) at once. This is fine for 100 items, but if you are trying to process millions of items, sooner or later your program will run out of memory and crash.

Generators solve this by being lazy. They don’t store the whole collection. They only calculate the next item when you specifically ask for it.

The Trade-off

Pros: Highly memory efficient. You can work with massive data that is bigger than your RAM.
Cons: You can only go forward. Once you “consume” a value - it’s gone. You can’t go back to index [0] unless you restart the generator.

Using `yield`

A generator is defined like a normal function, but instead of using return (which ends the function), you use yield.

Think of return as “The End” and yield as “See you in a bit.”

# Traditional approach: Creates the WHOLE list first
def get_squares_list(nums):
    result = []
    for i in nums:
        result.append(i * i)
    return result 

# Generator approach: Creates one number at a time
def get_squares_gen(nums):
    for i in nums:
        yield (i * i) # Python "pauses" here until you ask for the next one

When you call get_squares_gen([1, 2, 3]), it just hands you a Generator Object, which is basically a promise to give you numbers when you’re ready. The loop hasn’t run yet!

Generator Expressions

Just like list comprehensions, you can create generators in a single line. Simply swap the square brackets [] for parentheses ().

# List comprehension: Lives in your RAM immediately
my_nums_list = [x * x for x in range(1000)]

# Generator expression: Sits quietly and uses almost zero RAM
my_nums_gen = (x * x for x in range(1000))

How to Consume a Generator

Since a generator is an iterable, you can loop through it using a for loop. If you absolutely need to turn it back into a list (and you have enough memory), you can pass it to the list() function.

my_nums_gen = (x * x for x in range(1000))

# Loop through it (most common use)
# Note: You can only do this ONCE. 
for num in my_nums:
    print(num)
# The generator is now empty.


# Convert to list (Only if memory allows!)
my_nums_gen = (x * x for x in range(5)) # Restart it
print(list(my_nums)) # Output: [0, 1, 4, 9, 16]

This is exactly how modern AI works. When you ask Gemini or ChatGPT a question, the model doesn’t wait to generate the entire page before showing it to you. It yields one word (token) at a time so you can start reading immediately.

def ai_response_stream():
    yield "Hello"
    yield "how"
    yield "can"
    yield "I"
    yield "help?"

# We "consume" the generator one by one as they are produced
for word in ai_response_stream():
    print(word, end=" ")
# Output: Hello how can I help?

Lists vs. Generators

Feature	List (`[]`)	Generator (`yield`)
Creation	All values created and stored in RAM immediately.	Values are created on demand (lazily).
Memory	High	Very low
Access	Can be indexed (`my_list[3]`) and accessed repeatedly.	Can only be iterated over once, from start to finish.
Best Use	When you need to access items multiple times or out of order.	For large datasets, file streams, or sequences where you only look at each item once.