The observation is correct. python identifies that a function will be used as a generator by the existence of a yield statement anywhere, even unreachable, in the function's source code definition. The first call to the function only returns a generator. It does not execute any statements in the function.

Execution of the generator only happens when the next is invoked.

def fakegen(): # the presence of the yield statement causes a function to return a generator.
    # The function does not execute when invoked with  f=fakegen(). f is assigned the generator.
    # The first next(f) invoked causes the function to execute from the top. 
    # When it returns, rather than yields, the StopIteration exception condition is raised in the caller
    print("fakegen code executed")
    fakegen.wouldreturn="some value" #this is an attribute of the fakegen function, not any generator.
    return [42,"some other value"]  # will be the exception value message
    # the return raises the StopIteration exception condition.
    if False: yield 1 #unreachable

usage:

try:
    f=fakegen()
    print(f)
    fn=next(f)
except BaseException as ex:  # expect StopIteration 
    print("exception: ",ex.__class__.__name__,fn:=ex.value)
print("returned ",fn)

result

<generator object fakegen at 0x7f3660f09b40>

fakegen code executed

exception: StopIteration [42, 'some other value']

returned [42, 'some other value']

How does python know to [not execute the code immediately]?

The short answer is that Python has a compilation step, during which it scans the entirety of the function, spots it is a generator function, and so produces bytecode for a generator function that is different to the bytecode it would generate for a normal function. This means that at runtime, when a generator function is invoked, Python will know to create a generator object rather than to immediately start executing the body of the function.

The longer answer is that Python sits in a halfway house between strictly interpreted languages (eg. bash) and strictly compiled languages (eg. C). Instead, Python, like Java, has a compilation step that produces "bytecode". It is this bytecode that is interpreted by the Python virtual machine rather than the source code. Unlike Java, Python's compilation step is usually performed immediately before execution, rather than ahead of time. As such, some users may not even be aware that the compilation step exists. That is, when you do python mymodule.py, Python will both compile the module and then execute it. However, you can pre-compile Python code using the compileall module or the compile function eg.

# create the source code
echo "print('hello world')" > mymodule.py
# assert that no byte code file exists
[ -f __pycache__/mymodule.*.pyc ] || echo no byte code exists
# create the byte code file
python -m compileall mymodule.py
# check that the byte code file now exists
[ -f __pycache__/mymodule.*.pyc ] && echo byte code has been created
# directly execute the byte code rather than the source code
python __pycache__/mymodule.*.pyc

Outputs:

no byte code exists
Compiling 'mymodule.py'...
byte code has been created
hello world

Key differences between compiled and interpreted languages

In interpreted languages, each statement is evaluated one at a time. The interpreter will not consider statements after the current statement being evaluated. This means that a bit of source code could contain syntax errors, but still be partially executed by the interpreter eg.

# Bash source code file
echo start # will be executed normally
while for  # syntax error
echo end   # will *not* be executed because of prior syntax error

In compiled languages, a source unit must be compiled into a machine code before it can be executed. Any syntax errors will prevent compilation, and therefore nothing can be executed. eg.

// C source code file
#include <stdio.h>

// this function cannot be executed as the function that comes after it has a syntax
// error. So the source cannot be compiled successfully.
int main() {
    puts("start");
    puts("end");
    return 0;
}

void has_syntax_error() { // syntax error, missing a closing `}`

Python exhibits both traits of an interpreted language and a compiled language. Python is like an interpreted language because its programs cannot be executed without a Python interpreter. It is like a compiled language in that the source code for a single module is considered all at once, and a single syntax error will prevent any part of the code from being compiled/executed. Also, the source code is not directly executable, but must instead be compiled to a representation that the interpreter can understand. Sometimes this step is done entirely in memory and during the same process that will execute the program.

How Python knows not to start executing code in a generator function

It is during the compilation step that Python is able to detect the function is a generator. During compilation Python will scan the entirety of a function, it will spot the yield keyword and then produces bytecode for a generator function that is different to a normal function. When this function is invoked at runtime, Python will know to create a generator object rather than executing the body of the function. The implementation details can vary between different versions of Python, but the net effect remains the same.

At runtime, you can demonstrate that Python knows ahead of function invocation which functions are normal functions and which are generator functions. This is thanks to the initial compilation step. For example:

import inspect

def function(): pass
def generator(): yield

assert not inspect.isgeneratorfunction(function)
assert inspect.isgeneratorfunction(generator)

From the docs: Generator functions

...Such a function, when called, always returns an iterator object which can be used to execute the body of the function: calling the iterator’s iterator.next() method will cause the function to execute until it provides a value using the yield statement...

So, execution of the function is delayed until the first next call and then is controlled by yield and return. The function takes the place of the __next__ method in a hand-crafted iterator class. This makes sense because you don't want to calculate iterator values until you really need one.

If you want something to initialize right away, you should write your own class and put that in __init__.

Yes, that’s correct. In Python 3, when you call a generator function, its body does not execute immediately. The call only returns a generator object. Execution of the function starts only when you first call next() on the generator. Each subsequent next() resumes execution from the last yield until the function ends or raises StopIteration.

Python knows to do this because the presence of a yield statement marks the function as a generator. Internally, Python transforms the function into an iterator-like object whose __next__ method runs the function’s code lazily, producing values only when requested.

Code :

def fib():
    print("Executing the top of fib")
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

f = fib()          # Does not execute body yet
print("Called fib()")
print(next(f))     # Executes up to first yield
print(next(f))     # Continues execution