Are you using a reliable pivot, like jmp esp or are you jumping to a fixed address on the stack? If you're using the latter then your addresses are likely misaligned, and you should consider using a reliable pivot instead. If the former you'll need to inspect the crash by enabling coredumps and loding the core into a debugger.

The Shell Command

When you execute:

./vulnerable $(cat payload)

in the shell, the command is interpreted by the shell itself. Here’s what happens step-by-step:

1. $(cat payload) is a shell substitution. The shell executes cat payload and replaces $(cat payload) with the output of that command.
2. As a result, the ./vulnerable program is invoked with the contents of the payload file directly as its argument.

The Python Command

When you try the equivalent in Python:

import os
os.system("./vulnerable $(cat payload)")

the command fails with a segmentation fault. The reason for this lies in how the Python os.system function interacts with the shell and the substitution process. Let’s break this down:

1. Shell Substitution Issue:

The os.system function passes the entire string ./vulnerable $(cat payload) to a subshell. However, this subshell does not perform the substitution correctly in this context. Instead of substituting the output of cat payload into the command, it tries to pass the string $(cat payload) directly to ./vulnerable.

2. Argument Handling:

Unlike a direct shell execution, the Python os.system command does not process the $(...) syntax for command substitution. It requires explicit handling to achieve the same result.

Correct Approach

Here's an example how you could do it within Python:

import os

with open('payload', 'r') as file:
    payload = file.read().strip()

command = f"./vulnerable {payload}"
os.system(command)

In summary, the segmentation fault occurs because the command substitution $(cat payload) isn't processed correctly by os.system in Python. By reading the payload content into a variable and constructing the command string manually, you can achieve the intended behavior without encountering segmentation faults.