0

I was trying to run multiple machine learning experiments simultaneously to save time. My OS is Fedora 36, and I use emacs. To automate the experiment I used runs.py

import subprocess

subprocess.run("python train.py python train.py python train.py python train.py", shell=True)

where train.py trains the neural network and I am just using different seeds.

Then, I open multiple shells inside emacs where in each I run python runs.py. I can do at most 5 shells, when I open a sixth one and run python runs.py, then emacs and the terminal close with no error. I am sure there is a log somewhere, but I don't know where. Why could this happen? train.py uses the GPU, but there is enough memory, there is also enough RAM to run more, so I have no idea what's causing this.

I'd appreciate any help. I know the information I provided might not be enough, but I am not sure what else to say that could be potentially relevant.

Improve this question
6
  • Try opening emacs from the command line (your terminal) and run the programs. All logs (stdout and stderr) should be printed in the terminal. Thus when emacs crashes you may see what error is causing a crash. Commented Feb 2, 2023 at 6:25
  • I open it from the command line: 'emacs -nw'. But I can't see any logs because when it crashes, it closes the terminal automatically. Commented Feb 2, 2023 at 6:38
  • Oh I see, it crashes all process. Maybe redirecting output to a file might work: emacs -nw &>/home/user/somefile or stderr only: emacs -nw 2>/home/user/errfile Commented Feb 2, 2023 at 6:41
  • 1
    I'm thinking your process may be killed by another one. Do you have oom killer? Try running: systemctl status systemd-oomd Commented Feb 2, 2023 at 17:02
  • 1
    So, after all, it seems it might be a memory problem. I get Killed due to memory used. Thanks for your help! Now I know what needs to be fixed. My program saves intermediate results as it runs, so at some point I run out of memory. It wasn't obvious to me because at first there is enough memory, but it gets occupied quickly. Commented Feb 2, 2023 at 17:32

0

Reset to default

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.