4,942 questions
0
votes
0
answers
29
views
flask-scoketio: emitting from an external process without a message queue
I am developing a project using flask, flask-socketio, and the Multiprocessing python libraries.
My site runs on localhost (127.0.0.1:5000).
The goal is to update the html template every second from ...
- 1
1
vote
0
answers
77
views
Parent process unexpectedly exits during debugging when child receives SIGTERM (works fine outside debugger)
I'm experiencing an issue where the parent Python process terminates unexpectedly when debugging, but only when a child process is sent SIGTERM. The same code works perfectly when run normally (...
- 35
3
votes
1
answer
132
views
Isolation of a custom multiprocessing manager and how to update internal state
I try to use a custom multiprocessing manager, mostly following the example from the docs. The main difference is that my class updates internal state. It looks like this:
class IdIndex:
def ...
- 15.7k
3
votes
1
answer
56
views
Python multiprocessing: Responsibility for terminating daemons
I am using pythons multiprocessing library to create some subprocesses. I implemented a way to terminate my child-processes gracefully.
To terminate them even if my main-process has a crash, I made ...
- 43
0
votes
1
answer
44
views
How to Get the Return Value of a Function in Python Multithreading [duplicate]
I’m writing a Python web crawler, and I’m using multithreading to execute a function. The function returns a link when it finishes.
Here’s the general structure of the function:
def download_link(...
- 11
2
votes
2
answers
74
views
How to use multiprocessing module with interdependent tasks?
The existing examples of the multiprocessing-module assume, the tasks are independent, but my situation is "hairier".
I need to process several hundred thousands of widgets. These are ...
- 4,256
0
votes
1
answer
28
views
Python: problem with multiprocessing.Pool and deleting files (WindowsError: [Error 32])
I'm writing code in Python 2.7 (bigger framework compatibility reasons) that does the following:
Takes an ID_something.py file, opens it, reads the lines from it, then closes the file.
Creates an ...
- 13
1
vote
0
answers
50
views
Multiprocessing pool with stop flag
I am trying to implement a stop flag that gracefully prevents new parallel jobs from being started.
Specifically, I am running a large number of simulations that each takes a few to many hours; in ...
- 468
1
vote
0
answers
34
views
Why the cycle running in main proccess won't stop after being class state modified from another proccess? [duplicate]
Consider this sample:
import multiprocessing as mp
import time
class CycleThatDoesntStop:
shouldrun = True
def run(self):
while self.shouldrun:
print("im alive", self....
- 53
0
votes
1
answer
79
views
How to concurrently remove lines from a file in Python?
I have a cluster of compute nodes, each node with many CPUs. I want them to execute commands located in a file, one command per line. The drive where the command file is located is mounted on all ...
- 2,842
2
votes
1
answer
55
views
Running a python script in the background, then passing an event to end it after a given amount of time
In short, I'm trying to run a python script from another python script but in the background, allowing the parent script to continue executing, and then to pass a CTRL_C_EVENT to the child script to ...
- 29
0
votes
1
answer
73
views
How should CPU time be computed for calculations parallelized with the multiprocessing module?
I am trying to measure the processing time, or CPU time, of a CPU-intensive computation that has been parallelized with multiprocessing. However, simply bookending the parallelization of the ...
- 427
1
vote
1
answer
79
views
What are the basic rules for propagating lists and dictionaries across processes using the Manager of the multiprocessing module?
I am trying to use the multiprocessing module to parallelize a CPU-intensive piece code over multiple cores. This module looks terrific in several respects, but when I try to pass lists and ...
- 427
3
votes
1
answer
186
views
PyTorch Sharing CUDA tensors
I have a question considering sharing gpu-tensors between processes using the torch.multiprocessing module. Here is a minimal example:
import torch
import torch.multiprocessing as mp
from torch.nn ...
- 31
1
vote
1
answer
64
views
Static analysis with multiprocessing BaseManager, how can I avoid errors for functions which are added at runtime?
I have a problem when i register a class at runtime on my BaseManager. It's pretty obvious that static analysis cannot tell what that function is, but how can I avoid it?
import multiprocessing....
- 143
0
votes
0
answers
86
views
Filter data into binary classes on GPU
I have a ML problem where I want to leverage the power of Support Vector Classifiers (SVC) or any other 2-class classifier and compare them to my NN models.
The probelm is, that binary classifiers are ...
- 11
0
votes
1
answer
55
views
QueueListener breaks logging level filter?
It appears that logging.handlers.QueueHandler/.QueueListener is breaking the .setLevel of an attached logging.FileHandler in Python 3.12.9 on Windows.
Running the following minimal example results in ...
- 3,507
0
votes
0
answers
32
views
Significant overhead when calling DataLoader for a dataset within FastAPI endpoint using multiple processing
I am calling a machine learning model for a dataset that I have loaded using torch DataLoader:
class FilesDataset():
def __init__(self, path):
file_paths = glob.glob(os.path.join(path, "*....
- 367
1
vote
0
answers
55
views
Is there a reason not to replace the default logging.Handler lock with a multiprocessing.RLock to synchronize multiprocess logging
I've got some code that, for reasons not germane to the problem at hand:
Must write very large log messages
Must write them from multiple multiprocessing worker processes
Must not interleave the logs ...
- 158k
0
votes
1
answer
83
views
Python code coverage not working for multiprocessing inside site-packages
I have a simple code packages in a module/folder "src"
File sample.py
import multiprocessing
def f(x):
return x*x
def big_run(n):
with multiprocessing.Pool(5) as p:
p.map(f,...
2
votes
3
answers
122
views
Multiprocessing with SciPy Optimize
Question: Does scipy.optimize have minimizing functions that can divide their workload among multiple processes to save time? If so, where can I find the documentation?
I've looked a fair amount ...
- 23
0
votes
1
answer
79
views
How to profile code where the time is spent in imap_unordered?
This code (it will only work in Linux) makes a 100MB numpy array and then runs imap_unordered where it in fact does no computation.
It runs slowly and consistently. It outputs a . each time the square ...
- 21.5k
0
votes
0
answers
69
views
Python multiprocessing.Pool hangs under debugpy launcher on Windows ('spawn'), even with freeze_support
I'm encountering an issue where my Python script using multiprocessing.Pool hangs during startup, but only when launched via the debugpy debugger/launcher integrated with VS Code on Windows. When run ...
1
vote
1
answer
175
views
Why doesn't multiprocessing.Process.start() in Python guarantee that the process has started?
Here is a code to demo my question:
from multiprocessing import Process
def worker():
print("Worker running")
if __name__ == "__main__":
p = Process(target=worker)
p....
- 387
2
votes
1
answer
75
views
Python multiprocessing.Queue strange behaviour
Hi I'm observing a strange behaviour with python multiprocessing Queue object.
My environment:
OS: Windows 10
python: 3.13.1
but I observed the same with:
OS: Windows 10
python: 3.12.7
and:
OS: ...
- 642
0
votes
2
answers
96
views
Python Multiprocessing Pool Queue Works in Functional Code but not in OOP
I am learning multiprocessing in Python and am trying to incorporate a worker pool for managing downloads. I have deduced my queue issue down to something with OOP, but I don't know what it is. The ...
0
votes
0
answers
103
views
Pathos multiprocessing map not completing on Linux machine, works on Mac M2
I am using the pathos multiprocessing to parallelize a gradient calculation that is embarrassingly parallel, using finite difference.
Below is a high level example of how it is set up,
`
...
class ...
- 1
0
votes
0
answers
21
views
python ProcessPoolExecuter does not run imported method
I have almost the same problem as the person in this post: python ProcessPoolExecutor do not work when in function.
However in my code the funtion I am trying to call is an imported function located ...
0
votes
0
answers
46
views
Python database parallelization problem. My code gives errors
I have issues regarding my own implementation of a parellized database using the TinyDB and multiprocessing libs in python. It always give errors, such as this one:
"c:\Users\1765536\AppData\...
1
vote
1
answer
63
views
Multi-processing copy-on-write: 4 workers but only double the size
I'm running an experiment on copy-on-write mechanism on Python Multiprocessing.
I created a large file of 10GB and load the file into large_object in main.
file_path = 'dummy_large_file.bin'
try:
...
- 45
0
votes
1
answer
82
views
Process hangs when multiprocessing with XGBoost model batch prediction
Here's a batch prediction case using multiprocessing. Steps:
After with mp.Pool(processes=num_processes) as pool, there's a with Dataset(dataset_code) as data in the main process using websocket to ...
- 37
1
vote
1
answer
44
views
aiomultiprocess: Worker creating two processes instead of one
Starting with one of the two examples from the User Guide ( https://aiomultiprocess.omnilib.dev/en/latest/guide.html ) I started my testing with an own variation:
import asyncio
import psutil
import ...
- 103
1
vote
0
answers
40
views
Multiprocessing logs with S3 log Handler in Python
I am trying to push logs for a multiprocessing job into ECS S3. Following is my code snippet:
logger.py
import logging
from S3_log_handler import S3LogHandler
def setup_logger():
# Set up the ...
- 79
2
votes
1
answer
437
views
Python multiprocessing.Process hangs when large PyTorch tensors are initialised in both processes
Why does the code shown below either finish normally or hang depending on which lines are commented/uncommented, as described in the table below?
Summary of table: if I initialise sufficiently large ...
- 1,960
2
votes
1
answer
103
views
Terminating pool early intermittently crashes when return size is large
I have a script that's trying to analyse some images in parellel.
For some reason, I get intermittent crashes if the func passed to the pool returns variable sized data. This is only if I try to exit ...
- 127
-4
votes
2
answers
90
views
Why is there not much speedup when using parallel processing during bubble sort? [closed]
I want to compare the effect of multiprocessing for bubble sort.
Let us consider first the original one without multiprocessing:
import multiprocessing
import random
import time
import numpy as np
...
0
votes
1
answer
324
views
XIO: fatal IO error 0 (Success) on X server ":0" Kivy multiprocessing
Hi I am trying to create a program that when run, two windows open at the same time, from the same app. For this multihtreading is needed, but it seems I get some strange errors:
XIO: fatal IO error ...
0
votes
2
answers
82
views
Logger does not inherit config from parent process
Consider the following minimal setup:
/mymodule
├── __init__.py
├── main.py
└── worker.py
__init__.py is empty
main.py:
import sys
import logging
import multiprocessing
from test.worker import ...
- 336
0
votes
0
answers
77
views
How to implement QueueHandler and QueueListener inside flask factory
I have a flask application delivered by gunicorn that spawns multiple threads and processes from itself during the request. The problem is that when using the standard app.logger, some of the children ...
- 772
2
votes
2
answers
130
views
When using the multiprocessing pool without a context manager, is it necessary to call both close() and terminate()?
The python docs seem to suggest that it's required to call both close() and terminate() when using multiprocessing.pool.Pool without a context manager.
Warning - multiprocessing.pool objects have ...
- 23
0
votes
2
answers
89
views
How do I preserve and reconcile mutable object state across parallel tasks, when pickling breaks references in Python multiprocessing?
We're working on a concurrency problem where a single "Sample" object has multiple dependent tasks that must be executed in stages. For instance, stage 1 has tasks (1a, 1b), stage 2 has ...
- 142
2
votes
1
answer
91
views
Python Object detection multiprocessing speed issue
I'm trying to implement a multiprocessing version of object detection (video source can be both camera or video) with YOLO model of ultralytics.
I implemented a Queue where to add frames and a process ...
0
votes
0
answers
39
views
What is the best way to extract information from multiple gz file into single csv file?
I am working with openalex data, which are multiple gz files. I need to translate json-like data into csv format.
For example:
one line of the original file is like:
{"id": "https://...
2
votes
2
answers
317
views
Efficient parsing and processing of millions of json objects in Python
I have some working code that I need to improve the run time on dramatically and I am pretty lost. Essentially, I will get zip folders containing tens of thousands of json files, each containing ...
- 813
0
votes
0
answers
102
views
Azure authentication failure in spawned processes from multiprocessing pool
I have a very specific issue with the combination of multiprocessing + azure authentication and could not find a valuable open issue, so here I go:
I am working on a python program that uses ...
- 503
0
votes
3
answers
60
views
python ignores pool.join() and continues to run the following code?
import multiprocessing
import time
def task_function(x):
print(f"Processing {x} in process {multiprocessing.current_process().name}")
time.sleep(1)
def main():
data = [1, 2, 3, ...
- 168
0
votes
0
answers
107
views
How to use multiple processes of Selenium with one user-data-dir?
I'm trying to use Selenium to scrap and do some actions on a list of URLs.
The list itself contains some thousands of URLs.
So, I'm trying to use a pool of processes to manage multiple instances of ...
- 51
1
vote
0
answers
61
views
Python map makes a copy? I need to map by reference
I'm making a script that calls the same function several times with different parameters, then save the results in a spreadsheet (excel). To do this, I'm using multiprocessing.Pool, and I need to save ...
- 100
0
votes
1
answer
181
views
Type hints for a function (multiprocessing)
The following function creates an instance of a class that can be shared between processes:
def create_shared_object (constructor: ?) -> ?:
from multiprocessing.managers import BaseManager;
...
- 31
0
votes
0
answers
66
views
DistNetworkError when using multiprocessing_context parameter in pytorch dataloader
Because of some special reasons I want to use spawn method to create worker in DataLoader of Pytorch, this is demo:
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data ...
- 1,115