Python read image to numpy array with multiprocessing

Question

I am working on a python program which read a lot of images in batches (let's say 500 images) and store it in a numpy array.

Now it's single thread, and IO is very fast, the part which take a lot of time is creating numpy array and doing something on it.

By using multiprocessing module, I am able to read and create the array in other process. But I am having problem let the main thread access those data.

I have tried:

1: Using multiprocessing.queues: Very slow, I believe it's the pickle and unpickle waste a lot of time. Pickling and unpickling a large numpy array take quite some time.

2: Using Manager.list(): Faster than queues, but when try to access it in main thread, it 's still very slow. Even just iterate over the list and do nothing takes 2 seconds per item. I don't understand why it take so much time.

Any suggestions ? Thanks.

how much memory does one batch of images take? If its a lot, think about using += and stuff like that to overwrite memory-parts, since A = A + B takes a lot more time than A+=B, when A and B become large (the memory layout becomes more important when your arrays are Huge) — usethedeathstar
– usethedeathstar, Commented Nov 13, 2013 at 8:33
@usethedeathstar Each of the batch will take about 300 MB. Thanks for your tips, I have already avoided operation like that. — tly_alex
– tly_alex, Commented Nov 25, 2013 at 23:45

Community · Accepted Answer · 2017-05-23 10:25:40Z

2

Looks like I have to answer my own question.

The problem I was facing could be solved by using shared memory with numpy.

More details could be found at

Use numpy array in shared memory for multiprocessing

The idea is basically create the shared memory in the main process, and assign the memory to a numpy array. Later in other process, you can either read from it or write to it.

This approach works pretty well for me, it speeds up my program by a factor of 10.

Because I am processing a large amount of data and pickling is not an option for me.

The most critical code is :

shared_arr = mp.Array(ctypes.c_double, N)
arr = tonumpyarray(shared_arr)

edited May 23, 2017 at 10:25

CommunityBot

11 silver badge

answered Nov 25, 2013 at 23:44

tly_alex

1132 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python read image to numpy array with multiprocessing

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related