String issue for Python multiprocessing

Question

facing some issues with string parsing and the multiprocessing library. Here is my code, and I also outline the function calls and error.

def semi_func(tile):
        with open(tile, 'rb') as f:
           img = Image.open(BytesIO(f.read()))
           resized_im, seg_map = MODEL.run(img)
           vis_segmentation_tiles(str(tile),resized_im, seg_map)
           x = np.unique(seg_map)
           x = x.tolist()
           print("THIS IS X", x)
           ans_tiles[str(tile)] = x
           print(x)
        return ans_tiles
    
def split_tiles_new(image_path, tiledir):
        print("1")
        pool = Pool(processes=5)
        print("2")
        num_tiles = 9
        tiles = image_slicer.slice(image_path, num_tiles, save=False)
        print("3")
        print(tiles)
        image_slicer.save_tiles(tiles, directory=tiledir)
        print(tiles)
        print("TILES ABOEVE")
        
        onlytiles = [os.path.join(tiledir,f) for f in listdir(tiledir) if isfile(join(tiledir, f))]
       
        ans_tiles = {}
        print(onlytiles)
        onlytiles = list(map(str, onlytiles))
        for t in onlytiles:
            print(t)
        for tile in onlytiles:
            print(tile)
            pool.map(semi_func,tile)
        pool.close()
        pool.join()
        print(ans_tiles)
        return ans_tiles

Here's what I'm feeding in terms of my functions:

ans_tiles = split_tiles_new(local_jpg, tiledir)

local_jpg = 'wheat044146108.jpg'

tiledir = 'tiles044146108'

Inside tiledir (the directory), there's a bunch of tiled images:

['tiles044146108/_03_02.png', 'tiles044146108/_03_01.png', 'tiles044146108/_02_02.png', 'tiles044146108/_01_01.png', 'tiles044146108/_03_03.png', 'tiles044146108/_01_02.png', 'tiles044146108/_02_01.png', 'tiles044146108/_02_03.png', 'tiles044146108/_01_03.png']

That's what is in the variable 'onlytiles'.

But my issue is this error:

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "serve_wh.py", line 128, in semi_func
    with open(tile, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: 't'
"""

I am not sure why it is doing further slicing of the string? Any idea what I can do to ensure it just grabs each file from 'onlyfiles' list separately in this?

Albin Paul · Accepted Answer · 2021-08-19 17:07:18Z

2

Your iterable is a filename string thats why it's trying to open file with name t. Check Pool.map second argument.

pool.map(semi_func,tile)

You should use

pool.map(semi_func,onlytiles)

Without the for loop so that it iterates over the list rather than string.

answered Aug 19, 2021 at 17:07

Albin Paul

3,4292 gold badges17 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

mike0494 Over a year ago

I try this and it keeps timing out

Albin Paul Over a year ago

@mike0494 Please tell at which portion it is stuck. You can check that by adding lots of print statements. This change will fix the filename not found issue. But the timeout one is different.

Collectives™ on Stack Overflow

String issue for Python multiprocessing

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related