Using Dask in Python to run function in parallel

Question

I'm trying to use Dask.delayed to have a function be computed in parallel. The function BL.BLSN has 1 argument which is a string.

runs=['forcing.dat','forcing.dat','forcing.dat','forcing.dat','forcing.dat','forcing.dat']
dfs = [delayed(BL.BLSN)(fn) for fn in runs]

When I print dfs I get a list of the delayed objects. Which is what I was expecting.

[Delayed('BLSN-630cb6bf-7104-4d41-b5fb-544116fec7ed'), Delayed('BLSN-150a5d45-073a-48af-af03-6e85f809617c'), Delayed('BLSN-484e7d71-af09-4a1f-be28-42358ef745e9'), Delayed('BLSN-2f2f2cc1-76bc-4c29-839d-091255556f35'), Delayed('BLSN-8aae0a06-d607-4a98-89a7-cfd2c877bf11'), Delayed('BLSN-f0131a69-2bfd-4153-94c1-2d3ef4973772')

]

However, when I try:

dfs.compute()

I get:

AttributeError: 'list' object has no attribute 'compute'

What is the best way to have compute operate on each object in the list?

You have a list of Delayed objects since you are using the list comprehension. Call compute on each of them or see @mdurant 's answer — nmlq
– nmlq, Commented Nov 1, 2019 at 20:30
If you call compute() sequentially, you will wait for each result and not experience any parallelism — mdurant
– mdurant, Commented Nov 4, 2019 at 18:51

mdurant · Accepted Answer · 2019-11-01 19:28:41Z

2

You want the function form of compute rather than the method:

import dask
dask.compute(*dfs)

This will run in parallel (with the usual caveats), and if tasks share dependencies, they will be shared intelligently.

answered Nov 1, 2019 at 19:28

mdurant

28.8k5 gold badges49 silver badges79 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Using Dask in Python to run function in parallel

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related