Dask, create a dataframe from several dask arrays

Question

Suppose I have a set of dask arrays such as:

c1 = da.from_array(np.arange(100000, 190000), chunks=1000)
c2 = da.from_array(np.arange(200000, 290000), chunks=1000)
c3 = da.from_array(np.arange(300000, 390000), chunks=1000)

is it possible to create a dask dataframe from them? In pandas i could say:

data = {}
data['c1'] = c1
data['c2'] = c2
data['c3'] = c3

df = pd.DataFrame(data)

is there a similar way to do this with dask?

I suspect that you could do this with a combination of dd.from_dask_array and dd.concat(..., axis=1). — MRocklin
– MRocklin, Commented Mar 28, 2017 at 2:28

Arco Bast · Accepted Answer · 2017-03-28 07:39:13Z

13

The following should work:

import pandas as pd, numpy as np 
import dask.array as da, dask.dataframe as dd

c1 = da.from_array(np.arange(100000, 190000), chunks=1000)
c2 = da.from_array(np.arange(200000, 290000), chunks=1000)
c3 = da.from_array(np.arange(300000, 390000), chunks=1000)

# generate dask dataframe
ddf = dd.concat([dd.from_dask_array(c) for c in [c1,c2,c3]], axis = 1) 
# name columns
ddf.columns = ['c1', 'c2', 'c3']

answered Mar 28, 2017 at 7:39

Arco Bast

3,9502 gold badges31 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Dask, create a dataframe from several dask arrays

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related