I want to create a vector with sitze 10^15 with numpy and MPI.
I have no clue how to start. I tried to use
v = np.random.rand(10**15,1)
but it exceeds my memory.
1
-
Indeed you can't. However you can handle relatively large (10s of GB) arrays with Dask examples.dask.org/array.html#Create-Random-arrayArdweaden– Ardweaden2020-05-05 12:50:17 +00:00Commented May 5, 2020 at 12:50
Add a comment
|
2 Answers
You can't. This is thousands terabytes of data, too much even for storing on disk. Consider redesigning your program to reduce amount of data or processing data iteratively.
Comments
Dask arrays coordinate many Numpy arrays, arranged into chunks within a grid. They support a large subset of the Numpy API.
Call .compute() when you want your result as a NumPy array.
%%time
import dask.array as da
answer_sum = da.random.random((10**15)).sum().compute()
answer_mean = da.random.random((10**15)).mean().compute()
print(answer_sum)
print(answer_mean )
Source: https://examples.dask.org/array.html#Create-Random-array
1 Comment
rizerphe
While this code may solve the question, including an explanation of how and why this solves the problem would really help to improve the quality of your post, and probably result in more up-votes. Remember that you are answering the question for readers in the future, not just the person asking now. Please edit your answer to add explanations and give an indication of what limitations and assumptions apply.