0

I know there exists a duplicate question

But after such a long time, are there any new method to achieve the same target?


def process(id):
    temp_df = df[id]
    return temp_df.apply(another_function)

Parallel(n_jobs=-2)(delayed(process)(id) for id in df.columns)

The dataframe seems to be copied for each process, which is not possible for large dataframe. Are there any method or packages to fix this?

Improve this question
3
  • 1
    Multiprocessing in CPython enforce data to be copied. In fact, this is even worse : data needs to be pickled and unpicked. This is very slow. You can use shared Numpy array for native data (e.g. like np.intXX and np.floatXX types). That is not for non-uniform dataframes or the ones containing strings/objects. This also force you to use Numpy arrays rather than dataframe which is not convenient. CPython Multithreading is fundamentally limited by the GIL. Commented Dec 12, 2023 at 12:25
  • There might be some packages reimplementing Pandas with multiple native threads, but with strong limitations on the supported input type because of the GIL. There is no way to avoid the GIL for native types. This is an issue of the CPython interpreter itself. The same is true for multiprocessing (except for shared native data). Other Python implementation like PyPy may have less strong limitation but Python is clearly not great for parallel computing anyway (it as not been designed for that in the first place)... Commented Dec 12, 2023 at 12:28
  • 1
    One efficient solution I found so far is to convert the dataframe to Numpy arrays and use tools like Cython and Numba so to generate a native code using multiple threads without limitations of the GIL. This assume the dataframe contains native types (again because of the CPython's GIL). This solution mostly bypass CPython. If your application consists in computing dataframes in parallel like this and speed is critical, then it is certainly better to use another language (since Numba and Cython are far from being perfect for that). This is especially true for dataframes with strings. Commented Dec 12, 2023 at 12:37
Related questions
4241
How can I iterate over rows in a Pandas DataFrame?
4042
How can I use a global variable in a function?
2331
Delete a column from a Pandas DataFrame
Load 5 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.