1

Is there any resolution to the problem of memory error?

The memory is 8 GB, and I want to complete the broadcasting operation, but it did have a huge cost.

Given the condition:

interdata = data[:, None] - data[None]

The data is with shape [1000, 32, 32], and I want the interdata with a shape of [1000, 1000, 32, 32], and that seems to be too large for my memory.

Is there a method to solve the difficulty?

Improve this question
4
  • 1
    Why do you need that? Is this really the end goal? Commented Apr 20, 2018 at 9:09
  • If I get the interdata, then I will use it as operand of the next several operation. So I want the pre-calculation in order to reduce the total time cost. Commented Apr 20, 2018 at 9:21
  • 1
    It is quite likely that the solution you seek is not a direct solution to this question. Perhaps if you explain in more detail what you want to achieve, and what your approach is, then we may identify solutions that will circumvent your original problem all together. For example, there may be a solution that involves re-using results from the previous operation, so that you don't have to pre-compute interdata in one go. Commented Apr 20, 2018 at 9:29
  • It's difficult to describe the whole question. I'm trying to recurring a thesis, and that needs a several step. Each part is independently. Only with the previous result can we continue the next step. I'll rethink the whole process. Thanks for you time! Commented Apr 20, 2018 at 10:21

1 Answer 1

Reset to default
1

Is there a method to solve the difficulty?

Sure, but at a cost . . .

numpy can use a .memmap()-ed resource, where a disk-capacity is your next capacity "ceiling".

Yet, this will get you into units of [us] for SSD-device hosted .memmap()-storage or ~1E+1 [ms] for spinning drives, so bear that in mind.

If this is not acceptable, there are reasonably expensive COTS-platforms today providing multi-TB capacities or RAM, where [ns]-rule.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

I'm not so clear about the operation ".memmap". Would you mind to say more about the detail?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.