4

I want to process some code in parallel, I used threading.Thread and also multiprocessing.Process but weirdly they both took more time then normal execution which is the opposite of what i want.

The code works well on other systems, but on lambda function it does not works as expected.

I also tried this link: https://aws.amazon.com/blogs/compute/parallel-processing-in-python-with-aws-lambda/

But didn't get any performance upgrade.

Can anyone tell me if its possible over lambda and if yes how can i do that?

Improve this question
0

1 Answer 1

Reset to default
6

Yes, it is possible, you don't have to do anything special.

The reason you saw degraded performance is because your lambda function was most likely too small.

AWS measures CPU performance in vCPUs, where 1 vCPU is a virtualized thread on a CPU core.

With AWS lambda, the only thing you can scale vertically is memory, but CPU scales with it. To be precise, every 1,769MB of RAM corresponds to 1 vCPU.

Increase the vCPU count by increasing the RAM and you will see proportionally increased parallel processing performance.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Thanks! yes indeed. but i have a question, when i first time, set 7+ GB from 1+ GB it showed 2 processors, but afterwards when i set 5+ GB it showed 4 processors. Is there a reason for it? When the next time function will run, is it sure that there will be 4 processors?
What do you mean by "it showed 4 processors"? Sounds like this is a separate question about how Python detects different AWS vCPU configurations.
Sorry, i programmatically checked the number of processors for the lambda function and it showed 2 processors at 7+ GB but when i checked again it showed 4 processors for 5+ GB . I got number of processors using this command cat /proc/cpuinfo.
This is a different question, separate from your original one. You shouldn't rely on that information, rely on the AWS documentation I linked.
@gshpychka Will the single lambda invocation be able to access multiple vCPUs if i use multiprocessing library?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.