0

I have installed pandas, numpy, s3fs and created zip file to add as layer. added that zip file s3 bucket. But unable to find what is actual version it needs. Using Python 3.7.9.

I have installed pandas, numpy, s3fs and created zip file to add as layer. added that zip file s3 bucket. But unable to find what is actual version it needs. Using Python 3.7.9. Tried with manylinux wheel file but getting error not supporting to system. I want to work with pandas for reading of multiple csv files from s3 bucket which is of large size (>2GB). Please let me know which version I need to use. These concept and code working with local.

I am working on windows. Not using dockers. Simply installing packages, creating zip file, uploaded to S3 bucket. Adding that link to lambda layer. Then testing code but facing version issue. Let me know if these flow is wrong.

Improve this question
3
  • " Simply installing packages" - is not specific. Check my answer, and do as I did. Provide step by step instructions/screenshots how to reproduce the issue. Commented Aug 21, 2021 at 3:51
  • How did it go? Did you manage to prepare step-by-step instructions on how to reproduce your issue? Commented Aug 21, 2021 at 23:38
  • Yes. It's resolved. Thank you all Commented Aug 23, 2021 at 4:35

2 Answers 2

Reset to default
2

The best way to create lambda layers is using docker as described in AWS docs. Thus in your case the you would do (commands for linux):

  1. Create empty folder, e.g. mylayer.

  2. Go to the folder and create requirements.txt file with the content of

numpy
pandas
s3fs
  1. Run the following docker command for python3.8:
docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"
  1. Create the zip layer:
zip -9 -r mylayer.zip python

  1. Create lambda layer based on mylayer.zip in the AWS Console. Don't forget to specify Compatible runtime to python3.8.

  2. Add the the layer created in step 5 to your function.

Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

I am working on windows. Not using dockers. Simply installing packages, creating zip file, uploaded to S3 bucket. Adding that link to lambda layer. Then testing code but facing version issue. Let me know if these flow is wrong.
@varshspatil docker also works on Windows.
tried to install docker also but getting error "docker engine failed to start..."
@varshapatil In that case you have to update your question, and provide all steps and commands, just like I did, explaining what you did, how you did that.
C:\Users\varspati\OneDrive - Cisco\Desktop\Docker>docker run -v "%cd%":/var/task "lambci/lambda:build-python3.6" /bin/sh -c "pip install -r requirements.txt -t C:/Users/varspati/OneDrive - Cisco/Desktop/Docker/python/lib/python3.6/site-packages/; exit" ERROR: Invalid requirement: '-' WARNING: You are using pip version 21.0; however, version 21.2.4 is available. You should consider upgrading via the '/var/lang/bin/python3.6 -m pip install --upgrade pip' command. @Marcin, getting error while trying to add lambda layer using docker. 1st and 2nd step followed as you said.
|
0

You can have a look at this git repo which not only provides a layer that includes numpy but also has instructions on how to create such a layer. You can probably find whatever you need in there.

Improve this answer

2 Comments

These ways tried by me but facing issue of versions. Not getting which version I need to install and use.
This page (which was linked from the git repo) holds version info. Is this what you're looking for?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.