Lambda Not Loading Cryptography Shared Library

Question

I'm using the cryptography library in AWS Lambda. I've compiled the package using pip in an Amazon Linux VM. I have uploaded the package as a layer. Either way, every time I call the library, I have an error which is not descriptive at all:

Unable to import module 'lambda_function': libffi-ae16d830.so.6.0.4: cannot open shared object file: No such file or directory

As you can see, the error is not about not finding the lib, is another shared module which I haven't been able to find.

Here's an example of the code I'm trying to execute on Lambda:

from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
from cryptography.fernet import Fernet

password_provided = "test123" 
password = password_provided.encode() 
salt = b'test_' 
kdf = PBKDF2HMAC(
    algorithm=hashes.SHA256(),
    length=32,
    salt=salt,
    iterations=100000,
    backend=default_backend()
)
key = base64.urlsafe_b64encode(kdf.derive(password)) 


message = "message from db".encode()

f = Fernet(key)
encrypted = f.encrypt(message)

print(encrypted)

f = Fernet(key)
decrypted = f.decrypt(encrypted)

print(decrypted.decode("utf-8"))

It's not the first time I've compiled a library to work on AWS Lambda, but in this case even I compile the cryptography lib. What should I add or change?

Edit: I've found out the library was missing in the zip file I've created, as is inside a hidden folder. I zipped using '.' instead of '*' but now I'm running with a new problem: When I run the lambda, I get this:

Unable to import module 'lambda_function': /opt/cryptography/hazmat/bindings/_constant_time.so: undefined symbol: PyInt_FromLong

any idea?

I'm not completely sure, but it looks like your compiled library contains an SO file which is dependent by libffi-ae16d830.so - maybe check if you can include it in your package in some way. — Ronyis
– Ronyis, Commented Apr 4, 2019 at 8:52

Subhash Rajagopal · Accepted Answer · 2019-09-11 16:31:53Z

1

Even I faced the same issue, while zipping I forgot to include the hidden files(.libs_cffi_backend) in the site-packages. After Including it, I didn't see this error.

answered Sep 11, 2019 at 16:31

Subhash Rajagopal

211 bronze badge

Sign up to request clarification or add additional context in comments.

Comments

Shuvojit · Accepted Answer · 2019-04-09 15:30:20Z

0

As the library you're using requires native libraries, you have to pack the native .so files as well with the layer. I ran into a similar issue while trying to run wkhtmltopdf on aws lambda.

The binaries for the library has to be compiled in the same environment as a lambda instance. Lambda gets booted up using AWS Linux.

You can either boot up an EC2 running AmazonLinux or use docker, easiest way is to boot up a docker container.

$ sudo docker run -it amazonlinux bash

Now you need to download/unpack all .so files into a directory then zip it. Also, make sure to keep all .so files inside a folder called lib inside the zip. After zipping, the zip should look something similar to this:

.
├── lib
│   ├── libcrypto.so.10
│   ├── libcrypto.so.1.0.2k
│   ├── libfontconfig.so.1
│   ├── libfontconfig.so.1.7.0
.......

Then you can just zip it and upload it as a layer. It will be uploaded to /opt/ in your Lambda Container. AWS looks for library files under /opt/lib amongst many other locations.

The challenging part for you would be to figure out how to get all the required .so files in order for your dependency to run properly.

answered Apr 9, 2019 at 15:30

Shuvojit

1,4709 silver badges17 bronze badges

1 Comment

transposeglobal Over a year ago

Should the lib folder with the .so files be in the root of the layer zip?

byohay · Accepted Answer · 2022-02-24 09:49:06Z

0

Previously I manually packed the archive, deployed with aws cloudformation deploy, and had the same problem. I then switched to using sam build --use-container and it worked. My suspicion is that it works because SAM uses a specific container. This is the container it uses: https://gallery.ecr.aws/sam/build-python3.7

answered Feb 24, 2022 at 9:49

byohay

2824 silver badges11 bronze badges

Comments

gkaur · Accepted Answer · 2024-04-17 17:34:13Z

Ran into the same issue while creating a layer for snowflake-python-connector for arm64 using amazonlinux docker container.

About the snowflake-python-connector issue in case it helps someone - packaging the layer using docker on x86 arch gives the following error

{ "errorMessage": "Unable to import module 'lambda_function': /opt/python/lib/python3.9/site-packages/cryptography/hazmat/bindings/_rust.abi3.so: cannot open shared object file: No such file or directory", "errorType": "Runtime.ImportModuleError", "requestId": "07fc4b23-21c2-44e8-a6cd-7b918b84b9f9", "stackTrace": [] }

And using arm64 throws the following error

File "/usr/lib/python2.6/site-packages/cffi/api.py", line 56, in __init__
    import _cffi_backend as backend
ImportError: No module named _cffi_backend

Creating the layer as mentioned in another stackoverflow question under the accepted solution comments worked for me.

Posting the link here - https://repost.aws/knowledge-center/lambda-layer-simulated-docker

Collectives™ on Stack Overflow

Lambda Not Loading Cryptography Shared Library

4 Answers 4

Comments

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related