Error importing a cffi python module from AWS Lambda Layer

Question

I am having a problem importing the argon2-cffi python module (version 20.1.0) from an AWS Lambda Layer into an AWS Lambda function.

This seems to apply to all packages that have C as a base, because I tested this with pandas and got the same results.

My cloudformation config for the Layer is the following:

  MyLib:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: my-lib
      Description: In-house and 3rd party dependencies for my app.
      ContentUri: lambdas/lib/.
      CompatibleRuntimes:
        - python3.8

and the folder structure inside /lib is this:

lib
├── python
│   ├── app
│   │   └── ...
│   └── lib/python3.8/site-packages
│       ├── argon2
│       │   └── ...
│       ├── argon2_cffi-20.1.0.dist-info
│       │   └── ...
│       ├── ...
│       └── stdnum
│           └── ...
└── requirements.txt

as per recommendations from AWS, where the folder python/app contains my custom library and the folder python/lib/python3.8/site-packages contains 3rd party packages.

I know that since argon2 depends on C-code, it must be installed on the runtime environment that the Lambda function using it will be hosted in, so I use a docker image provided by AWS to install the packages into lib/python3.8/site-packages using the command

docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"

The lambda function I'm trying to run is configured in the cloudformation template as

  AuthorizerFunction:
    Type: 'AWS::Serverless::Function'
    Properties:
      CodeUri: lambdas/handlers
      Handler: authorizer.handler
      Runtime: python3.8
      ...
      Layers:
        - !Ref MyLib

and starts with these imports:

# authorizer.py

import app.utils as utils
import stdnum
from argon2 import PasswordHasher

But when I try to run it, I get the following error

[ERROR] Runtime.ImportModuleError: Unable to import module 'authorizer': No module named 'argon2._ffi'

meaning that app and stdnum both get imported successfully, but not argon2.

Any ideas what might be wrong?

I would also welcome any suggestions on another hashing library with a tried and tested hashing algorithm.

UPDATE: Updating my cffi, pip and setuptools as is recommended here did not do the trick.

UPDATE 2: I can see that the argon being installed using the docker command is the linux compatible one, since the output includes these lines:

Collecting argon2-cffi==20.1.0
  Downloading argon2_cffi-20.1.0-cp35-abi3-manylinux1_x86_64.whl (97 kB)

Nanna · Accepted Answer · 2020-12-18 14:43:14Z

3

Alright, so the answer to my problem is a bit of an awkward one. I'll post it here anyhow in case it might depthen an understanding for someone.

I am using git for source control and used a template from VS Code for the .gitignore file. Furthermore, I have set up an automatic build and deployment process, which builds the Lambda application based on the content in the git repo.

After days of trying to debug this problem I found out that .so files were missing from the argon2_cffi folder in the Lambda layer. The culprit was the aforementioned .gitignore file, which contained these lines:

# C extensions
*.so

No wonder none of the C-based python packages were working!

I removed said lines from .gitignore and committed all the .so files to the repo that had never made it to the Lambda Layer before, and now everything is working fine.

answered Dec 18, 2020 at 14:43

Nanna

5852 gold badges10 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Error importing a cffi python module from AWS Lambda Layer

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related