How can I solve cx_Oracle problem in python when running on aws lambda

Question

I had encountered below error message when I test on aws lambda.

Unable to import module 'lambda_handler': No module named 'cx_Oracle'

This lambda consists of python code, but especially works well on my localhost.

Also, I executed pip install cx_Oracle -t ./ command on same level of lambda_handler.py file path, and uploaded zip file's file path is below that.

./lambda_handler.py

./cx_Oracle.cp36-win_amd64.pyd

./cx_Oracle-6.4.1.dist-info/*

./cx_Oracle-doc/*

I can't understand why lambda_handler can't find cx_Oracle library in only lambda function.

What am I missing?

import cx_Oracle
import boto3
from docutils.nodes import row

def lambda_handler(event, context):

    url = 'myOracleUrl'
    user = 'myOralceUserId'
    pwd = 'myOraclePwd'
    con = cx_Oracle.connect(user + '/' + pwd + '@' + url)
    cur = con.cursor()
    cur.execute("select 1 from dual")
    cur.close()
    con.close()

    return {
        "statusCode": 200,
        "body": context.invoked_function_arn
    }

Don't forget the Oracle Instant Client and matching VS Redistributable. This may not solve the immediate problem because I'd expected a different error message. — Christopher Jones
– Christopher Jones, Commented Sep 11, 2018 at 6:33

alvaro · Accepted Answer · 2018-12-03 10:34:57Z

8

Short Answer:
The cx_Oracle library you are using is for Windows. Aws Lambda environment is Amazon Linux, *.pyd are Python libraries for Windows.

Long Answer: You have to deploy the linux version The first thing you are getting wrong is the python library you are providing along your code to AWS Lambda. I guess you are developing locally in a Windows machine so when you do:

pip install cx_Oracle -t ./

you get a python precompiled library for Windows which is the cx_Oracle.cp36-win_amd64.pyd file. AWS Lambdas run in Linux containers. You can find some info about the environment here. The precompiled library for Linux is cx_Oracle.cpython-36m-x86_64-linux-gnu.so. You can download it manually from the Pypi repository.

So when you execute a python script in a Linux environment it will try to find the linux precompiled version of the library in the current path, the cx_Oracle.cpython-36m-x86_64-linux-gnu.so file. That is why you are getting the error message:

Unable to import module 'lambda_handler': No module named 'cx_Oracle'

The second thing important here is that the cx_Oracle python library calls the Oracle Instant Client which is a native library. If you have a working example on your windows machine you must have this installed. For windows this library is in the form of .dll files. For linux this native library is in the form of .so files. You will need to provide the Oracle client for Linux along with your code in your deployment package because the are not already available in the Amazon Linux environment.

And the last thing is to tell Python how to find this dynamic library. In Linux systems shared libraries are looked for in the directories present in this environment variable LD_LIBRARY_PATH. In Amazon Linux this variable contains:

LD_LIBRARY_PATH:/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib

The /var/task directory is whre your deployment resides. So if you add the Oracle instant client for Linux under /var/task/lib which is ./lib in your development directory you are done.

Your deployment package should be similar to this:

├── lib                         
│   ├── libclntsh.so                            <-- Oracle instant client linux files
│            .
│            .
│            .
├── python_handler.py                           <-- Lambda function code
└── cx_Oracle.cpython-36m-x86_64-linux-gnu.so   <-- cx_Oracle library

edited Dec 3, 2018 at 10:34

answered Nov 21, 2018 at 13:31

alvaro

4625 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user2793390 Over a year ago

With regards to the Oracle instant client linux files... The download from Oracle is 227MB. Which files should I put in the 'lib' folder?

alvaro Over a year ago

You just have to include libclntsh.so, which is the binary dependency that the cx_Oracle python library will call

user2793390 Over a year ago

Based on my experience, do not do it in Windows. Do it in Linux. Don't get the zip files but rather use the RPMs to install. Get the basiclite version. The basic one will exceed 50MB lambda limit when zipped up. You'll have to add /lib64/libaio* files o /usr/lib/oracle/.../lib. Relax permissions on the files in the lib folder, and then zip them up - zip --symlinks -r9 ~/lamda.zip *

bimsapi · Accepted Answer · 2018-09-11 11:49:29Z

0

cx_Oracle requires not just pip install, but the native Oracle client, which the wheel depends on. Lambda functions run on an EC2 with a limited selection of native libraries. For what it's worth, you'd have the same problem with almost any RDBMS - PostgreSQL, MySQL, etc.

See this article for more information: http://kb.sites.apiit.edu.my/knowledge-base/articles/aws/aws-chalice-deployment-with-oracle-vendor/

Note that it is for Chalice-based deployment, but the underlying requirement is the same.

answered Sep 11, 2018 at 11:49

bimsapi

5,0652 gold badges22 silver badges27 bronze badges

Comments

trnrao · Accepted Answer · 2018-09-11 15:46:37Z

0

We had run into this exact problem and we then had to implement the data access functions into AWS Lambda Java 8 instead of Python 3.6

In Java 8, ojdbc*.jar (* --> version number of Oracle driver) can be simply bundled into the final zip and there are no other steps like trying to get Oracle Client Native Libraries.

We did spend good amount of time to make AWS Lambda Python Function access Oracle DB, but I felt developing such modules in AWS Lambda Java 8 function is much simpler.

answered Sep 11, 2018 at 15:46

trnrao

941 silver badge2 bronze badges

1 Comment

Miboon Kim Over a year ago

Thank you so much, I solved this problem through include oracle instance client in zip file. but it spend more time than using java.

Collectives™ on Stack Overflow

How can I solve cx_Oracle problem in python when running on aws lambda

3 Answers 3

3 Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest