Installing snowflake-connector-python[pandas] in Azure Functions

Question

I'm trying to deploy python code utilizing the python-snowflake connector with pandas extras to azure functions using the VS Code extension. Locally running the function works & the deployment itself works as well

My requirements.txt for the installed dependencies during deployment looks like this:

azure-functions
xlrd
numpy
pandas
azure-storage-blob
snowflake-connector-python

The imports in the code are the following:

import numpy as np
import pandas as pd
import azure.storage.blob
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
import snowflake.connector
from snowflake.connector import DictCursor
#from snowflake.connector.pandas_tools import write_pandas

With these requirements everything works.

The issue is, as soon as I change the requirements.txt to include pandas extras for the snowflake connector (which are required in the code):

azure-functions
xlrd
numpy
pandas
azure-storage-blob
snowflake-connector-python[pandas]

I get the following error message trying to execute the function:

Result: Failure Exception: KeyError: 'snowflake-connector-python' Stack: File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 262, in _handle__function_load_request func = loader.load_function( File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 32, in call return func(*args, **kwargs) File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/loader.py", line 76, in load_function mod = importlib.import_module(fullmodname) File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1014, in _gcd_import File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "<frozen importlib._bootstrap>", line 1014, in _gcd_import File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 783, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/site/wwwroot/HttpTrigger1/__init__.py", line 23, in <module> import snowflake.connector File "/home/site/wwwroot/.python_packages/lib/site-packages/snowflake/connector/__init__.py", line 17, in <module> from .connection import SnowflakeConnection File "/home/site/wwwroot/.python_packages/lib/site-packages/snowflake/connector/connection.py", line 47, in <module> from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH File "/home/site/wwwroot/.python_packages/lib/site-packages/snowflake/connector/cursor.py", line 48, in <module> from .arrow_result import ArrowResult File "arrow_result.pyx", line 16, in init snowflake.connector.arrow_result File "/home/site/wwwroot/.python_packages/lib/site-packages/snowflake/connector/options.py", line 35, in <module> _pandas_extras = pkg_resources.working_set.by_key['snowflake-connector-python']._dep_map['pandas']

Any help or ideas on how to get the function running correctly? Thank you!

datamonk · Accepted Answer · 2020-10-07 00:33:19Z

It appears to be a bug in the pandas tools, I have been running into the same issue. Depending on what it is you plan to use the pandas tools you may be able to get around it.

My function relied on the fetch_pandas_all() function when I was querying Snowflake, but I was able to work around it by only installing the base snowflake-connector-python package and using a direct install of pandas, then changing my code to use the following:

import logging
import snowflake.connector
import pandas as pd

class SnowflakeReader(object):
    def __init__(self, accnt: str, warehouse: str, dbuser: str, pw: str, db: str, dbschema: str):
    self.accnt = accnt
    self.warehouse = warehouse
    self.db = db
    self.dbschema = dbschema
    self.dbuser = dbuser
    self.pw = pw


    def form_connection(self):
        try:
            ctx = snowflake.connector.connect(
                user = self.dbuser,
                password = self.pw,
                account = self.accnt,
                warehouse = self.warehouse,
                database = self.db,
                schema = self.dbschema
            )
        except Exception as e:
            logging.error('Failed to connect to Snowflake: ' + str(e))
        return ctx

    def read_all(self, query: str, conn) -> pd.DataFrame:
        self.query = query
        self.conn = conn

        try:
            cs = conn.cursor()
            sql_cmd = self.query
            cs.execute(sql_cmd)
            result = cs.fetchall()
            result_df = pd.DataFrame(result)
        except Exception as e:
            logging.error('Failed to read from Snowflake: ' + str(e))

        return result_df

It looks like you were trying to import write_pandas which leads me to think you were hoping to write a DataFrame directly to a table in Snowflake. If that's the case you may be able to build a similar workaround using the older SQLAlchemy based approach, or just convert your DataFrame to a dict and use the execute() function to pass to an insert statement.

You are right, I was trying to use the write_pandas() function to write a DataFrame directly to a table in Snowflake. In the end I did exactly what you suggested and rewrote the code to use the older SQLAlchemy approach, which worked out.

Collectives™ on Stack Overflow

Installing snowflake-connector-python[pandas] in Azure Functions

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related