upload a sample pyspark dataframe to Azure blob, after converting it to excel format

Question

I'm trying to upload a sample pyspark dataframe to Azure blob, after converting it to excel format. Getting the below error. Also, below is the snippet of my sample code.

If there is a other way to do the same, pls let me know.

from pyspark.sql.types import StructType,StructField, StringType, IntegerType

import pandas as ps
#%pip install xlwt
#%pip install openpyxl
#%pip install fsspec

my_data = [
            ("A","1","M",3000),
            ("B","2","F",4000),
            ("C","3","M",4000)
          ]

schema = StructType([ \
    StructField("firstname",StringType(),True), \
    StructField("id", StringType(), True), \
    StructField("gender", StringType(), True), \
    StructField("salary", IntegerType(), True) \
  ])

df = spark.createDataFrame(data=my_data,schema=schema)

pandasDF = df.toPandas()

pandasDF.to_excel("wasbs://[email protected]/output_file.xlsx")

ValueError: Protocol not known: wasbs

Utkarsh Pal · Accepted Answer · 2022-04-21 09:26:37Z

1

You are directly using python library pandas to write the data. This isn't work this way. You need to first mount the Azure Blob storage container and then write the data.

To mount, use following command:

dbutils.fs.mount(
  source = "wasbs://<container-name>@<storage-account-name>.blob.core.windows.net",
  mount_point = "/mnt/<mount-name>",
  extra_configs = {"<conf-key>":dbutils.secrets.get(scope = "<scope-name>", key = "<key-name>")})

To write, use below commands:

df.write
 .mode("overwrite")
 .option("header", "true")
 .csv("dbfs:/mnt/azurestorage/filename.csv"))

answered Apr 21, 2022 at 9:26

Utkarsh Pal

4,5781 gold badge8 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

kanishk kashyap Over a year ago

Could you please help me to make some column as read-only (let's say Salary column ) while writing to excel format?

Collectives™ on Stack Overflow

upload a sample pyspark dataframe to Azure blob, after converting it to excel format

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related