Write data from pyspark to azure blob?

Question

I want to write dataframe from pyspark to azure blob? Any suggestions or code how to do it?

I have location and key of blob

enter image description here

Yes in approach it helps but facing issue while writing the data as csv. Please find the link for the error in question part. — Vigneshwar Thiyagarajan
– Vigneshwar Thiyagarajan, Commented Jul 12, 2019 at 6:26

Jay Gong · Accepted Answer · 2019-07-11 07:27:40Z

3

You could follow this tutorial to connector your spark dataframe with Azure Blob Storage.

Set connection info:

session.conf.set(
    "fs.azure.account.key.<storage-account-name>.blob.core.windows.net",
    "<your-storage-account-access-key>"
)

Then write data into blob storage:

sdf = session.write.parquet(
    "wasbs://<container-name>@<storage-account-name>.blob.core.windows.net/<prefix>"
)

Also,you could refer to this case:pyspark write to wasb blob storage container

edited Jul 11, 2019 at 7:27

answered Jul 11, 2019 at 7:16

Jay Gong

23.8k2 gold badges30 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Vigneshwar Thiyagarajan Over a year ago

I tried the approach which you mentioned above and getting exception as "Caused by: java.lang.IllegalArgumentException: The String is not a valid Base64-encoded string."

Vigneshwar Thiyagarajan Over a year ago

I have attached a screenshot on the error message i got below the question. Please have a look on it and can you tell me what the exact error is?

Collectives™ on Stack Overflow

Write data from pyspark to azure blob?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related