Write CSV file in append mode on Azure Databricks

Question

Wanted to write csv file in append mode on Azure Databricks. The below code is working fine on my local machine.(Jupyter notebook)

df = pd.read_csv("/dbfs/mnt/dev/tmp/ml_p/csv_append.csv")
df+6

Ans [1]: https://i.sstatic.net/sXsgH.png

when I opened the same csv file and wanted to save the file after performing the operation. I got , OSError: [Errno 95] Operation not supported

with open('/dbfs/mnt/dev/tmp/ml_p/csv_append.csv', 'a') as f:
   (df + 6).to_csv(f, header=False)

Is there is another alternative to write the CSV file in append mode? or Can I achieve the same using pyspark.

kb.databricks.com/en_US/dbfs/errno95-operation-not-supported — Alexander
– Alexander, Commented Aug 20, 2022 at 1:05
Nearly the same question, with accepted answer: stackoverflow.com/q/78284654/12846804 — OCa
– OCa, Commented Apr 9, 2024 at 7:54

Alex Ott · Accepted Answer · 2021-04-23 06:22:26Z

4

There are some limitations on what operations could be done with files on DBFS (especially via /dbfs mount point), and you hit this limit. The workaround would be to copy file from DBFS to local file system, modify it the same as you do it, and then upload back. Copying of the file could be done with dbutils.fs commands, like:

dbutils.fs.cp("dbfs:/mnt/dev/tmp/ml_p/csv_append.csv", "file:/tmp/csv_append.csv")
df = pd.read_csv("/tmp/csv_append.csv")
df+6
with open('/tmp/csv_append.csv', 'a') as f:
   (df + 6).to_csv(f, header=False)
dbutils.fs.mv("file:/tmp/csv_append.csv","dbfs:/mnt/dev/tmp/ml_p/csv_append.csv")

answered Apr 23, 2021 at 6:22

Alex Ott

88.1k10 gold badges110 silver badges157 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Write CSV file in append mode on Azure Databricks

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related