How to read data from Databricks DBFS using Rest API in csv or Excel format?

Question

I am using Databricks Rest API to read datasets stored on DFBS. The output is coming in 64bit encoded format and in json format. I need the output in tabular format which is easy to read.

Output of Rest Api: { "bytes_read": 4601, "data": "U2VwYWxMZW5ndGgsU2VwYWxXaWR0aCxQZXRhbE" }

output needed:

enter image description here

Are you sure there's no more output? Running that data value through a Base64 decoder gives part of the column names from your image, so it seems like there should be more information in the response. — c-x-berger
– c-x-berger, Commented Jul 25, 2020 at 7:40
The output was a bigger string. I shortened it for the sake of simplicity. I there a way to decode it and get a table structure in the in the API Itself. — manish007
– manish007, Commented Jul 25, 2020 at 8:19
Not sure. I'd guess is you base64 decoded the full string, you'd get the table in CSV form. Can't tell without the full string though. — c-x-berger
– c-x-berger, Commented Jul 26, 2020 at 16:58

Priyank Sharma · Accepted Answer · 2020-08-21 17:34:44Z

1

You can use below code to download csv file,

jsonbody = {"path": dbfspath}

TOKEN = "dapi....." # you can generate this token from users settings

response = requests.get('https://eastus.azuredatabricks.net/api/2.0/dbfs/read/', headers={'Authorization': 'Bearer %s' % self.TOKEN }, json= jsonbody )

filedata = json.loads(response.text)["data"]

with open(outputpath, "wb") as file:

  file.write(base64.b64decode(filedata))

Let me know if you need any other help

answered Aug 21, 2020 at 17:34

Priyank Sharma

111 bronze badge

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to read data from Databricks DBFS using Rest API in csv or Excel format?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related