How to Create Spark or Pandas Dataframe from str output in Apache Spark on Databricks

Question

I have assigned the variable "myoutput" to a string as follows

myoutput = result.content

myoutput is as follows:

Out[10]: 'Company A Invoice\nInvoice For:\nAddress:\n567 Main St.\nRedmond, WA\n555-555-5555\nBilbo Baggins\n123 Hobbit Lane\nRedmond, WA\n555-555-5555\nSubtotal: 300.00\nTax: 30.00\nTip: 100.00\nTotal: 430.00\nSignature: ____Bilbo Baggins__________\nItem\nQuantity\nPrice\nA\n1\n10.99\nB\n2\n14.67\nC\n4\n15.66\nD\n1\n12.00\nE\n4\n10.00\nF\n6\n12.00\nG\n8\n22.00'

I would like to create a spark dataframe or a pandas dataframe from "myoutput".

Any ideas?

Christian Eslabon · Accepted Answer · 2022-05-29 14:32:24Z

1

 import pandas as pd
 str_output = 'Company A Invoice\nInvoice For:\nAddress:\n567 Main St.\nRedmond, WA\n555-555-5555\nBilbo Baggins\n123 Hobbit Lane\nRedmond, WA\n555-555-5555\nSubtotal: 300.00\nTax: 30.00\nTip: 100.00\nTotal: 430.00\nSignature: ____Bilbo Baggins__________\nItem\nQuantity\nPrice\nA\n1\n10.99\nB\n2\n14.67\nC\n4\n15.66\nD\n1\n12.00\nE\n4\n10.00\nF\n6\n12.00\nG\n8\n22.00'
 df_data = pd.DataFrame({'ColumnA':str_output.splitlines()})
 df_data

Reference: How to split a Python string on new line characters

answered May 29, 2022 at 14:32

Christian Eslabon

1854 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Patterson Over a year ago

Ian, thanks for reaching. Surely, I wouldn't have to type out all of "'Company A Invoice\nInvoice For:\nAddress:\n567 Main St.\nRedmond, WA\n555-555-5555\nBilbo Baggins\n123 Hobbit Lane\nRedmond, WA\n555-555-5555\nSubtotal: 300.00\nTax: 30.00\nTip: 100.00\nTotal: 430.00\nSignature: ____Bilbo Baggins__________\nItem\nQuantity\nPrice\nA\n1\n10.99\nB\n2\n14.67\nC\n4\n15.66\nD\n1\n12.00\nE\n4\n10.00\nF\n6\n12.00\nG\n8\n22.00'"

Patterson Over a year ago

What if i have a document with over a thousands lines??

Patterson Over a year ago

Oh, I got it, you were just using your answer with the full text as an illustration. I did the following and it work just as well import pandas as pd df_data = pd.DataFrame({'ColumnA':mydf.splitlines()}) df_data

Collectives™ on Stack Overflow

How to Create Spark or Pandas Dataframe from str output in Apache Spark on Databricks

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related