PySpark - Insert Rows or create new dataframe using json response

Question

Here is a simplified dataframe that I'm using.

+----+---------------+-------+
|col1|           col2|   col3|
+----+---------------+-------+
|   a|   Hello World!|   en  |
+----+---------------+-------+

Using this dataframe, I take col3's row value and subtract it from a another set to get a list like so:

for row in collect:
  lang = set([row['col3']])
  req_languages = set(['en','zh-Hans','ko','ja'])
  translate_list=list(req_languages-language)

The list is the languages I need to send as a parameter to the API. So for a, translate_list would be ['zh','ko','ja']. col2 is what is going to be translated and is in the body.

The api response then is going to return 3 translations and will look like...

"translations": [
            {
                "text": "世界您好！",
                "to": "zh-Hans"
            },
            {
                "text": "전 세계 여러분 안녕하세요!",
                "to": "ko"
            },
            {
                "text": "ハローワールド！",
                "to": "ja"
            }
        ]

The output then should look like:

+----+-------------------------+---------+
|col1|                     col2|     col3|
+----+-------------------------+---------+
|   a|             Hello World!|    en   |
|   a|           ハローワールド!|    ja   |
|   a| 전 세계 여러분 안녕하세요!|    ko   |
|   a|                 世界您好| zh-Hans |
+----+-------------------------+---------+

How can I manipulate the response to either insert new rows or construct a new dataframe to match the one above?

Note: The actual dataframe has many rows, so I need to send the value from each to row. As well as this call will be getting made thousands of times per day.

PotSkill · Accepted Answer · 2021-10-06 01:12:19Z

1

I would recommend that you just iterate through your dictionary and add a row every time with your needed information. You would start with your initial top rows.

data = """
+----+---------------+-------+
|col1|           col2|   col3|
+----+---------------+-------+"""

Then iterate through your translations and add a line every time.

for x in translations: #iterates through the 3 dictionaries
    text = x["text"] #python didn't like doing this inside the f-string
    to = x["to"]
    data += f"\n|   a|  {text}|  {to}  |" # uses f-strings to input variables into a string

data += "\n+----+---------------+-------+"

With the final code looking like this:

translations = [
            {
                "text": "世界您好！",
                "to": "zh-Hans"
            },
            {
                "text": "전 세계 여러분 안녕하세요!",
                "to": "ko"
            },
            {
                "text": "ハローワールド！",
                "to": "ja"
            }
        ]

data = """
+----+---------------+-------+
|col1|           col2|   col3|
+----+---------------+-------+"""

for x in translations: #iterates through the 3 dictionaries
    text = x["text"]
    to = x["to"]
    data += f"\n|   a|{text}   |  {to}  |" # uses f-strings to input variables into a string

data += "\n+----+---------------+-------+"


print(data)

it gives the output of:

+----+---------------+-------+
|col1|           col2|   col3|
+----+---------------+-------+
|   a|世界您好！   |  zh-Hans  |
|   a|전 세계 여러분 안녕하세요!   |  ko  |
|   a|ハローワールド！   |  ja  |
+----+---------------+-------+

Although a thing you could look into is lining the lines up, as in finding the length of the text, taking that away from the default value of spaces, i.e like 30 (which would require 30 "-" at the top). and then if the text variable is 24 long, another 6 spaces are added on the end. This is just for aesthetics but it might mean your data is easier to read. A fun little challenge maybe!

answered Oct 6, 2021 at 1:12

PotSkill

2173 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

AdibP Over a year ago

The OP is asking for a solution using Pyspark, not just printing the data

PotSkill Over a year ago

Sorry I didn't see that, i only saw that data was inputted and the desired output was outputted.

Collectives™ on Stack Overflow

PySpark - Insert Rows or create new dataframe using json response

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related