adding element of a list to a column of dataframe

Question

I have defined such a dataframe:

from pyspark.sql.types import StructType, StructField, IntegerType, StringType
schema = StructType([
    StructField("date", StringType(), True), StructField("amount", StringType(), False), StructField("fontred", StringType(), False)
])

and I have such a list:

Elements=['01.07.', '01.07.', '02.07.', '02.07.', '02.07.', '02.07.']

How can I add elements of this list to first column of dataframe?

date      amount  fontred
----------------------------
01.07.
01.07.
02.07.
02.07.
02.07.
02.07.

mck · Accepted Answer · 2020-12-21 19:50:49Z

1

I think you've confused the nullable parameter... I've changed it for you. See if the code below is what you want:

from pyspark.sql.types import StructType, StructField, IntegerType, StringType

schema = StructType([
    StructField("date", StringType(), False),
    StructField("amount", StringType(), True),
    StructField("fontred", StringType(), True)
])

Elements = ['01.07.', '01.07.', '02.07.', '02.07.', '02.07.', '02.07.']

df = spark.createDataFrame([[i,None,None] for i in Elements], schema=schema)

df.show()
+------+------+-------+
|  date|amount|fontred|
+------+------+-------+
|01.07.|  null|   null|
|01.07.|  null|   null|
|02.07.|  null|   null|
|02.07.|  null|   null|
|02.07.|  null|   null|
|02.07.|  null|   null|
+------+------+-------+

answered Dec 21, 2020 at 19:50

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Kaja Over a year ago

thank you for your answer, if use your logic for amount -> column: df = spark.createDataFrame([[None,i,None] for i in Elements], schema=schema) then I loose information in date column. do you have any Idea what should i do?

mck Over a year ago

If you have multiple columns that you want to combine, you can do, for example, df = spark.createDataFrame([[i,j,None] for i,j in zip(Elements,AnotherElements)], schema=schema)

Red · Accepted Answer · 2020-12-21 19:53:02Z

0

You can use a pandas dataframe:

import pandas as pd
import numpy as np

Elements = ['01.07.', '01.07.', '02.07.', '02.07.', '02.07.', '02.07.']
df = pd.DataFrame({'date': Elements})
df['amount'] = df['fontred'] = np.nan
print(df)

Output:

     date  amount  fontred
0  01.07.     NaN      NaN
1  01.07.     NaN      NaN
2  02.07.     NaN      NaN
3  02.07.     NaN      NaN
4  02.07.     NaN      NaN
5  02.07.     NaN      NaN

answered Dec 21, 2020 at 19:53

Red

27.7k8 gold badges44 silver badges63 bronze badges

Collectives™ on Stack Overflow

adding element of a list to a column of dataframe

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related