Convert Column value in Dataframe to list

Question

I have the following source file. I have a name called "john" in my file wanted to split to list ['j','o','h','n']. Please find the person file as follows.

Source File:

id,name,class,start_data,end_date
1,john,xii,20170909,20210909

Code:

from pyspark.sql import SparkSession

def main():
    spark = SparkSession.builder.appName("PersonProcessing").getOrCreate()

    df = spark.read.csv('person.txt', header=True)
    nameList = [x['name'] for x in df.rdd.collect()]
    print(list(nameList))
    df.show()

if __name__ == '__main__':
    main()

Actual Output:

[u'john']

Desired Output:

['j','o','h','n']

hamza tuna · Accepted Answer · 2018-09-04 08:07:15Z

5

If you want to in python:

nameList = [c  for x in df.rdd.collect() for c in x['name']]

or If you want to do it in spark:

from pyspark.sql import functions as F

df.withColumn('name', F.split(F.col('name'), '')).show()

Result:

+---+--------------+-----+----------+--------+
| id|          name|class|start_data|end_date|
+---+--------------+-----+----------+--------+
|  1|[j, o, h, n, ]|  xii|  20170909|20210909|
+---+--------------+-----+----------+--------+

edited Sep 4, 2018 at 8:07

answered Sep 4, 2018 at 7:19

hamza tuna

1,4971 gold badge13 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

megamind · Accepted Answer · 2018-09-04 07:11:43Z

0

nameList = [x for x in 'john']

answered Sep 4, 2018 at 7:11

megamind

6271 gold badge9 silver badges17 bronze badges

Comments

Samuel Kazeem · Accepted Answer · 2018-09-04 07:36:36Z

0

.tolist() turns a pandas series into a python list, so you should create a list first from the data and loop over the list created.

namelist=df['name'].tolist()
for x in namelist:
    print(x)

edited Sep 4, 2018 at 7:36

answered Sep 4, 2018 at 7:22

Samuel Kazeem

8051 gold badge9 silver badges16 bronze badges

1 Comment

hamza tuna Over a year ago

It is not a pandas dataframe.

Sachin Shetty · Accepted Answer · 2020-01-17 13:12:26Z

0

If you are doing this in spark scala (spark 2.3.1 & scala-2.11.8 ) Below code works. We will get an extra record with blank name hence filtering it .

import spark.implicits._ val classDF = spark.sparkContext.parallelize(Seq((1, "John", "Xii", "20170909", "20210909"))) .toDF("ID", "Name", "Class", "Start_Date", "End_Date")

classDF.withColumn("Name", explode((split(trim(col("Name")), ""))))
  .withColumn("Start_Date", to_date(col("Start_Date"), "yyyyMMdd"))
  .withColumn("End_Date", to_date(col("End_Date"), "yyyyMMdd")).filter(col("Name").=!=("")).show

answered Jan 17, 2020 at 13:12

Sachin Shetty

671 gold badge2 silver badges11 bronze badges

Collectives™ on Stack Overflow

Convert Column value in Dataframe to list

4 Answers 4

Comments

Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related