Ignore nulls columns while making an array with struct

Question

I added a column to a dataframe which is an array of other columns.

How can I ignore the null values when I construct the column name_source? For example the line for Robert would show [internet, Robert] instead of [internet, Robert,]. The one with null name would show [internet,65878] instead of [internet,, 65878]

Please let me know the answer given with filter works for you. — Felix K Jose
– Felix K Jose, Commented Jul 17, 2021 at 0:13

Felix K Jose · Accepted Answer · 2021-07-17 04:18:41Z

1

You can achieve the same as follows:

df = df.withColumn('name_source', expr('filter(name_source, x -> x is not null)'))

Scala:

df.select(filter(col("name_source"), x => x.isNotNull))

edited Jul 17, 2021 at 4:18

answered Jul 17, 2021 at 0:11

Felix K Jose

9029 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Learn Hadoop Over a year ago

can you please provide in scala version

Felix K Jose Over a year ago

@LearnHadoop I have updated the answer, could you check whether that helps

Erlinska · Accepted Answer · 2021-07-16 20:59:15Z

0

Assuming you want to create your new column from all the ones in the initial dataframe:

df["name_source"] = df.apply(lambda x : x.dropna().values, axis=1)

answered Jul 16, 2021 at 20:59

Erlinska

4335 silver badges16 bronze badges

Comments

wwnde · Accepted Answer · 2021-07-17 01:08:35Z

0

Array all the rows and the array except the None after you cast it to string

df.withColumn("name_source",array(*[c for c in df.columns ])).withColumn("name_source",array_except(col('name_source'),array(lit(None).cast('string')))).show(truncate=False)

answered Jul 17, 2021 at 1:08

wwnde

26.7k6 gold badges21 silver badges38 bronze badges

Collectives™ on Stack Overflow

Ignore nulls columns while making an array with struct

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related