Drop duplicate column with same values from spark dataframe

Question

Code:

import sparkSession.sqlContext.implicits._
val table_df = Seq((1, 20, 1), (2, 200, 2), (3, 222, 3), (4, 2123, 4), (5, 2321, 5)).toDF("ID", "Weight", "ID")
table_df.show(false)

Input:

+---+------+---+
|ID |Weight|ID |
+---+------+---+
|1  |20    |1  |
|2  |200   |2  |
|3  |222   |3  |
|4  |2123  |4  |
|5  |2321  |5  |
+---+------+---+

Expected Output:

+---+------+
|ID |Weight|
+---+------+
|1  |20    |
|2  |200   |
|3  |222   |
|4  |2123  |
|5  |2321  |
+---+------+

I am using drop in following way

table_df.drop("ID").show(false)

This dropping both of the "ID" columns. How can I drop duplicated second column "ID" here?

you could rename the column first and then drop the newly named column. — Michael Heil
– Michael Heil, Commented Jul 16, 2020 at 12:20

suresiva · Accepted Answer · 2020-07-16 13:39:26Z

2

You may use the Dataframe map method to trim the duplicate ID column as given below,

table_df.map(row => (row.getInt(0),row.getInt(1))).toDF("ID","Weight").show() 


+---+------+
| ID|Weight|
+---+------+
|  1|    20|
|  2|   200|
|  3|   222|
|  4|  2123|
|  5|  2321|
+---+------+

New schema will be as below,

table_df.map(row => (row.getInt(0),row.getInt(1))).toDF("ID","Weight").schema.treeString

root
 |-- ID: integer (nullable = false)
 |-- Weight: integer (nullable = false)

answered Jul 16, 2020 at 13:39

suresiva

3,1831 gold badge20 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Shantanu Kher · Accepted Answer · 2020-07-16 21:44:22Z

0

You can drop the column after renaming the particular instance, that you intend to drop, of this column.

Sample code meeting this requirement -

val table_df = Seq((1, 20, 1), (2, 200, 2), (3, 222, 3), (4, 2123, 4), (5, 2321, 5)).toDF("ID", "Weight", "ID")

val newColNames = Seq("ID","Weight","X1")

table_df.toDF(newColNames:_*).show(false)
+---+------+---+
|ID |Weight|X1 |
+---+------+---+
|1  |20    |1  |
|2  |200   |2  |
|3  |222   |3  |
|4  |2123  |4  |
|5  |2321  |5  |
+---+------+---+


table_df.toDF(newColNames:_*).drop("X1").show(false)
+---+------+
|ID |Weight|
+---+------+
|1  |20    |
|2  |200   |
|3  |222   |
|4  |2123  |
|5  |2321  |
+---+------+

answered Jul 16, 2020 at 21:44

Shantanu Kher

1,0841 gold badge9 silver badges14 bronze badges

Collectives™ on Stack Overflow

Drop duplicate column with same values from spark dataframe

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related