Map n string values to int column

Question

I have a dataframe like this where a column have alphanumerical values. Now I want to map all those values to some integer and also have some dictionary where I have those mappings so I can use them later. They're not all unique values.

column
  a
  c1
  3vc
  c1
  .
  .
  .
  n

Output:

column
  1
  2
  3
  2
  .
  .
  .
  n

过过招 · Accepted Answer · 2022-06-17 08:12:58Z

3

If there is no special requirement for the mapping order, you can directly sort the rankings according to the character order to map to integers.

data = [
    ('a',),
    ('c1',),
    ('3vc',),
    ('a',),
    ('c1',),
    ('fd',)
]
df = spark.createDataFrame(data, ['column'])
df = df.selectExpr('column', 'dense_rank() over (order by column) as int_col') \
    .withColumn('mapping', F.expr('map(column, int_col)'))
df.show(truncate=False)

# +------+-------+----------+
# |column|int_col|mapping   |
# +------+-------+----------+
# |3vc   |1      |{3vc -> 1}|
# |a     |2      |{a -> 2}  |
# |a     |2      |{a -> 2}  |
# |c1    |3      |{c1 -> 3} |
# |c1    |3      |{c1 -> 3} |
# |fd    |4      |{fd -> 4} |
# +------+-------+----------+

answered Jun 17, 2022 at 8:12

过过招

4,3372 gold badges7 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Chris_007 Over a year ago

Thnaks man, this was really helpful!

Chris_007 Over a year ago

Hey, this does change my partition to just 1, and I can repartition later but that's creating problem while saving as rank step has just one partition and it's not dividing the stage into multiple task. Is there any other way I can do it.

Collectives™ on Stack Overflow

Map n string values to int column

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related