I would like to create column with sequential numbers in pyspark dataframe starting from specified number. For instance, I want to add column A to my dataframe df which will start from 5 to the length of my dataframe, incrementing by one, so 5, 6, 7, ..., length(df).
Some simple solution using pyspark methods?
You can do this using range
df_len = 100
freq =1
ref = spark.range(
5, df_len, freq
).toDF("id")
ref.show(10)
+---+
| id|
+---+
| 5|
| 6|
| 7|
| 8|
| 9|
| 10|
| 11|
| 12|
| 13|
| 14|
+---+
only showing top 10 rows
Three simple steps:
from pyspark.sql.window import Window
from pyspark.sql.functions import monotonically_increasing_id,row_number
df =df.withColumn("row_idx",row_number().over(Window.orderBy(monotonically_increasing_id())))
Although the question was asked long time ago, I though I could share my solution which I found very convenient. Basically to add a column of 1,2,3,... you can simply add first a column with constant value of 1 using "lit"
from pyspark.sql import functions as func
from pyspark.sql.window import Window
df= df.withColumn("Id", func.lit(1))
Then apply a cumsum (unique_field_in_my_df is in my case a date column. Probably you can also use the index)
windowCumSum = Window.partitionBy().orderBy('unique_field_in_my_df').rowsBetween(Window.unboundedPreceding,0)
df = df.withColumn("Id",func.sum("Id").over(windowCumSum))
This worked for me. This creates sequential value into the column.
seed = 23
df.withColumn('label', seed+dense_rank().over(Window.orderBy('column')))
df = df.rdd.zipWithIndex().toDF(cols + ["index"]).withColumn("index", f.col("index") + 5)wherecols = df.columnsandfrefers topyspark.sql.functions. But you should ask yourself why you're doing this, bc almost surely there's a better way. DataFrames are inherently unordered, so this operation is not efficient.max(id) + spark_func.row_number().over(Window.orderBy(unique_field_in_my_df)