PySpark. Multi-dataframe operations

Question

I have the following two dataframes

Catalog:
+--------+-------+
| Type   | Value |
+========+=======+
| Cat    | 3     |
+--------+-------+
| Dog    | 2     |
+--------+-------+
| Goose  | 1     |
+--------+-------+

And

+----+-------+----------+
| ID | ITEM  | QUANTITY |
+====+=======+==========+
| 1  | CAT   | 10.0     |
+----+-------+----------+
| 1  | DOG   | 1.0      |
+----+-------+----------+
| 1  | GOOSE | 0.1      |
+----+-------+----------+
| 2  | CAT   | 0.01     |
+----+-------+----------+
| 2  | DOG   | 0.001    |
+----+-------+----------+
| 3  | GOOSE | 0.0001   |
+----+-------+----------+

My goal is to create the following new column

+----+-------+----------+--------+
| ID | ITEM  | QUANTITY | Value  |
+====+=======+==========+========+
| 1  | CAT   | 10.0     | 30     |
+----+-------+----------+--------+
| 1  | DOG   | 1.0      | 2      |
+----+-------+----------+--------+
| 1  | GOOSE | 0.1      | 0.1    |
+----+-------+----------+--------+
| 2  | CAT   | 0.01     | 0.03   |
+----+-------+----------+--------+
| 2  | DOG   | 0.001    | 0.002  |
+----+-------+----------+--------+
| 3  | GOOSE | 0.0001   | 0.0001 |
+----+-------+----------+--------+

Using PySpark, I need to multiply (or in some cases divide) the values in the quantity column by the values in the Value column as matched by the item/type?

murtihash · Accepted Answer · 2020-03-22 18:50:18Z

df.show()
df1.show()

+-----+-----+
| Type|Value|
+-----+-----+
|  Cat|    3|
|  Dog|    2|
|Goose|    1| #df
+-----+-----+

+---+-----+--------+
| ID| ITEM|QUANTITY|
+---+-----+--------+
|  1|  CAT|    10.0|
|  1|  DOG|     1.0|
|  1|GOOSE|     0.1|
|  2|  CAT|    0.01|
|  2|  DOG|   0.001| #df1
|  3|GOOSE|  1.0E-4|
+---+-----+--------+

You can do join on upper(Type) because ITEM is all upper case, then create new column value by * multiply.

df1.join(df,F.expr("""ITEM=upper(Type)""")).drop("Type")\
   .withColumn("Value", F.col("Value")*F.col("QUANTITY")).orderBy("ID","ITEM").show(truncate=False)

+---+-----+--------+------+
|ID |ITEM |QUANTITY|Value |
+---+-----+--------+------+
|1  |CAT  |10.0    |30.0  |
|1  |DOG  |1.0     |2.0   |
|1  |GOOSE|0.1     |0.1   |
|2  |CAT  |0.01    |0.03  |
|2  |DOG  |0.001   |0.002 |
|3  |GOOSE|1.0E-4  |1.0E-4|
+---+-----+--------+------+

Collectives™ on Stack Overflow

PySpark. Multi-dataframe operations

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related