Decoding a string URL column in PySpark

Question

I am using Python 2.7 and Spark 2.2.0. I have created a data frame in PySpark which has a string column type and contains URLs.

df = spark.createDataFrame([('example.com?title=%D0%BF%D1%80%D0%B0%D0%B2%D0%BE%D0%B2%D0%B0%D1%8F+%D0%B7%D0%B0%D1%89%D0%B8%D1%82%D0%B0',)], ['url'])
df.show(1, False)

+-------------------------------------------------------------------------------------------------------+
|url                                                                                                    |
+-------------------------------------------------------------------------------------------------------+
|example.com?title=%D0%BF%D1%80%D0%B0%D0%B2%D0%BE%D0%B2%D0%B0%D1%8F+%D0%B7%D0%B0%D1%89%D0%B8%D1%82%D0%B0|
+-------------------------------------------------------------------------------------------------------+

To decode all the URLs in the column I tried to use urllib. I created a udf. I'm using it like this:

from pyspark.sql.types import StringType
from pyspark.sql.functions import udf
decode_url = udf(lambda val: (urllib.unquote(val).decode('utf8'), StringType()))

After applying a udf over my column I was expecting this :

+---------------------------------+
|url                              |
+---------------------------------+
|example.com?title=правовая+защита|
+---------------------------------+

But I got an error:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 18-33: ordinal not in range(128)

    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
    at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:234)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)
    at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$doExecute$1.apply(BatchEvalPythonExec.scala:144)
    at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$doExecute$1.apply(BatchEvalPythonExec.scala:87)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

If I take out some url from the column and try to decode it separately, it's working fine:

import urllib 
url='example.com?title=%D0%BF%D1%80%D0%B0%D0%B2%D0%BE%D0%B2%D0%B0%D1%8F+%D0%B7%D0%B0%D1%89%D0%B8%D1%82%D0%B0'
print urllib.unquote(url).decode('utf8')
example.com?title=правовая+защита

Tanjin · Accepted Answer · 2019-03-02 00:15:01Z

It seems like under the hood there is some strange encoding going on. Why don't you explicitly encode it yourself?

>>> decode_udf= udf(lambda val: urllib.unquote(val.encode('utf-8')).decode('utf-8'), StringType())
>>> df.withColumn('decoded_url', decode_udf('url')).show(truncate=False)
+-------------------------------------------------------------------------------------------------------+---------------------------------+
|url                                                                                                    |decoded_url                      |
+-------------------------------------------------------------------------------------------------------+---------------------------------+
|example.com?title=%D0%BF%D1%80%D0%B0%D0%B2%D0%BE%D0%B2%D0%B0%D1%8F+%D0%B7%D0%B0%D1%89%D0%B8%D1%82%D0%B0|example.com?title=правовая+защита|
+-------------------------------------------------------------------------------------------------------+---------------------------------+
``

Shubham Shah · Accepted Answer · 2021-12-21 07:32:48Z

3

If you want to do this in spark sql you can do like this:

Note : encoded field name is dctr

Input: im_segments%3Debejz4gv%2Ck1GmZLwg%2C8zY92P4g%2Cka6ee4eb%2CgPKlZXXb%2CqkVvpGk9%2Cky1ee4Dk%2CgvqKoW0b%2CgO5l6Zrk%2CgO5lGpdk%2CxkD6AYgm%2CgO5rENWk%2Cg7VrxvDb

Expected output: im_segments=ebejz4gv,k1GmZLwg,8zY92P4g,ka6ee4eb,gPKlZXXb,qkVvpGk9,ky1ee4Dk,gvqKoW0b,gO5l6Zrk,gO5lGpdk,xkD6AYgm,gO5rENWk,g7VrxvDb

Answer:

select distinct reflect('java.net.URLDecoder','decode', dctr , 'utf-8') as dctr from table

answered Dec 21, 2021 at 7:32

Shubham Shah

991 gold badge1 silver badge7 bronze badges

1 Comment

Al Johri Over a year ago

Nice! This works great in pyspark too: .withColumn('page', F.expr("reflect('java.net.URLDecoder','decode', uri, 'utf-8')"))

ZygD · Accepted Answer · 2023-09-26 00:14:09Z

0

Spark 3.5+

F.url_decode('url')

Spark 3.4+

F.expr("url_decode(url)")

Examples:

from pyspark.sql import functions as F

df.withColumn('url', F.url_decode('url')).show(1, 0)
# +---------------------------------+
# |url                              |
# +---------------------------------+
# |example.com?title=правовая защита|
# +---------------------------------+

df.withColumn('url', F.expr("url_decode(url)")).show(1, 0)
# +---------------------------------+
# |url                              |
# +---------------------------------+
# |example.com?title=правовая защита|
# +---------------------------------+

answered Sep 26, 2023 at 0:14

ZygD

24.8k41 gold badges106 silver badges144 bronze badges

Collectives™ on Stack Overflow

Decoding a string URL column in PySpark

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related