Convert pyspark dataframe to dynamic dataframe

Question

I have a pyspark dataframe. I was able to convert dynamic dataframe to spark dataframe by persons.toDF(). I want to convert the spark dataframe again back to dynamic dataframe in pyspark.I wanted to cast my column to timestamp and again convert it to dynamic dataframe to resolveChoices. Please help me

Sergii Sholovii · Accepted Answer · 2021-04-16 12:22:12Z

4

at least you need pyspark.context, awsglue.context and awsglue.dynamicframe There is example :

from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.dynamicframe import DynamicFrame

sc = SparkContext()
glueContext = GlueContext(sc)

NewDynamicFrame = DynamicFrame.fromDF(persons, glueContext, "nested")

"persons" is your DataFrame

Please check following links :

answered Apr 16, 2021 at 12:22

Sergii Sholovii

614 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Harsh Bafna · Accepted Answer · 2019-09-24 08:14:28Z

3

You can create a dynamic frame from dataframe using the fromDF function.

Basic Syntax

dyf = fromDF(dataframe, glue_ctx, name)

where,

dataframe – The Apache Spark SQL DataFrame to convert (required).
glue_ctx – The GlueContext Class object that specifies the context for this transform (required).
name – The name of the resulting DynamicFrame (required).

Reference : Dynamic frame from dataframe

answered Sep 24, 2019 at 8:14

Harsh Bafna

2,2341 gold badge17 silver badges26 bronze badges

Collectives™ on Stack Overflow

Convert pyspark dataframe to dynamic dataframe

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related