display DataFrame when using pyspark aws glue

Question

how can I show the DataFrame with job etl of aws glue?

I tried this code below but doesn't display anything.

df.show()

code

datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "flux-test", table_name = "tab1", transformation_ctx = "datasource0")
sourcedf = ApplyMapping.apply(frame = datasource0, mappings = [("id", "long", "id", "long"),("Rd.Id_Releve", "string", "Rd.Id_R", "string")])
 sourcedf = sourcedf.toDF()
 data = []
 schema = StructType(
[
    StructField('PM',
        StructType([
            StructField('Pf', StringType(),True),
            StructField('Rd', StringType(),True)
    ])
    ),
    ])
 cibledf = sqlCtx.createDataFrame(data, schema)
 cibledf = sqlCtx.createDataFrame(sourcedf.rdd.map(lambda x:    Row(PM=Row(Pf=str(x.id_prm), Rd=None ))), schema)
 print(cibledf.show())
 job.commit()

Can you verify if your source has data.Try printing Dynamic Frame contents before converting to data frame. — Prabhakar Reddy
– Prabhakar Reddy, Commented Dec 25, 2019 at 6:55
@Yuva yep I tried .. can you show me where the df have to be shown?? the interface? thank you — ceo
– ceo, Commented Dec 27, 2019 at 9:05

Yuva · Accepted Answer · 2019-12-27 14:46:02Z

4

In your glue console, after you run your glue job, in job listing there would be a column for Logs / Error logs.

Click on the Logs and this would take you to the cloudwatch logs associated to your job. Browse though for the print statement.

also please check here: Convert dynamic frame to a dataframe and do show()

ADDed working/test code sample

Code sample:

zipcode_dynamicframe = glueContext.create_dynamic_frame.from_catalog(
       database = "customer_db",
       table_name = "zipcode_master")
zipcode_dynamicframe.printSchema()
zipcode_dynamicframe.toDF().show(10)

Screenshot for zipcode_dynamicframe.show() in cloudwatch log:

edited Dec 27, 2019 at 14:46

answered Dec 27, 2019 at 9:42

Yuva

3,2139 gold badges48 silver badges76 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ceo Over a year ago

I did exactly what you said . didn't find data frame in the log. Thank you

Yuva Over a year ago

added code snippet and a screenshot from the cloud watch log...the logs by default would be collapsed, and you have expand and look out for ur df.show().

Collectives™ on Stack Overflow

display DataFrame when using pyspark aws glue

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related