How to pass input parameter to AWS Glue Map.apply function

Question

I am working on an AWS Glue job where I have a function "some_function" that I want to apply on DynamicFrame dy_f, but I also want to pass an input param to some_function.

Map.apply(frame=products_combination, f=search)

where some_function's definition is:

some_function(record, k)

What I've tried so far: Works:

Map.apply(frame=products_combination, f=search) ##provided i'm not taking k as input in some_function def as well

What is giving error:

Map.apply(frame=products_combination, f=search(k=10))

This returns "TypeError: search() missing 1 required positional argument: 'record'"

How can I pass a parameter to Map.apply function? I have gone through the documentation [here] but couldn't find my solution there.1

bzu · Accepted Answer · 2022-01-21 15:03:34Z

3

You can always pass in additional parameters with lambda:

Map.apply(frame=products_combination, f=lambda record: search(record, k=10))

answered Jan 21, 2022 at 15:03

bzu

1,6041 gold badge10 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

MAC · Accepted Answer · 2020-12-15 13:49:22Z

0

I eventually used Spark udf instead of DynamicFrame.Map.apply. Worked for me.

answered Dec 15, 2020 at 13:49

MAC

1721 silver badge9 bronze badges

1 Comment

Brian Over a year ago

This requires you to leave the dynamicframe guardrails. any resolution that doesn't require to you use a dataframe?

Collectives™ on Stack Overflow

How to pass input parameter to AWS Glue Map.apply function

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related