Optimizing repeated transformations in Apache Beam/DataFlow

Question

I wonder if Apache Beam.Google DataFlow is smart enough to recognize repeated transformations in the dataflow graph and run them only once. For example, if I have 2 branches:

p | GroupByKey() | FlatMap(...)
p | combiners.Top.PerKey(...) | FlatMap(...)

both will involve grouping elements by key under the hood. Will the execution engine recognize that GroupByKey() has the same input in both cases and run it only once? Or do I need to manually ensure that GroupByKey() in this case proceeds all branches where it gets used?

Pablo · Accepted Answer · 2018-07-10 21:50:16Z

2

As you may have inferred, this behavior is runner-dependent. Each runner implements its own optimization logic.

The Dataflow Runner does not currently support this optimization.

answered Jul 10, 2018 at 21:50

Pablo

11.1k1 gold badge49 silver badges72 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

kpax Over a year ago

Thank you, Pablo. May I ask if this optimization is on the roadmap for the DataFlow runner?

Temu Over a year ago

@00111100b, you might want to give a star to this Feature Request in order to prioritize it and maybe open your own one asking for whatever you consider.

kpax Over a year ago

@Temu thank you. I am getting "access denied" trying to follow your link. Is that expected? I am logged in with my Gmail account.

Temu Over a year ago

You are right, I sent you the wrong link. There is the good one.

Collectives™ on Stack Overflow

Optimizing repeated transformations in Apache Beam/DataFlow

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related