Apache Beam/Dataflow: Passing in attributes between Transforms

Question

Is there a way to pass meta data/attributes between PTransforms? This information is not part of a element, but should be accessible when processing elements. I have a constraint that this information cannot be passed in as a constructor argument.

Alex · Accepted Answer · 2018-10-30 01:19:35Z

2

Why can't it be passed in as a constructor argument?

Is the value unknown until you get to a particular Transform? If so, the you could retrieve it as an additional output, and then input it everywhere it is needed as a side-input by using pvalue.

https://beam.apache.org/documentation/programming-guide/#additional-outputs

https://beam.apache.org/documentation/programming-guide/#side-inputs

https://beam.apache.org/releases/pydoc/2.8.0/apache_beam.pvalue.html#apache_beam.pvalue.AsSingleton

answered Oct 30, 2018 at 1:19

Alex

5,32115 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Apache Beam/Dataflow: Passing in attributes between Transforms

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related