0

I am building a pipeline to move data from CosmosDB to Delta table. ADF supports change feed for CosmosDB by default ONLY IF I select Container as input in the source. Due to requirements I have to use Query as input and as a result I can't select change feed in the source. What's the alternative way to implement Change feed in data flow? I would like to be able to upsert (and possibly delete) records into the delta table.

Currently I am using the Alter row modifier and set Upsert to true() in the alter row conditions section. And this modifier is piped from the source data. Not sure if this is the correct implementation.

Improve this question

1 Answer 1

Reset to default
0

Due to requirements I have to use Query as input and as a result I can't select change feed in the source

Currently, the Change feed in only supported for container in ADF dataflow. You can raise a feature request about it for the query.

AFAIK, the only way to achieve your requirement can be apply the dataflow transformations after the source(with container).

You can use the transformations like Filter and Select to do the job that your query going to do.

enter image description here

You can apply the combination of these based on your query and transform to your sink.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.