0

i have data in container of parquet files for every hour in container i have 6 folder say a,b,c,d,e,f and same name table has been created in adx and in container folder a/year=2025/month=01/day=01/hour=00/.parquet file this is how data is stored but if i send every hour data directly i am creating a lot of data every hour which is mostly same in azure data explorer but a lot of redundant information is getting collected what i want is a strategy by which we only store the data whenever there is some changes in the data which is coming from parquet.

i have a solution using staging table but again i will be storing a lot of data in staging table so no use and i want the automated solution.

please suggest the best tool and strategy.

Improve this question
2
  • Could you provide a sample structure of the directory level and file demonstrating how you are processing your environment? Commented Mar 26 at 12:38
  • in blob blob container data is being stored as year->month->day->hour Commented Mar 27 at 5:10
Related questions
1
Azure Synapse - Incremental Data Load
0
How to deal with duplicate insertion in Azure Data Explorer
0
Azure Data Factory - run script on parquet files and output as parquet files
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.