42 questions
1
vote
1
answer
46
views
copy incremetal data from shopify using copy activity in azure synapse using version 2.0 linked service
currently we are fetching data from shopify using copy activity with version 1.0 shopify linked service. version 1.0 is going to get deprecated by the 3rd week of Oct 2025. version 1.0 supports query ...
- 224
0
votes
1
answer
88
views
Difference between checkpoints and change streams
I am trying to understand the difference between the two most common strategies of incremental data load.
What is the difference between a streaming checkpoint vs a change stream in Databricks Delta ...
- 3,312
2
votes
1
answer
153
views
How to do incremental refresh in power bi without duplication
I have a table like following in my SQL Database and is loaded into Power BI where the Last Update Date have been setup for Incremental Refresh with 2 days windows.
Primary Key
Col B
Last Update Date
...
- 542
0
votes
1
answer
73
views
Will BigQuery Incremental Transfer reload files when GCS lifecycle archives them?
I’m evaluating BigQuery Data Transfer Service to ingest daily files that land in a GCS bucket.
My goal:
Load each new daily file once into BigQuery.
After 30 days, archive files using a GCS ...
- 12.7k
0
votes
0
answers
142
views
how to implement incremental load of parquet files where source is azure blob and destination is azure data explorer
i have data in container of parquet files for every hour
in container i have 6 folder say a,b,c,d,e,f and same name table has been created in adx and in container folder a/year=2025/month=01/day=01/...
0
votes
1
answer
139
views
Business Central API to Azure SQL Database - ADF will not interact with or filter incoming API data
We are trying to incrementally load Business Central API data to an Azure SQL database in ADF. ADF only seems to allow you to copy the API data to the target source, however nothing happens when you ...
0
votes
1
answer
103
views
How to implement incremental load on ADF
We have 2 folders in adls source and destination. source folder has 4 file and destination has 3. How can we implement the Increment load in ADF. What all activities we have to perform to achieve ...
- 375
0
votes
1
answer
780
views
Is it a good practice to use update_timestamp for incremental loading in DBT with insert_overwrite strategy?
Suppose I have a dbt/bigquery model reading from a simple upstream model partitionned by day using "partition_date" and clustered by "update_timestamp" and that would look like
the ...
- 63
0
votes
0
answers
235
views
How to Use BigQuery Views for Incremental Data Loading in BI Tools When They Can’t Utilize _TABLE_SUFFIX?
I’m currently working with Google BigQuery and a BI tool that handles incremental data loading automatically. In BigQuery, I have a partitioned table where each partition is represented by a ...
- 61
0
votes
0
answers
211
views
Effective way to track source deleted records in incremental load in Azure Databricks
Scenario :
The source is snowflake and it is in readonly mode.
Implementing full load, saving maximum timestamp in databricks for incremental load.
But in source side apart from record updates, ...
- 197
0
votes
0
answers
40
views
Error code near the 'INSERT' section during incremental ETL insert
USE UsineFlex_DW
GO
DROP VIEW IF EXISTS vue_FaitProduction
GO
CREATE VIEW vue_FaitProduction
AS
SELECT
P.NoProduit,
U.NoUsine,
LP.DateProduction,
M.NoMachine,
LP.Quantite,
LP....
0
votes
1
answer
690
views
How to implement increment load in pentaho (spoon)
I want to implement increment load in pentaho. I have two tables in my OLTP and I want to apply left join them and drop them as single table in OLAP. OlTP and OLAP are in different database connection ...
- 1
-1
votes
2
answers
1k
views
Data Factory succeeds but no data at target
I am running an incremental process for multiple tables where I am using old watermark, new watermark, copy activity, and a stored procedure to update the watermark value. One of my table succeeds ...
- 1
0
votes
0
answers
589
views
How to perform incremental load in snowflake
I have a table T1 in Snowflake that get's truncated and loaded with data weekly. I have to create another table T2 where I should pass all the initial full load from T1 to T2. Then after each week ...
0
votes
1
answer
339
views
How to load data from github graphql using since like rest API
I have written a pipeline to load issues from GitHub to big query; I want to make it incremental, for example, load only the data from the last run to the present run; I tweaked the pipeline code to ...
- 15
0
votes
1
answer
694
views
Add additional header from previous activity to rest call in copy data activity
I've a pipeline which should sync data from a REST API source to a SQL table. There are 2 steps in this pipeline:
Get the last changed datum field from the data set in the previous run, so that I ...
- 29
0
votes
1
answer
1k
views
Handling Null Values in ADF Pipeline
I am performing an incremental load using timestamp as watermark column. I have few null values in the date column in my source. When I am replicating the data using copy activity, only rows whose ...
- 1
0
votes
2
answers
748
views
Incremental load in Azure Data Factory
I am replicating my data from Azure SQl DB TO Azure SQL DB. I have some tables with date columns and some tables with just the ID columns which are assigning primary key. While performing incremental ...
- 1
2
votes
0
answers
799
views
Duplicates in Snowflake Stream
With the setting SHOW_INITIAL_ROWS = TRUE, we created a stream on top of a view (which has many joins).
We created a Stored procedure with a single merge statement that ingests all of the data from ...
- 87
0
votes
1
answer
307
views
Azure Delta Load won't recognize Epoch timestamp (ms) as Watermark Column Name
I am trying to perform a delta load (incremental load) with Azure Data Factory from SQL Server to Blob Storage. My tables have an updateStamp column that is Epoch Time in milliseconds, numeric(19,0) ...
- 3
1
vote
1
answer
398
views
Is there a way to make the dbt_cloud_pr_xxxx_xxx a clone of an existing data?
so using dbt cloud, and having a run on every pull request, but my incremental models are fully refreshed since everything runs in a new db destination (dbt_cloud_pr_xxxxx_xxx) any way of solving this?...
- 3,761
0
votes
1
answer
69
views
Displaying images in gridview using paging
I have a gridview with incremental loading that displays book covers from a local folder.
XAML:
<GridView
x:Name="komikGridView"
DataFetchSize="18"
...
- 611
3
votes
1
answer
3k
views
while doing incremental using dbt i want to to aggregation if that row exist else insert
I am using DBT to incremental load data from one schema in redshift to another to create reports. In DBT there is straight forward way to incrementally load data with upsert. But instead of doing the ...
- 397
0
votes
0
answers
229
views
Incremental load in SSIS
I have a source table that has some millions of rows and the job runs the SSIS package to load the data daily.
Now the daily job is taking for hours to run the entire load due to several rows.
I want ...
- 33
4
votes
1
answer
5k
views
Delta Live Tables for Batch Incremental Processing
Is it possible to use Delta Live Tables to perform incremental batch processing?
Now, I believe that this code will always load all of the data available in the directory when a pipeline is run,
...
- 2,115
0
votes
1
answer
2k
views
Azure Synapse - Pipelines - Copy Data - incremental load without table
I am using a very simple architecture to copy data from an external source into Azure Data Lake Storage gen 2 and serve it to PowerBI via a Serverless pool (where I perform some aggregations).
For the ...
- 5
1
vote
1
answer
259
views
Displaying images in gridview using incremental loading
I have a gridview that displays 435 images on a local package. I tried using Incremental Loading.
XAML:
<GridView
x:Name="komikGridView" Loaded="...
- 611
2
votes
0
answers
916
views
SAP incremental data load in Azure Data Factory
I'm trying to implement an Extractor pipeline in ADF, with several Copy Data activities (SAP ERP Table sources). To save some processing time, I'd like to have some deltas (incremental load). What's ...
- 21
0
votes
1
answer
502
views
Aggregation in Azure Data Flow is Returning Invalid Value
I have created a data flow in Data Factory.
Step 1. Read the parquet file.
Step 2. Aggregate the file to get the Max(DateField)
Step 3. Use a derived column to write in a Value.
Step 4. Alter row task ...
0
votes
1
answer
2k
views
Custom Pagination of Rest API in Azure Data Factory
I would like to retrieve all results from Rest API endpoint.The URL has the below form
https://myapi.com/relativeapi?project=&repo=&prId=&page=&pageSize=&startTime=&endTime
By ...
0
votes
1
answer
896
views
Incremental load from multiple tables for join queries in Nifi
For Incremental load we will be using QueryDatabaseTable processor which extracts data incrementally from one table. For writing sql query which extracts data from multiple tables we are using ...
0
votes
2
answers
135
views
Microsoft PowerApps Excel incremental load
I'm very new to PowerApps and have some task of making the app which will involve incremental data load. Currently we have chosen Excel sheets as the data source. So the actual task was make the data ...
- 37
1
vote
1
answer
2k
views
ADF to Snowflake incremental load and streams
I am trying to load files from my Azure blob to Snowflake table incrementally. After which in snowflake, I put streams on that table and load the data to the target table.
I am unable to do ...
- 111
7
votes
1
answer
1k
views
Is there something like Glue "Bookmark" feature in spark which keeps track at job level?
I am looking to see if there is something like AWS Glue "bookmark" in spark. I know there is checkpoint in spark which works well on individual data source. In Glue we could use bookmark to ...
- 155
0
votes
2
answers
8k
views
Azure Data Factory Error: "incorrect syntax near"
I'm trying to do a simple incremental update from an on-prem database as source to Azure SQL database based on a varchar column called "RP" in On-Prem database that contains "date+...
- 3
0
votes
0
answers
243
views
REST API to Excel integration Azure Logic Apps
I have data extracted through REST API from one of my applications and store it in a spreadsheet. I am using Azure Logic apps to extract data based on the date range for incremental loading everyday. ...
- 1
0
votes
1
answer
844
views
How can I do incremental load into Elasticsearch using logstash+jdbc and Database Trigger
I have elasticsearch cluster and i am using logstash + jdbc to load the data.
I Would like to know is there any way that we can do a incremental load based on database trigger instead of depending on ...
0
votes
1
answer
412
views
Best way to replicate MongoDB NoSQL into SQL tables
How can i replicate (incremental load) MongoDB (NoSQL) to SQL tables.
We have a web-based solution that loading data into MongoDB. The data size is almost 1TB. We need to do BI Reporting in the Looker ...
- 49
0
votes
1
answer
1k
views
How to perform hourly incremental extracts from an oracle datasource using last_mod_dt (timestamp) in Airflow?
Need to do hourly table refresh/loads using Airflow/Python from an oracle datasource using last_modified_dt timestamp column.
In Airflow there's airflow.models.taskinstance API which exposes data from ...
- 407
0
votes
0
answers
50
views
selecting starting record from defined value
Hi stackoverflow community,
I would like to achieve the following result. I am doing incremental loading. Where i load the max & last values of the previous load grouped by M_ID into the query of ...
- 87
1
vote
4
answers
2k
views
Power BI Athena Incremental Refresh
I have been successfully using Power BI’s incremental refresh daily with a MySQL data source. However, I can't get this configured with AWS Athena, because seemingly the latter interprets the values ...
- 4,677
0
votes
2
answers
338
views
AzureDataFactory Incremental Load using Python
How do I create azure datafactory for incremental load using python?
Where should I mention file load option(Incremental Load:LastModifiedOn) while creating activity or pipeline??
We can do that ...
- 6,339