2,143 questions
1
vote
0
answers
23
views
Azure ODATA Source supporting compression
I‘m using an Azure Data Factory with an ODATA source from an external system. While everything works fine, I noticed the slow performance. So I checked this with postman and spotted immediately, that ...
- 71
0
votes
1
answer
114
views
ADF Oracle Linked Service 2.0 performance issue with CLOB column [closed]
For a project I’m upgrading the Oracle linked service from V1 to V2. Now we have a big performance issue.
We are loaded data from an Oracle datasource to Parquet-files on a BLOB. For 1 specific job we ...
- 11
0
votes
1
answer
74
views
Azure Data Factory how to conditionally update cell based on condition of another cell in ADF (something like 'if(cell>x, cell*y, cell'?)
I am relatively new to the ADF, I am trying to achieve following:
Have csv file:
price
new price
102
91
1145
I need to apply to both "new price" cells following function:
if(A2>1000, A2*1....
1
vote
2
answers
69
views
Oracle ADF how to pass dynamic values into JS inside <af:resource type="javascript">
In Oracle ADF I am having
<af:resource type="javascript">
var chatSettings = {
url: "google.com",
id: "test"
}
function init(){
...
}
</af:resource>
<...
- 630
0
votes
1
answer
76
views
My parquet column isn't automatically splitting out and is all in 1 column
parquet fileI am trying the copy function in ADF and pulling in a Parquet file from an S3 Bucket.
When I try to map the columns out it shows they are all grouped together in one column as so:
(Screen ...
0
votes
0
answers
38
views
Adf Faces components are not rendering in Design View
In Oracle JDeveloper 12.2.1.4.0, ADF Faces components are not rendering in Design View. The page only displays component names instead of their UI structure. However, when running the application (Run)...
- 41
1
vote
0
answers
18
views
ADF - Row count validation before and after dataflow
I am wanting to verify within the pipeline that the row count before and after a dataflow matches, accounting for any rows which have been filtered out.
For example:
a) rowsRead = 500 b) rowsWritten = ...
- 11
0
votes
1
answer
30
views
ADF Process XML
I get a response as xml formatted string from some web activity in ADF
<Transactions>
<Transaction
xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">
<...
- 470
0
votes
1
answer
115
views
Is there a way to keep ADF and Database versions in sync?
I've finally made the jump from on prem to cloud and have set up an initial data warehouse with a couple of basic stars in the schema. I'm near enough done on the Data Factory data pipelines to load ...
- 1
0
votes
0
answers
64
views
Populate View Objects Programmatically
I am new to ADF and using JDeveloper 12.2.1.4.0.
I have the following use case.
I have a view object DepartmentVo defined on departments table(DepartmentEO) and have the REST API's exposed on it.
The ...
- 33
0
votes
1
answer
250
views
ADF, utcNow() convert to DateTime
in ADF I have a pipeline parameter that is set to utcNow(). I need to pass this parameter to a stored procedure activity that is expecting a datetime value in a parameter. I am trying to convert the ...
- 97
0
votes
0
answers
31
views
errror "Failed to access the data source." with all REST API URLs
https://drive.google.com/file/d/1CFMONqqhEk9jp-HwQLBzs_MuKrNhKfEl/view?usp=drive_link
I am working in Oracle ADF and attempting to configure a REST API for my application. However, every time I try to ...
0
votes
0
answers
38
views
ADF WebActivity results in "Invalid_client" error
I am using ADF's web activity to get the authentication token.
When in Postman, it works correctly for a request looking like this
and with the Headers
Then, I am going to implement this in the ...
- 470
1
vote
0
answers
68
views
how to get multi selected row values from a table of fusion screen using sandbox page composer
We have a requirement from Oracle fusion process requisition screen, when user selects the multiple rows from search results of table in process requisition screen. we need requisition number values ...
0
votes
1
answer
102
views
How to build expression in ADF from json using parameterized variable?
I have one activity output like this
{
"firstRow": {
"mapping": "{\"CHRSCRT\":\"CHRT\",\"CST\":\"CUSTOM\"}"
}
}
...
0
votes
1
answer
188
views
Azure Data Factory - SharePoint List connection issue
I'm connecting ADF to sharepoint site to collect data from multiple sharepoint lists.
Connector is working perfectly fine - ADF can get data from 2 lists stored on this sharepoint but one of them is ...
- 1
0
votes
0
answers
228
views
SonarQube Still Showing Blocker Bug on Dashboard After Fix Has Been Applied
I’m facing an issue with SonarQube where a blocker bug is still reported on the dashboard for a branch I'm working on, even though the issue has been fixed in the code and it works fine in my local ...
- 86
0
votes
1
answer
123
views
How to run SCOPE scripts in ADF with C# nuget
Looking at the documentation I see no example of how to run this script against ADLS instance; is there an example of what activity type etc. should be used for this functionality?
- 57
0
votes
1
answer
229
views
Mapping array column to dynamic in ADX in ADF mapping
I have array of numbers in the dataframe, which i want to map to dynamic column in kusto, But kusto does not support array mapping in ADF yet. Is there any alternative way to achieve the same.
I am ...
- 15
0
votes
0
answers
69
views
Unable to identify the errors in the SQL query
I am using this query in my ADF pipeline:
{@if(
equals(pipeline().parameters.IsUpsert,false),pipeline().parameters.SourceSQL,
if(
equals(pipeline().parameters....
0
votes
1
answer
36
views
Error related to Automated checks and frame suppression are incompatible FrameBustingUtils
This is the problem, in the Oracle ADF development environment, I did a start IntegratedWebLogicServer. I went to the WebLogic console, to the test page of my application and a white screen appears. ...
- 52
0
votes
0
answers
142
views
Run GraphQL query in ADF dataflow
I am trying to create an ADF dataflow to extract data from API using GraphQL query. It has nested structures, and I believe copy activity does not support nested structures.
I tried using dataflow, ...
- 13
0
votes
1
answer
422
views
Using Lookup Activity to execute stored procedure using script by passing lookup value
Summary: in ADF pipeline, using lookup to select all the data and passing each row at a time to ForEach activity. Inside ForEach activity, first I have copy data activity where I read value from ...
- 59
0
votes
0
answers
52
views
adf copy data tool - mongo db filtering
I have a collection in Mongo called 'collectionX'. I want to query whole data, except I want to filter out by event_date that is after July 1st (event, creationdate - unix milisecond timestamp). ...
0
votes
1
answer
113
views
Processing Event Hub XML file Data using ADF to store on SQL table
We have 2 systems 1 system send the data in XMl file and store into Azure Event hub, I need to process XML file data & store it into a SQL Server table.
Multiple files are sent in event hub in 1 ...
2
votes
1
answer
392
views
ADF/Databricks : The table is not delta format
Getting error, when trying to copy data from CSV (Azure Blob) to Databricks Delta table. Using ADF Copy activity, Blob Storage as Source, Databricks Delta Lake as Sink. Getting error that the table is ...
0
votes
1
answer
108
views
Unzip zipfile which contains subfolders stored in blob storage using Azure Data Factory
I want to extract all files and folders from a zipfile which is stored on an Azure Blob Storage. The extracted data should be stored in a output folder.
I have a zipfile stored in Azure Blob Storage. ...
- 13
0
votes
1
answer
160
views
Azure Data Factory, cant use Excel Dataset with Wildcard
My Problem:
I have a Data Lake Gen 2 Storage Account with an import directory which contains .xlsx files.
Storage Account
Now im trying to create a Dataset pointing to this directory. The Dir will ...
- 1
0
votes
1
answer
137
views
Azure Copy Activity issue
My project has Maria DB as on-prem source system, it is connected with ADF via Self hosted integration runtime. I want to copy only the required fields from 100 tables of MariaDB to Data Lake but I ...
0
votes
1
answer
118
views
ADF Compare list from sql query to getMetaData output
I have a SQL table that contains the columns FileNamePrefix and TableName. I have files in blob storage that begin with the FileNamePrefix with some datetime in the name:
SQL table:
FileNamePrefix ...
- 57
0
votes
0
answers
71
views
Stored procedure automatically triggered from ADF can't update or insert tables
I have a SP as below. When this is trigged from ADF using scheduled trigger it is not inserting the data as per the conditions. However, when this is triggered manually from ADF it is working as ...
1
vote
1
answer
249
views
Azure ADF: Losing Value of Pipeline Variable when re-running an Activity
In ADF, I have a demo pipeline with three activities and one pipeline variable called parentRunId:
pipeline variable
The first activity is a "Set Variable" activity and generates a UUID for ...
- 35
0
votes
1
answer
221
views
on adf mapping data flow while writing data from adls to synapse dedicated pool i am facing issue while writing error to adls
on adf mapping data flow while writing data from adls to synapse dedicated pool i am facing issue while writing error to adls Job failed due to reason: at Sink 'sink1': Exception is happened while ...
0
votes
1
answer
240
views
dynamic conversion of Decimal(38,18)data type that converts from default decimal datatype in ADF
ADF copy activity by default converts decimal datatype to decimal(38,18) , how can I overcome it to use original datatype decimal as source for multiple tables dynamically with different schemas for ...
-1
votes
1
answer
127
views
why does my HTML to ADF script exiting with code 0?
I am trying to create a script that will convert HTML into Jira's ADF format. When I try to test the script using Visual Studio 2019 it just gives this output - "The program '[36520] HTMLtoADF....
- 11
0
votes
1
answer
161
views
How to parametrise the Data flow source 1 and source 2 in the same SQL DB but two different tables
I am building a Data flow pipeline which will call control flow pipeline.
The solution I am trying to achieve is to implement the Fuzzy logic for two master table to find relevant matching.
As show on ...
1
vote
1
answer
747
views
How to write ADF dynamic expression with SQL statements in multiple levels
I am trying to write ADF dynamic content for filter activity where I need to filter files, having dates in their name, coming from get metadata activity by multiple conditions. I am writing a SQL ...
- 23
1
vote
0
answers
32
views
Issue in af:showDetailItem under panelAccordion component not getting the label after migrating source code from 12.2.1.3 to 12.2.1.4 J developer
Problem Description
Our code is running in 12.2.1.3.0 Jdeveloper and now deployed upgraded version on the 12.2.1.4.0 weblogic server
due to this we faced issue in af:showDetailItem under ...
0
votes
1
answer
68
views
Why Get-AzDataFactoryV2PipelineRun doesn't return InProgress pipeline?
I am running Azure Function App with the following PowerShell script to get ADF pipeline runs. It is returning all history of pipeline runs, but it is skipping the Active( In Process) run. It is ...
0
votes
1
answer
464
views
Azure Data Factory Copy Data HTTP source zip file and sink as azure blob storage, asking to create two copy activities to work around it
I am getting this error message when trying to copy data from a HTTP source and my sink destination is an azure blob storage
I have a simple setup. I am visting the URL anonymously as I do not require ...
user11090827
0
votes
1
answer
720
views
ADF Pipeline Cost tagging for team based seperation
We are ingesting data from different resources to our data lake with ADF pipelines and have hundreds of pipelines (under 1 resource group). Cost per pipeline is already activated and I'm embedding ...
0
votes
1
answer
174
views
How to Open a JEE, STRUTS, and Oracle ADF Project Without .jws or .jpr Files in Oracle JDeveloper Studio?
Good afternoon,
I've encountered a challenge with a project handed to me. It's structured with JEE, STRUTS, and Oracle ADF. Initially, I attempted to work with it in IntelliJ. However, upon reviewing ...
- 1
0
votes
1
answer
1k
views
Azure Data Factory Private Endpoint provision fails without explanation
I am facing an issue when creating a private endpoint to Azure SQL Server in ADF. It always fails no matter what settings of the SQL server I choose. I tried creating the endpoint separately from the ...
0
votes
1
answer
136
views
How to verify the given node exists and read the value from azure data factory
I have a lookup where I am performing a query to get the employee_no
SELECT employee_no FROM HRS.EMPLOYEE_MASTER WHERE SOC_SEC_NO = '@{variables('FE_SoSecNo')}' this might return the data as follows
{
...
- 8,697
0
votes
1
answer
280
views
HASHBYTES in Azure Data Factory
So we found this strange use case where result of function HASHBYTES with MD5 gives different output in COPY Data vs. ADF DataFlow using (MD5 function).
So Copy Data uses (allows) to leverage ...
0
votes
0
answers
113
views
used query is not supported in odbc connector: Received this error while running through ADF - Database is Snowflake
Query is insert statement along with with as and select statement. It was running good previously but suddenly in production it errored out with this message. Couldn't find the reason behind it
There ...
- 3
1
vote
0
answers
150
views
How to resolve problem with Cookie JSESSIONID in Oracle ADF?
I have an application written in Oracle Adf. This application has a vulnerability
"The timestamp was disclosed by the application/web server - Unix"
The problem is in this Set-Cookie line:
...
- 52
0
votes
1
answer
591
views
Synapse/ADF Pipeline to Make HTTP API Request for every record in a database table
I have a table with 120,000 records in it. This will be my source table.
For each record in the table, I want to send a request to a REST API and then make a stored procedure call. There are reasons ...
- 1,535
1
vote
0
answers
139
views
ADF Filter Activity throwing error - Property selection is not supported on values of type 'String'
I have an ADF pipeline in which there is a Web Activity called Web1 that pulls JSON data from a REST API. Below is the sample data:
I am passing this data to a filter activity where I am trying to ...
- 11
0
votes
1
answer
1k
views
How to pass Variables return value in adf pipelines
In child pipeline1 I passed if condition in that true statement variable 1 is executed and in false statement variable2 is executed.
Now in ChildPipeline2 parameter I am passing value is
@or(activity('...
