Filtering rows in Azure Synapse/Data Factory dataflow

Question

I have been stuck for a bit trying to sort this out and for some reason it is not working.

I am reading a parquet file on a Pipeline in Synapse, after that I am selecting the columns I want to keep and before doing the sink, I want to add a filter (WHERE clause in SQL) to only keep specific rows.Here is the set up

Set up

Currently if I do a Data preview of the SelectColumns output I get data just fine.

data

And here is the definition of my filter

filter definition

But if I check the Data Preview for the filter task, it says no doutput data.

Can someone help me finding out whats going on please? I even tried a simpler filter SCTSTCO=='ABCD' just for testing and the same.

Am I just being too dumb?

Thanks a lot for the help

Rakesh Govindula · Accepted Answer · 2023-11-09 05:21:36Z

0

The above expression is working fine for me.

enter image description here

The expression CTSOCO=='ABCD' || CTSOCO=='ACRF' also working fine in the filter transformation.

If your data preview is resulting no rows, you might need to refresh the dataflow debug here.

enter image description here

answered Nov 9, 2023 at 5:21

Rakesh Govindula

11.9k2 gold badges5 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Rakesh Govindula Over a year ago

I don't think so. whatever might be file type, as per the input data preview, it should give required output.

kaintxu Over a year ago

Hi @Rakesh, thanks a lot for your explanation. That is exactly what I did, both with the IN function or just testing with ==. Pretty much the design you have there is exactly the same one that I have (please check the images I attached, for some reason you need to click on them, they are not showing directly) Could it not be working for me due to some type of configuration error? Many thanks

kaintxu Over a year ago

I have also reset the Data flow debug by turnting it off yesterday and just turned it back on and still no data coming out. I even tried the conditional split way to get rows you can do and still nothing. I am really at a loss here :S

Rakesh Govindula Over a year ago

I think by looking at your issue, its better to raise a support ticket because these kind of issues are not related to your input data or output data. This seems to an internal issue.

kaintxu Over a year ago

Thanks Rakesh, do they actually reply to tickets like this? I am afraid to raise a ticket and them telling me I am just being dumb in my dessing and that I am missing something.

|

Collectives™ on Stack Overflow

Filtering rows in Azure Synapse/Data Factory dataflow

1 Answer 1

10 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Your Answer

Sign up or log in

Post as a guest

Related