How to use a output of a Databricks activity in future activity inside ADF?

Question

I have a Databricks activity in ADF and I pass the output with the below code:

 dbutils.notebook.exit(message_json)

Now, I want to use this output for the next Databrick activity.

As my search, I think add the last output into base parameters in the second activity. Am I right?

and other questions, How can I use this output inside the Databrick notebook?

Edited: The output is a JSON as the below screenshot.

@AliHasan I added at end of the question. The output is JSON, as in the above screenshot. I want to load this JSON in the filter notebook. — Shadi
– Shadi, Commented Jul 28, 2021 at 11:40
You can do this in databricks, save file in temp table if possible or in dbfs and then again read file in next databricks notebook. As you also have intentions to use this in databricks notebook. — Ali Hasan
– Ali Hasan, Commented Jul 28, 2021 at 11:51
@AliHasan I'm looking for a solution using the output of the last activity. — Shadi
– Shadi, Commented Jul 28, 2021 at 20:14

Utkarsh Pal · Accepted Answer · 2021-07-30 11:28:21Z

3

As per doc, you can consume the output of Databrick Notebook activity in data factory by using expression such as @{activity('databricks notebook activity name').output.runOutput}.

If you are passing JSON object you can retrieve values by appending property names.

Example: @{activity('databricks notebook activity name').output.runOutput.PropertyName}.

I reproduced the issue and it's working fine.

Below is the sample notebook.

import json
dates = ['2017-12-11', '2017-12-10', '2017-12-09', '2017-12-08', '2017-12-07']
return_json = json.dumps(dates)
dbutils.notebook.exit(return_json)

This is how the Notebook2 Activity Seeting looks like:

Pipeline ran successfully.

answered Jul 30, 2021 at 11:28

Utkarsh Pal

4,5781 gold badge8 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Shadi Over a year ago

Thanks for your response and looks great. Other than my question, how can I consume this output to notebook2? Could you share the sample code? for example in notebook2 if the date == '2017-12-10' print yes.

Utkarsh Pal Over a year ago

you can use the If Condition activity in ADF and based on True/False can trigger another activity for further process in the pipeline. refer learn.microsoft.com/en-us/azure/data-factory/…

Shadi Over a year ago

Thanks, I mean use notebook1's output in the body of notebook2. If Condition is a sample operation. I try to clarify my request.

Shadi Over a year ago

I mean how to use return_json in the body of notebook2?

Utkarsh Pal Over a year ago

unable to understand the exact requirement. do you want to write the return_json in notebook2? or want to iterate json objects? or anything else?

|

Collectives™ on Stack Overflow

How to use a output of a Databricks activity in future activity inside ADF?

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related