Loading csv data to bigquery using python in Terraform

Question

Read csv file and load it to bigquery through dataflow job - use python coding for this instead of templates How perform this task using terraform(GCP) anyone help

I trying to do it but not understanding what terraform script should I write for it

Terraform deploy resources, it runs nothing. Don't mix the usages! — guillaume blaquiere
– guillaume blaquiere, Commented Nov 14, 2022 at 10:25

Mazlum Tosun · Accepted Answer · 2022-11-13 21:10:03Z

1

It's not the responsability of Terraform to deploy a Dataflow job.

There is only a Terraform resource to instantiate a Dataflow template

You can deleguate this to your CI CD.

Example with Beam Python :

Develop the Job with Beam Python
Through your CI CD, deploy the Python Beam code to a Cloud Storage bucket
Run the Dataflow job and main file with Python command line

Example with Beam Java and mvn compile :

Develop the Job with Beam Java and Maven or Gradle
Through your CI CD, run mvn compile command to execute the Dataflow job

Example with Beam Java and a fat jar :

Develop the Job with Beam Java and Maven or Gradle
Through your CI CD, generate a fat jar
Deploy this fat jar to a Cloud Storage bucket
Run the Dataflow job and the Main inside the fat jar with java -jar command

Example with Beam Python and Airflow/Cloud Composer :

Develop the Job with Beam Python
Through your CI CD, deploy the Python Beam code to the Cloud Composer bucket with gcloud composer
In the Airflow code, uses BeamRunPythonPipelineOperator to instantiate the Dataflow job
Run the Airflow DAG to run the Dataflow job

Example with Beam Java and Airflow/Cloud Composer :

Develop the Job with Beam Java
Through your CI CD, generate a fat jar
Deploy this fat jar to a Cloud Storage bucket
In the Airflow code, uses BeamRunJavaPipelineOperator to instantiate the Dataflow job targeting on the path of the fat jar
Run the Airflow DAG to run the Dataflow job

answered Nov 13, 2022 at 21:10

Mazlum Tosun

6,6822 gold badges11 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

TEJAS Over a year ago

Hi Mazlum Tosun I have try it but not understanding which terraform scripts use for it so if u have any script for it can u provide please

Mazlum Tosun Over a year ago

Ok I am going to share with you some sample scripts.

TEJAS Over a year ago

Hi Mazlum Tosun , Thanks for ur help please share scripts

TEJAS Over a year ago

Hi Mazlum Tosun , Thanks for ur help please share scripts

Collectives™ on Stack Overflow

Loading csv data to bigquery using python in Terraform

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related