All, I am new to Terraform and have few question. So terraform main.tf file is creating redshift cluster and there are python scripts in a different directory (same project) that needs to use the output of jdbc url. Wondering how we can use that dynamically in the Python code. In CI/CD both will run together and output of newly created redshift url must be used as next DAG which is a python file. Thanks
2 Answers
Without knowing your specific CI/CD environment, and without knowing if you are running these in the same CI/CD step or in separate steps, it is difficult to give an exact answer. Here are some general suggestions:
You could set the Terraform output value as an environment variable, that is then read by the Python script, like:
export MY_OUTPUT_VARIABLE=$(terraform output my_output_variable)
Or you could configure your Python script to take it as an input argument, like:
python my_python_script.py --input-var $(terraform output my_output_variable)
Or you could pipe the Terraform output to a file, which could be helpful if you are running Terraform and Python as separate CI/CD steps. Then the Python script could read the file directly, or you could wrap it in a shell script that reads the value from the file and passes it as input or an environment variable to the Python script, for example:
terraform output my_output_variable > terraform_output
then later:
python my_python_script.py --input-var $(cat terraform_output)
Comments
Another option would be to store the output as a Cloud Secret, then access the secret programmatically from your Python scripts.
For example (in my case, I'm adding a random suffix to the bucket name, which I want to be able to access):
resource "google_storage_bucket" "my_bucket" {
project = my_project_id
name ="my-bucket"
}
resource "google_secret_manager_secret" "my_bucket_name_secret" {
project = my_project_id
secret_id = "my-bucket-name"
}
resource "google_secret_manager_secret_version" "some_name" {
secret = google_secret_manager_secret.my_bucket_name.id
secret_data = google_storage_bucket.my_bucket.name
}
terraform output ...to get the output value from a state