Bash script inside Cloudformation

Question

I am trying to deploy a Sagemaker Lifecycle with AWS CloudFormation.
The Lifecycle is importing ipynb notebooks from s3 bucket to the Sagemaker notebook instance.
the bucket name is specified in the parameters, I want to use it in a !Sub function inside the bash script of the Lifecycle. The problem is that the CF runs first on the template and tries to complete its own functions (like !Sub) and then the scripts upload as bash script to the Lifecycle.

This is my code:

LifecycleConfig:
  Type: AWS::SageMaker::NotebookInstanceLifecycleConfig
  Properties: 
   NotebookInstanceLifecycleConfigName: !Sub
   - ${NotebookInstanceName}LifecycleConfig
   - NotebookInstanceName: !Ref NotebookInstanceName
   OnStart: 
    - Content:
      Fn::Base64: !Sub 
       - |
        #!/bin/bash -xe
        set -e
        CP_SAMPLES=true
        EXTRACT_CSV=false
        s3region=s3.amazonaws.com
        SRC_NOTEBOOK_DIR=${Consumer2BucketName}/sagemaker-notebooks
        Sagedir=/home/ec2-user/SageMaker
        industry=industry
        notebooks=("notebook1.ipynb" "notebook2.ipynb" "notebook3.ipynb")
        download_files(){
          for notebook in "${notebooks[@]}"
            do
             printf "aws s3 cp s3://${SRC_NOTEBOOK_DIR}/${notebook} ${Sagedir}/${industry}\n"
             aws s3 cp s3://"${SRC_NOTEBOOK_DIR}"/"${notebook}" ${Sagedir}/${industry}
            done
        }
        if [ ${CP_SAMPLES} = true ]; then
          sudo -u ec2-user mkdir -p ${Sagedir}/${industry}
          mkdir -p ${Sagedir}/${industry}
          download_files
          chmod -R 755 ${Sagedir}/${industry}
          chown -R ec2-user:ec2-user ${Sagedir}/${industry}/.
        fi
       - Consumer2BucketName: !Ref Consumer2BucketName

Raised the following error:

Template error: variable names in Fn::Sub syntax must contain only alphanumeric characters, underscores, periods, and colons

It seems that was a conflict with the Bash Vars and the !Sub CF function.

In the following template I changed the Bash Vars and removed the {}:

LifecycleConfig:
  Type: AWS::SageMaker::NotebookInstanceLifecycleConfig
  Properties: 
   NotebookInstanceLifecycleConfigName: !Sub
   - ${NotebookInstanceName}LifecycleConfig
   - NotebookInstanceName: !Ref NotebookInstanceName
   OnStart: 
    - Content:
      Fn::Base64: 
       !Sub 
        - |
         #!/bin/bash -xe
         set -e
         CP_SAMPLES=true
         EXTRACT_CSV=false
         s3region=s3.amazonaws.com
         SRC_NOTEBOOK_DIR=${Consumer2BucketName}/sagemaker-notebooks
         Sagedir=/home/ec2-user/SageMaker
         industry=industry
         notebooks=("notebook1.ipynb" "notebook2.ipynb" "notebook3.ipynb")
         download_files(){
           for notebook in $notebooks
             do
              printf "aws s3 cp s3://$SRC_NOTEBOOK_DIR/${!notebook} $Sagedir/$industry\n"
              aws s3 cp s3://"$SRC_NOTEBOOK_DIR"/"${!notebook}" $Sagedir/$industry
             done
         }
         if [ $CP_SAMPLES = true ]; then
           sudo -u ec2-user mkdir -p $Sagedir/$industry
           mkdir -p $Sagedir/$industry
           download_files
           chmod -R 755 $Sagedir/$industry
           chown -R ec2-user:ec2-user $Sagedir/$industry/.
         fi
        - Consumer2BucketName: !Ref Consumer2BucketName

The problem here is the for loop is not running through all the notebooks in the list but importing only the first one.
After going through some solutions I tried adding [@] to the notebooks:
for notebook in $notebooks[@]
and
for notebook in “$notebooks[@]“/”$notebooks[*]“/$notebooks[@]
I got the same error.

kichik · Accepted Answer · 2022-01-20 18:32:19Z

4

It seems that was a conflict with the Bash Vars and the !Sub CF function.

That's correct. Both bash and !Sub use ${} for variable substitution. You can escape the bash variables with ${!}. For example:

for notebook in "${!notebooks[@]}"

Also mentioned in the docs:

To write a dollar sign and curly braces (${}) literally, add an exclamation point (!) after the open curly brace, such as ${!Literal}. AWS CloudFormation resolves this text as ${Literal}.

answered Jan 20, 2022 at 18:32

kichik

34.9k7 gold badges102 silver badges121 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Bash script inside Cloudformation

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related