I am new to using the AWS Sagemaker services as well as docker. I have afew questions around designing the architecture to deploy my setup from test AWS account to production.
I have a single domain with multiple spaces. These spaces each run models for specific use-cases. Each of these spaces has a certain set of python dependencies/libraries that need to be installed once the notebook is first created or when a new library is added.
The two ways this could be done as per my search are by creating custom docker images or by using lifecycle configurations.
I eliminated the lifecycle configuration option for 2 reasons:
- It would load all the libraries every time a new session was created and I have few heavy libraries which I don't want to install at each new session. I want to run them only once at the first run of the notebook.
- I require to download a checkpoint from a colab workbook which I assume would be a bit tricky to do through lifecycle configurations. Please correct me if my understanding is wrong and you think lifecycle is a suitable approach.
Moving to custom docker image, my understanding of the process is that I should:
- create a docker image to load all the libraries. Include the command to download the checkpoint and store the downloaded file in an S3 bucket.
- upload the image into an ECR repo and push it to the desired environment. Question: do I need to build my sagemaker spaces based on the custom image or do I point it to the custom image? Basically, how do I utilize the custom image? Because I have space specific dependencies. Also, will this image only be pulled once when the notebook is run for the very first time or every time a new session is created? As if that's the case, then why not use lifecycle config?
Any help on understanding how the custom image creation works would be super appreciated. I have gone through the AWS documentation but there is still a gap in my understanding.
Thank you.
