Real-time Data Pre-processing in Lambda for SageMaker Endpoint

Question

Am working on a project of consumer behaviour analysis on websites and predict the malicious activity of users in real-time. Click data is being collected for each click made by users.

Am using multiple AWS services like kinesis stream, Lambda and sagemaker. I have created an autoencoder model and deployed it as sagemaker endpoint which will be invoked using lambda when it receives new click data from the website through Kinesis stream.

Since sagemaker endpoint contains the only model but click data which lambda function receives is raw data with URLs, texts and date. How can I pass raw data into required preprocessing steps and send processed data to sagemaker endpoint in the required format?

Example of raw data:-

{'URL':'www.amazon.com.au/ref=nav_logo', 'Text':'Home', 'Information':'Computers'}

KPH · Accepted Answer · 2021-06-21 05:42:31Z

1

You can use Sagemaker inference Pipeline. You need to create preprocessing script comprising of your preprocessing steps and create a Pipeline including Preprocess and model. Deploy pipeline to an endpoint for real time inference.

Reference: https://aws.amazon.com/blogs/machine-learning/preprocess-input-data-before-making-predictions-using-amazon-sagemaker-inference-pipelines-and-scikit-learn/

https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/scikit_learn_inference_pipeline/Inference%20Pipeline%20with%20Scikit-learn%20and%20Linear%20Learner.ipynb

answered Jun 21, 2021 at 5:42

KPH

641 silver badge8 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Yuri Santos Over a year ago

Do I need to include the model in the Pipeline? I was wondering about transformations and then getting preprocessed data from the preprocess endpoint. Something like creating a method, including it on the Pipeline, and then deploying the Pipeline containing this preprocessing method.

Collectives™ on Stack Overflow

Real-time Data Pre-processing in Lambda for SageMaker Endpoint

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related