263 questions
-1
votes
1
answer
42
views
Getting unrecognized arguments: --federated-token in creating pipeline in microsoft/MLOpsPython
I am using this repo to create a mlops pipeline in Azure Devops.
When i tried to run the CI pipeline, I am getting the unrecognized Federal Token. I asked chatgpt, it says to update the cli version. I ...
0
votes
0
answers
116
views
My model registry component is throwing an error: [INTERNAL_ERROR: Response: {'error': 'unsupported endpoint,}
I'm trying to register a model using mlflow.register_model() while using DagsHub as the MLflow tracking URI. My goal is to use model versioning and transition stages for CI/CD workflows.
Here's the ...
0
votes
1
answer
77
views
How to keep the same version number in AWS SageMaker ModelPackageGroup when updating model with evaluation metrics?
I’m working with AWS SageMaker Model Registry and have a training pipeline that creates and registers a new model package in a ModelPackageGroupName. After that, I have a separate evaluation pipeline ...
- 13
0
votes
0
answers
69
views
How to reference all models in a registry folder in deployment.yml for azure ml studio?
I want a single managed online endpoint that can serve dozens of forecasting models (7 algorithms × N service-centre “hubs”) by routing requests to the correct model at runtime.
Locally this already ...
0
votes
1
answer
91
views
How to run scheduled GPU inference with auto-shutdown using Azure ML?
I have a PyTorch model and a simple inference script that I want to run a few times per day on GPU (specifically H100) in Azure. Ideally, this should be triggered by an event, like a new file in Blob ...
1
vote
0
answers
40
views
Ingesting data into Azure ML model deployment
I am building an ML pipeline in Azure but it is failing when trying to invoke the endpoint with my model. The error reads:
Error Code: ScriptExecution.StreamAccess.NotFound
Native Error: error in ...
- 11
0
votes
0
answers
77
views
SageMaker Real-Time Endpoint Timeout Issues with Lambda for Parallel Data Processing
I’m new to AWS and struggling with an architecture involving AWS Lambda and a SageMaker real-time endpoint. I’m trying to process large batches of data rows efficiently, but I’m running into timeout ...
0
votes
0
answers
109
views
Python `mlflow.transformers.log_model` is attempting to log model to a repo when I want it to log to the model registry
I am working in azure trying to run a job that calls a training notebook. I can train and even evaluate my model just fine within said notebook but when I try to log it at the end it throws errors. ...
0
votes
0
answers
115
views
How to launch a FastAPI webserver from Dagster?
I am creating an MLOps pipeline which process data, train model, and deploy model into an inference service. I am trying to create this all without Cloud services such as S3. I am using Dagster as the ...
- 3,071
0
votes
0
answers
115
views
Properly configuring use_IO_bindings in ONNX ORTModelForSequenceClassification to improve inference speed on GPU
I'm currently running into an issue where running the changes (in green) in the following diff leads to worse performance on GPU (it adds an additional 100ms/request on average, with significantly ...
- 181
0
votes
1
answer
179
views
How to Configure Model Monitoring in Vertex AI using aiplatform.BatchPredictionJob.create()?
I encountered an issue while setting up Vertex AI Model Monitoring using the aiplatform SDK, specifically when configuring BatchPredictionJob.create(). The documentation was unclear and lacked ...
0
votes
0
answers
60
views
Create custom mlflow PythonModel with `predict()` function that takes no arguments
I'd like to create a custom mlflow PythonModel whose predict function does not take any arguments. It looks like mlflow.pyfunc.load_model (which I use to reover the custom moel) returns a PyFuncModel ...
- 567
0
votes
1
answer
129
views
How do I run a parallel server in the background inside a docker container to test it
I want to create a CI/CD pipeline that will automatically build, test and publish a docker container as soon as I push a commit onto github.
To do this I am creating a container using a Dockerfile ...
- 35
0
votes
1
answer
175
views
How to access evaluation metrics in new SageMaker Studio UI after doing model.register?
I am building MLOPs pipelines for a machine learning model. How do I access the evaluation metrics of my model in the SageMake Studio UI after registering the model?
Here's my sample evaluation.json ...
0
votes
1
answer
516
views
`mlflow.transformers.log_model()` does not finish
Problem
I want to use mlflow.transformers.log_model() to log a finetuned huggingface model.
However, when the mlflow.transformers.log_model method is running, it simply does not finish - runs forever -...
- 2,543
-2
votes
1
answer
142
views
Run pods in parallel in Azure Devops to run a single job
I want to run my azure devops agent pool using kubernetes. I am using this to run a Jupyter notebook which takes quite a bit of time to run. I want to know if it is possible for one job which runs the ...
- 3
0
votes
1
answer
142
views
Share code across sagemaker pipeline steps without
I am trying to create Sagemaker pipeline with multiple steps. I have some code which I would like to share across different steps. Next example is not exact but simplified version for illustration.
I ...
- 936
1
vote
1
answer
247
views
How to organize registerd models in R with mlflow?
Using aliases in mlflow is best practice and convienient to organize registered models.
But unfortunately, even the latest mlflow R Package (Version 2.14.1) does not include any methods that allow ...
- 2,543
2
votes
1
answer
73
views
TaskRun failed to finish due to an error for Coretex BioInformatics workflow
After starting bioinformatics workflow in Coretex, I am getting the following message even though data seems to be in order:
"Failed to determine which column contains sampleIDs/names..." ...
- 23
1
vote
0
answers
39
views
The features selected by SelectKBest do not match those transformed by ColumnTransformer
I am in the process of deploying a machine learning model for study purposes and I have some questions about it:
My POST method will send to the API my original features (without transformations ...
0
votes
1
answer
47
views
Is an intermediary persistent store needed before storing features in Feast + Cassandra?
I am currently building a big data pipeline for an MLOps project, the pipeline is intended for batch processing.
This is the current setup:
I am storing my raw structured data in Hive.
Spark jobs ...
- 13
0
votes
1
answer
418
views
How to create a kubeflow component from already running python scripts
I am very very new to both AI and MLOP's, please forgive me if my question is dumb. I am trying to learn about kubeflow but there is too much information on kubeflow documentation, then there are ...
- 53
0
votes
1
answer
1k
views
GitHub connection to Azure ML workspace
Is the only way available to make changes to m notebook in Azure ML workspace through the interface itself? Is it possible for me to make a CI/CD pipeline that takes the changes from a Git repository ...
- 3
1
vote
1
answer
319
views
Unable to deploy hugging face model to sagemaker endpoint - C:\\.sagemaker-code-config not found
I'm trying to make a sagemaker endpoint using sagemaker and hugging face libraries.
import sagemaker
sess = sagemaker.Session()
sagemaker_session_bucket=None
if sagemaker_session_bucket is None and ...
- 133
5
votes
1
answer
685
views
Integrating MLflow with GitLab: Getting 404 Project Not Found Error
I am working on a project that involves using MLflow and GitLab integration. I have set up a Jupyter notebook where I aim to integrate data from MLflow into GitLab. Following the instructions provided ...
- 51
0
votes
1
answer
226
views
Mlflow log_figure deletes artifact
I am running mlflow with autologging to track an xgboost model. By default, under artifacts it saves the model, requirements, and feature importances. Cool stuff I want to keep.
But, if I try to add ...
- 385
1
vote
1
answer
167
views
Extract current running stage from dvc
I'm conducting an experiment using 'dvc repro -f', where multiple stages are executed according to the dvc.yaml configuration. For instance:
Stages:
Training:
foreach:
-cycle: 0
-cycle: 1
...
- 99
2
votes
1
answer
473
views
How can I download data from just one of the DVC repositories?
I have a project that uses several databases, to avoid versioning huge files in git, I used DVC to manage it on gdrive.
I followed the following step by step on DVC
Start DVC (dvc init)
dvc add #...
1
vote
1
answer
166
views
feast.errors.FeatureViewNotFoundException: Feature view driver_stats does not exist
I'm using feast to create a Feature store for my MLOPs project. However when I execute the command:
# Retrieve training data
training_df = fs.get_historical_features(
entity_df=orders,
...
- 35
0
votes
1
answer
115
views
I have a prolem with feast[redis]
I am installing some packages for environment and I followed the steps:
I clone this repo: https://github.com/MLOpsVN/mlops-crash-course-code
pip install -r dev_requirements.txt (in folder mlops-...
- 35
0
votes
1
answer
92
views
Model serving - tools and components
I am working on a solution for providing a custom platform catering to manage and run LLM applications using RAG and LLM models using user provided document repository.
While planning and designing a ...
1
vote
1
answer
339
views
Unable to properly register model and create Sagemaker Endpoint using Sagemaker Pipelines
I'm trying to register and deploy a custom model using a Pytorch container in Sagemaker Pipelines inside Sagemaker Studio but the endpoint fails when sending a response using invoke_endpoint:
The code ...
- 31
0
votes
1
answer
2k
views
Can MLFlow be used without the `with mlflow.start_run()` block?
I want to track an entire notebook and log the parameters of cleaning steps that occur before training a model. I'd like to use mlflow to do it, but on all the docs it looks like you have to track ...
- 1,146
0
votes
1
answer
313
views
Databricks DBX and Asset Bundles: Support for Storing config files in Container/Storage Account
I'm trying to deploy a Databricks workflow that is configured using yaml files. Currently I'm using dbx. Is there any way that, instead of using the YAML files within my project locally that is then ...
- 65
0
votes
1
answer
553
views
how to import ml model (python) into another programming language
Can you explain which methods (pros and cons) are available in order to import ml-model
(sk-learn/tf/pytorch), etc into other code (java/c/C++) etc.
I mean that I want to take the pickle file (or ...
- 11
2
votes
2
answers
789
views
Sagemaker batch transformer with my own pre-trained model
I'm trying to run inference on demand for yolo-nas using sagemaker batch transformer.
Using pre trained model with pre trained weights.
But I am getting this error:
python3: can't open file '//serve':...
0
votes
1
answer
705
views
Zenml+Mlflow : Model not deploying on localhost
Step deployment_trigger has started.
Step deployment_trigger has finished in 0.051s.
Caching disabled explicitly for mlflow_model_deployer_step.
Step mlflow_model_deployer_step has started.
Updating ...
1
vote
0
answers
232
views
DVC GET error self._sslobj.do_handshake() Connection reset by peer
Someone could help me with this issue? I'm newbie on DVC and I dont know what I'm doing wrong. I have a git repository with a dvc project. But when I try to get a file following DVC oficial ...
- 11
3
votes
0
answers
587
views
Best way to host multiple pytorch model files for inference?
Context:
I'm working with an end to end deep learning TTS framework (you give it text input it gives you a wav object back)
I've created a FastAPI endpoint in a docker container that uses the TTS ...
- 417
0
votes
1
answer
114
views
Access key must be provided in Client() arguments or in the V3IO_ACCESS_KEY environment variable
I got the error ValueError: Access key must be provided in Client() arguments or in the V3IO_ACCESS_KEY environment variable during data ingest in MLRun CE (version 1.5.0).
I used this code:
import ...
- 1,390
1
vote
1
answer
205
views
How to specify an arbitrary binary file in catalog.yml
I'm currently working on a data science project using LLMs (Large language models). Weights for models usually come in different formats, most frequently .bin or .gguf, and I'd like to keep it that ...
- 101
0
votes
1
answer
80
views
Sagemaker Monitor - MonitoringDatasetFormat as gz
I have created a monitoring schedule to monitor predictions from a Batch Transform job. The schedule runs fine when the input dataset_format in BatchTransformInput is csv. However, my batch job is ...
- 26
0
votes
2
answers
1k
views
Azure DevOps Unable to find file "requirements.txt"
everyone!
First of all, thank you very much for taking the time to read this question. Without further delay, let me explain my situation.
I'm using Azure DevOps, specifically the Pipelines feature. I ...
0
votes
0
answers
306
views
Register Dataset to Azure ML using Axure Devops pipeline
datastore = Datastore.register_azure_blob_container(
workspace=workspace,
datastore_name=datastore_name,
account_name=datastore_account_name, # Storage account name
...
1
vote
1
answer
394
views
Is there a way I can setup default graphs via the MLFlow API
I create a new 'experiment' every time I change the validation set (which is often), but each time I want basically the same graphs. This is the easiest way for me to make relevant comparisons across ...
- 376
3
votes
1
answer
1k
views
Proper way of making my python module available to the mlflow during mlflow models build-docker
I am trying to build a docker image that I could host with an endpoint of my model and have experienced issues on how to make my code available during the build so that the image would later run and ...
- 31
0
votes
1
answer
159
views
MLRun ingestion, ConnectionResetError 10054
I got this error during ingest data to Parquet in MLRun CE:
2023-09-03 14:01:47,327 [error] Unhandled exception while sending request: {'e': <class 'ConnectionResetError'>, 'e_msg': ...
- 1,390
1
vote
3
answers
661
views
How to pass only necessary features to pipeline after SelectKBest
I have a regular tabular dataset, 100 features from the database are added
I want to push it into a regular sklearn.pipeline in which there will be preprocessing, encoding, some custom transformers, ...
- 33
0
votes
1
answer
755
views
MLFlow error during `mlflow ui` in terminal
The previous week everything was fine and I did experiment tracking. This week when I ran it, that is, when I wrote mlflow ui in the terminal in the same directory where the project was, it started me ...
1
vote
0
answers
397
views
Unable to load ONNX model for inference
I have a scikit-learn pipeline for heart disease prediction. I am successful in converting the same to ONNX representation. I am unable to load this model for inference because of the following error:...
