Google Vertex AI Prediction: Why is TorchServe showing 0 GPUs?

Question

I have deployed a trained PyTorch model to a Google Vertex AI Prediction endpoint. The endpoint is working fine, giving me predictions, but when I examine its logs in Logs Explorer, I see:

INFO 2023-01-11T10:34:53.270885171Z Number of GPUs: 0

INFO 2023-01-11T10:34:53.270888834Z Number of CPUs: 4

This is despite the fact that I set the endpoint to use NVIDIA_TESLA_T4 as the accelerator type:

Why does the log show 0 GPUs and does this mean TorchServe is not taking advantage of the accelerator GPU?

Hi @urig the availability of each type of GPU depends on the region you use for your model. Could you specify the region? — kiran mathew
– kiran mathew, Commented Jan 12, 2023 at 7:35
Thanks @kiranmathew 🌷 . I'm in europe-west4 where NVIDIA_TESLA_T4 GPUs are regularly available to me for custom jobs in training. If Vertex AI was unable to make one available, should it not have indicated this to me somehow? — urig
– urig, Commented Jan 12, 2023 at 9:05

kiran mathew · Accepted Answer · 2023-01-27 13:48:52Z

2

This is a common problem with PyTorch and CUDA. GPU support is only enabled when the right version of PyTorch is installed, i.e. one which compiles for CUDA. So it’s recommended that you use images which have PyTorch's CUDA capabilities.

answered Jan 27, 2023 at 13:48

kiran mathew

2,3931 gold badge7 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Google Vertex AI Prediction: Why is TorchServe showing 0 GPUs?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related