1

I have a torchserve service running on kubernetes and I am already able to track metrics with it on port 8082. My problem is that from the kubernetes pod I can see it logs hardware metrics like:

[INFO ] pool-3-thread-2 TS_METRICS - CPUUtilization.Percent
[INFO ] pool-3-thread-2 TS_METRICS - DiskAvailable.Gigabytes
[INFO ] pool-3-thread-2 TS_METRICS - GPUMemoryUtilization.Percent

Although, if i check the metrics that I am currently scraping I am only able to see:

# TYPE ts_inference_requests_total counter
ts_inference_requests_total 144.0
ts_inference_requests_total 20.0
# HELP ts_inference_latency_microseconds Cumulative inference duration in microseconds
# TYPE ts_inference_latency_microseconds counter
ts_inference_latency_microseconds 6.051944813839998E8
ts_inference_latency_microseconds 4.7464253726E7
# HELP ts_queue_latency_microseconds Cumulative queue duration in microseconds
# TYPE ts_queue_latency_microseconds counter
ts_queue_latency_microseconds 2633867.5069999998
ts_queue_latency_microseconds 1080.43

Is it possible to also scrape the metrics that are being logged on kubernetes? Thanks for any help!

Improve this question
3
  • 1
    what does your metrics.yaml file look like? Commented May 2, 2023 at 9:10
  • 1
    cAdvisor can expose container metrics in a format compatible with Prometheus, enabling users to scrape and store these metrics using Prometheus for further analysis and visualization. For advanced use the node_exporter can be passed an optional list of collectors to filter metrics. The collect[] parameter may be used multiple times. In Prometheus configuration you can use this syntax under the scrape config. Commented May 2, 2023 at 12:17
  • Hi @Prosciutt0, Feel free to update the status of the question. Let me know the comment above helps to resolve your issue? I am happy to help you if you have any further queries. Commented May 5, 2023 at 5:30
Related questions
2
Scrape CoreDNS metrics with External Prometheus
1
kubernetes cadvisor endpoint is not scraped by prometheus
4
Prometheus not receiving metrics from cadvisor in GKE
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.