2

Here is python script that sends test trace directly to tempo instance :

import base64
from typing import Tuple

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import \
    OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import Tracer, TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

def create_span_exporter_for_tempo() -> Tuple[Tracer, BatchSpanProcessor]:

    resource = Resource(attributes={"service.name": constants.APP_NAME})

    headers = {
        "Authorization": f"Basic {base64.b64encode(f'{constants.GRAFANA_USERNAME}:{constants.GRAFANA_PASSWORD}'.encode('ascii')).decode('ascii')}"
    }
    tempo_exporter = OTLPSpanExporter(endpoint=constants.TEMPO_ENDPOINT, headers=headers)

    trace.set_tracer_provider(TracerProvider(resource=resource))
    tracer = trace.get_tracer(__name__)
    span_processor = BatchSpanProcessor(tempo_exporter)

    trace.get_tracer_provider().add_span_processor(span_processor)
    return tracer, span_processor

tracer, span_processor = create_span_exporter_for_tempo()
with tracer.start_as_current_span("test") as span:
        print("test!")
        span.set_attribute("check", "works")

My Grafana Cloud setup looks like this : enter image description here

Here are some of variations of tempo endpoint tried :

1. https://tempo-prod-04-prod-us-east-0.grafana.net/tempo
2. https://tempo-prod-04-prod-us-east-0.grafana.net/tempo/v1/traces
3. https://tempo-prod-04-prod-us-east-0.grafana.net/v1/traces
4. https://tempo-prod-04-prod-us-east-0.grafana.net:443/tempo/v1/traces
5. https://tempo-prod-04-prod-us-east-0.grafana.net/tempo/v1/trace

All of these endpoint return the same output :

Failed to export batch code: 404, reason: 404 page not found

enter image description here

Please note that, I have tested out this tempo instance by setting up grafana agent & I was able to send traces via grafana agent.

My requirement is to send traces without deploying grafana agent as I am sending traces from lambda & do not want to run agent somewhere else 24X7.

Here is python library being used

1 Answer 1

2

You have to use OTLP Gateway. Make note of the following values that you’ll need in order to write OLTP:

  • OTLP Endpoint URL: https://otlp-gateway-<zone>.grafana.net/otlp (ex: https://otlp-gateway-prod-us-central-0.grafana.net/otlp).
  • Basic Auth Username: <instanceID from Grafana details page>
  • Basic Auth Password: <Grafana API Key>

Doc: https://grafana.com/docs/grafana-cloud/data-configuration/otlp/send-data-otlp/

Sign up to request clarification or add additional context in comments.

6 Comments

For anyone stumbling across this: If you want to send traces directly from the code as in the example above, append /v1/traces to the endpoint mentioned above, so https://otlp-gateway-<zone>.grafana.net/otlp/v1/traces (see community.grafana.com/t/opentelemetry-endpoint-of-grafana-cloud/…)
@jan-garaj, @moertel : Now this is not working. I have been trying to send traces to instance setup using new grafana cloud account and it keeps giving me authentication error: invalid authentication credentials. URL I am using is : https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/traces I tried cloud policy token as well as api key both. Any idea if something has been changed ?
@UtsavChokshi authentication error: invalid authentication credentials is not a problem with URL, but with used credentials. You have wrong authentication, e.g. wrong Basic Auth Username and/or Basic Auth Password. Only you have access to your credentials/org, so only you can find what is wrong.
@JanGaraj That's the problem. The code I have put it in question work for one grafana instance and does not work with another one. I have checked my credentials so many times. It is correct. I thought maybe the way auth works at grafana has changed with new instances.
@UtsavChokshi that OTLP gateway doesn't have SLA. But you can still complain to your Grafana cloud support.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.