Would like to use otel4s as alternative for Scala Spark Structured Streaming project

[patpatpat123]

Hello there,

This is my first post in this repo.
If not anything else, I just wanted to say thank you for the work.
Especially, the docs are very clear, easy to follow.

I wanted to reach out in order to ask for help.
We have a Scala Spark Structured Streaming Kafka consumer application that we would like to integrate and leverage Otel4s.
Our business flow is fairly simple. There are multiple upstream applications, written in different languages, which will put a message into Kafka. The message put in Kafka is properly enriched with traces. There is a valid W3C traceparent in the header. Then, the message arrives in Kafka.

Our Scala Spark Structured Streaming job will consume the message (that has valid traces), perform some enrichment, and store it in the DB (like the examples in the doc with Redis).

Upstrean app -> kafka <-> Spark app -> DB

We would like to 1) propagate, continue the trace from the message in Kafka (which has a valid trace parent header) and 2) trace the DB call (but let us focus on 1 )

We have written the following in Scala, using the latest 1.19.0 opentemeletry suite (API, SDK, Exporter OTLP)

def call(traceParent: (String, String)): String = {
    val spanExporter = OtlpGrpcSpanExporter.builder().setEndpoint("https://tempo-prod-04-prod-us-east-0.grafana.net:443").addHeader("Authorization", "Basic ODEwNzcxOmdsY19leUp2SWpvaU1UQTFOVFV4T1NJc0ltNGlPaUp6ZEdGamF5MDROVGcyTnpJdG[...]npkQzB3SW4xOQ==").build()
    val myResource = Resource.create(Attributes.builder().put("service.name", "scalaservice").build())
    val tracerProvider = SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spanExporter)).build()
    val openTelemetry = OpenTelemetrySdk.builder().setTracerProvider(tracerProvider).build()
    val remoteContext = SpanContext.createFromRemoteParent(traceParent._1, traceParent._2, TraceFlags.getSampled, TraceState.getDefault)
    val tracer = openTelemetry.getTracer("from-scala")
    val spanBuilder = tracer.spanBuilder("scala-name").setSpanKind(SpanKind.CONSUMER)
    spanBuilder.setParent(Context.current().`with`(Span.wrap(remoteContext)))
    spanBuilder.setAttribute("testAttributeScala1", "testAttributeScala2")
    val mySpan = spanBuilder.startSpan()
    try {
      Thread.sleep(400) //mock our work
      println(traceParent._1 + traceParent._2 + mySpan.toString + mySpan.isRecording + remoteContext)
      mySpan.toString
    } finally {
      mySpan.end()
    }
  }

For some reason unknown to use, this code is not working, in terms that we can see a good span object with the child (from the print function) but we cannot see any trace in any of our tracing backends (we tried Tempo, Lightstep, Datadog, ElastocSeach APM, no luck)

From the reading of your post, it might be due to Context and ThreadLocal, but we are not sure, hence, continue to read your docs.

Do you allow me to ask:

• Do you think using your project instead of our current approach would work?
• We are not seeing any "export" from the samples, could you please point out where the actual sending of the trace (let's say to Tempo)
• Would you happen to know what the sample we wrote (it is a minimal reproducible code) is not working by any chance?

Thank you again and good day.

[iRevive]

Hey!

otel4s uses opentelemetry-java under the hood.
I suggest experimenting with an auto-configured instance and using env variables as configuration options:

OtelJava.autoConfigured[IO].use { otelJava =>
 ... // your code
}

The env variables are:

• https://opentelemetry.io/docs/languages/sdk-configuration/general/
• https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/

And here are a few configuration examples:

• https://typelevel.org/otel4s/examples/honeycomb/
• https://typelevel.org/otel4s/examples/grafana/