Using Jaeger in OpenShift/Kubernetes

General Instructions

  1. Clone the git repository

    git clone https://github.com/ibm-cloud-architecture/learning-distributed-tracing-101.git

  2. Change to the lab directory

    cd learning-distrubing-tracing-101/lab-jaeger-ocp

Understanding Jaeger

Installing the Jaeger operator

(OpenShift 3) Creating an instance of Jaeger

Create a Custom Resource for Jaeger

  1. Log in to the OpenShift Container Platform web console.

  2. Select project operators

  3. Verify Jaeger Operator installed

  4. From the command line, create a Jaeger instance in the default namespace

kubectl apply -f - <<EOF
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: my-jaeger
EOF

(OpenShift 4) Creating an instance of Jaeger

Create a Custom Resource for Jaeger

  1. Log in to the OpenShift Container Platform web console.

  2. Select project openshift-operators

  3. Navigate to Operators → Installed Operators

  4. Click the Jaeger Operator provided by Red Hat

  5. Under Provided APIs

  6. Click Create Instance

  7. Edit namespace to your project for example default

  8. Review yaml and Click Create

Verify Jaeger instance created

  1. Verify the Jaeger services in the default namespace

    oc get services -n default | grep jaeger
my-jaeger-agent                ClusterIP      None             <none>      5775/UDP,5778/TCP,6831/UDP,6832/UDP      7m
my-jaeger-collector            ClusterIP      172.21.191.135   <none>      9411/TCP,14250/TCP,14267/TCP,14268/TCP   7m
my-jaeger-collector-headless   ClusterIP      None             <none>      9411/TCP,14250/TCP,14267/TCP,14268/TCP   7m
my-jaeger-query                ClusterIP      172.21.89.78     <none>      443/TCP                                  7m
Take a look at the my-jaeger-collector on port 14268/TCP this is the service to be used by our services to send the Jaeger traces containing the spans, you will configure this in the kubernetes deployment yaml manifest.

  1. Find the route to the Jaeger UI

    oc get route my-jaeger
NAME        HOST/PORT                            PATH   SERVICES          PORT    TERMINATION   WILDCARD
my-jaeger   my-jaeger-default.apps-crc.testing          my-jaeger-query   <all>   reencrypt     None

  2. Open the Jaeger UI in the browser using value HOST/PORT https://my-jaeger-default.apps-crc.testing

Deploy the Application

  1. Deploy the services service-a and service-b

    Use the file jaeger-nodejs.yaml for Node.js or the file jaeger-java.yaml for Java

    Here is an example using Node.js services and deployments:

    oc apply -f jaeger-nodejs.yaml -n default

    Let’s look at the file content on how the services are defined to be deploy into OpenShift cluster:

    ---
apiVersion: v1
kind: Service
metadata:
  name: service-a
  labels:
    app: service-a
spec:
  ports:
    - port: 8080
      name: http
  selector:
    app: service-a
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service-a
  labels:
    app: service-a
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service-a
  template:
    metadata:
      labels:
        app: service-a
        version: v1
    spec:
      containers:
        - name: app
          image: csantanapr/service-a-nodejs
          env:
            - name: JAEGER_ENDPOINT
              value: http://my-jaeger-collector:14268/api/traces
            - name: SERVICE_FORMATTER
              value: service-b
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: service-b
  labels:
    app: service-b
spec:
  ports:
    - port: 8081
      name: http
  selector:
    app: service-b
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service-b
  labels:
    app: service-b
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service-b
  template:
    metadata:
      labels:
        app: service-b
        version: v1
    spec:
      containers:
        - name: app
          image: csantanapr/service-b-nodejs
          env:
            - name: JAEGER_ENDPOINT
              value: http://my-jaeger-collector:14268/api/traces
          imagePullPolicy: Always
          ports:
            - containerPort: 8081

    In the yaml deployment manifest there are few items to point out:

    • Ports

      • The port for the container is specified in the service and the container in the deployment, for example service-a with port 8080 and service-b with port 8081

    • Environment Variables

      • The variable JAEGER_ENDPOINT is specified to indicate to the Jaeger client library to send the traces using http to the jaeger collector service http://my-jaeger-collector:14268/api/traces that is deployed on the same namespace default as the services. You could also opt for using a side card and use UDP to send traces to an agent side card and this will foward the traces to the jaeger collector for more info see the jaeger operator documentation on how to enable this with an annotation.

      • The variable SERVICE_FORMATTER used by service-a to indicate the hostname of service-b that will use to format the hello message.

  2. Verify services are deployed and running:

    oc get all -l app=service-a -n default
oc get all -l app=service-b -n default
NAME                             READY   STATUS    RESTARTS   AGE
pod/service-a-785975554d-5cql2   1/1     Running   0          19m
pod/service-b-674b748766-t7464   1/1     Running   0          19m

NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/service-a   ClusterIP   172.30.182.142   <none>        8080/TCP   20m
service/service-b   ClusterIP   172.30.108.212   <none>        8081/TCP   19m

NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/service-a   1/1     1            1           19m
deployment.apps/service-b   1/1     1            1           19m

  3. Expose the service service-a with a route

    oc create route edge  --service=service-a -n default

  4. Get the hostname for the route:

    oc get route service-a -n default
NAME        HOST/PORT                            PATH   SERVICES    PORT   TERMINATION   WILDCARD
service-a   service-a-default.apps-crc.testing          service-a   http   edge          None

Find Traces

  1. Use curl or open browser with the endpoint URL using the HOST/PORT of the route

    curl -k https://service-a-default.apps-crc.testing/sayHello/Carlos
Hello from service-b Carlos!

    From the result you can see that service-a called service-b and replied back.

  2. In the Jaeger UI select service-a and click Find Traces

    ocp jaeger traces

  3. Click on one of the traces and expand the spans in the trace

    ocp jaeger spans

Check one of the labs Lab Jaeger - Node.js or Lab Jaeger - Java for a more in depth lab for Opentracing with Jaeger.