Working on MicroProfile-Service-Mesh, Istio and Kiali
Recently I have been working on the Eclipse MicroProfile ServiceMesh effort. This effort is meant to investigate how MicroProfile can work together with a service mesh like Istio, to find gaps and to produce recommendations, as some of the Istio capabilities are similar to capabilities in some MicroProfile specs like the retry-on-failure logic in Fault-Tolerance.
A first work item consisted of bringing Thorntail into the existing picture and to allow to use alternative implementations for the called service Service B. The next picture shows the setup with some traffic and visualised in Kiali.
If you want to replicate this setup, you can to go the MicroProfile Service Mesh GitHub page and follow the instructions in the samples/ directory.
If you don’t want to build all the images yourself, you can use the images I built with this deployment file:
curl https://gist.githubusercontent.com/pilhuhn/604a5111070b6f0b35e691663686fe9c/raw/92dc76f4bb6325fb4869aa636f0ec60116d084c8/deploy-stuff.yml |\
istioctl kubeinject -f - |
oc apply -fOf course you also need to do the “usual” magic for elevated privileges of the namespace to use Istio.
And then also deploy the gateway.yml and services.yml files from the Service Mesh repository.
When all this is set up and the containers are pulled, you can throw traffic at the Ingressgateway, see the mesh and the services in action and can see from the replies that traffic is alternating between Service B on Liberty and Thorntail (you may have a different IP depending on your setup).
curl http://istio-ingressgateway-istio-system.172.31.7.9.nip.io/mp-servicemesh-sample/serviceAFire up Kiali and see it in action as pictured above.
Looking at distributed tracing…
When I switched over to the Distributed Tracing tab of Kiali, I saw that there were only traces with spans between Ingress Gateway and Service A and others between Service A and Service B, which puzzled me a bit.
If you look at the next diagram, it probably gets clearer. Requests enter the system from the Browser via the orange line into the Ingress-GW. The GW then starts a new trace it’s tracing identifiers into the request headers (in green now). This enters the Sidecar of Service A and also Service A itself (yellow JAX-RS-Endpoint).
Now Service A makes a request to Service B via the MicroProfile Rest-Client, purple lines. There is no automatic forwarding of the tracing context between the JAX-RS-Endpoint and the Rest-Client, which means that Envoy will create an new trace for the request from Service A to Service B, which is completely idependent from the first trace (with the green arrows).
And this is why one sees the trace never going from Ingress-GW to ServiceA to Service B.
In the longer run, this is something the MicroProfile OpenTracing spec and the RestClient-Spec have to solve.
In the meanwhile I have opened a pull-request for ‘manual’ propagation and with it, we get the desired result:
You may wonder why serviceA shows up two times. The first time is when the request hits serviceA from the IngressGW and the 2nd time is when the 2nd request from serviceA is made to serviceB.
