image

Starting Istio 1.4 and up, the way the Observability metrics are created, exchanged and scraped changed. Here is how I configure Prometheus-Operator resources to scrape metrics from Istio 1.6 and install latest Grafana Dashboards

ServiceMonitor

Prometheus-Operator is far more dynamic than the default Prometheus install. It adds some CRD to dynamically and transparently re-configure your Prometheus cluster.

A ServiceMonitor is a resource describing which pods to scrape based on a Service.

In Istio 1.6+ we have two types of things to monitor: Istio control-plane resources and Istio-proxy data-plane.

For that we create 2 different ServiceMonitor:

Control-Plane

apiVersion: monitoring.coreos.com/v1  
kind: ServiceMonitor  
metadata:  
  name: prometheus-oper-istio-controlplane  
  labels:  
    release: prometheus  
spec:  
  jobLabel: istio  
  selector:  
    matchExpressions:  
      - {key: istio, operator: In, values: [mixer,pilot,galley,citadel,sidecar-injector]}  
  namespaceSelector:  
    any: true  
  endpoints:  
  - port: http-monitoring  
    interval: 15s  
  - port: http-policy-monitoring  
    interval: 15s

If you know a bit of Prometheus, this is pretty easy to read:

  • look for any service with label istio equals to mixer , pilot…
  • scrape port named http-monitoring and http-policy-monitoring every 15s

The only thing to be careful about are the labels at the beginning: they are selectors that MUST match the Prometheus install serviceMonitorSelector. If you fail to do so, Prometheus will not consider this resource.
You can check how yours is configured by looking at the prometheus resource:

kubectl get prometheus  -o yaml | grep -A4 serviceMonitorSelector

    serviceMonitorSelector:  
      matchLabels:  
        release: prometheus

In my case, it is release: prometheus

As you can see from my example, this Prom Operator was installed using Helm. I know… sorry…

Data-Plane

The Data-Plane resource is quite the same but is targeting all the Istio-Proxy containers amd adds some relabeling:

apiVersion: monitoring.coreos.com/v1  
kind: ServiceMonitor  
metadata:  
  name: prometheus-oper-istio-dataplane  
  labels:  
    monitoring: istio-dataplane  
    release: prometheus  
spec:  
  selector:  
    matchExpressions:  
      - {key: istio-prometheus-ignore, operator: DoesNotExist}  
  namespaceSelector:  
    any: true  
  jobLabel: envoy-stats  
  endpoints:  
  - path: /stats/prometheus  
    targetPort: http-envoy-prom  
    interval: 15s  
    relabelings:  
    - sourceLabels: [__meta_kubernetes_pod_container_port_name]  
      action: keep  
      regex: '.*-envoy-prom'  
    - action: labelmap  
      regex: "__meta_kubernetes_pod_label_(.+)"  
    - sourceLabels: [__meta_kubernetes_namespace]  
      action: replace  
      targetLabel: namespace  
    - sourceLabels: [__meta_kubernetes_pod_name]  
      action: replace  
      targetLabel: pod_name

Again, pure Prom config. Just make sure you have the right label so the Operator will take care of the resource.

Add a label istio-prometheus-ignore=”true” to your deployments in case you don’t want Prometheus to scrape the proxy’s metrics.

Result

After few seconds for the whole thing to settle, you can connect to your Prom frontend, using Port-Forward on port 9090 or using the Istio Ingress-Gateway that you configured with SSL cert using SDS (check my older posts).

image

Grafana Dashboards

Now that you have Istio Telemetry V2 into your Prometheus cluster, you maybe want to see the graphs with Grafana.

Glad you read so far. I know this blog is missing some pictures and colors… but who cares, we are engineers right ?

Istio Dashboards for Grafana are stored in many places. You can find the latest in the Istio Github repo, but the best solution for you is to grab the one that matches your Istio install from the Istio install zip (or tar) where you grabbed istioctl !

From Istio docs, get it with:

curl -L [https://istio.io/downloadIstio](https://istio.io/downloadIstio) | sh -

This will create a folder with all the Istio stuffs. Note that Addons (Grafana, Kiali, Prometheus..) will NOT be managed by istioctl quite soon. You can find all the deployment scripts in this folder.

Dashboards are also located in this folder (istio-1.6.7 as the time of this writing) at manifests/charts/istio-telemetry/grafana/dashboards/

For them to be used by Grafana (the one installed by Prom Operator), you need to copy them inside a secret. Here’s the script I use for that (do a cd istio-<your-version>before using it:

#!/bin/bash
# go into the dashboards folder  
pushd manifests/charts/istio-telemetry/grafana/dashboards

# create the basic command to create the configmap  
ISTIO_DASHBOARD_SECRET="kubectl -n monitoring create cm prometheus-oper-istio-dashboards "

# append each file to the secret  
for i in *.json ; do  
  echo $i  
  ISTIO_DASHBOARD_SECRET="${ISTIO_DASHBOARD_SECRET} --from-file=${i}=${i}"  
done

# run the secret creation command  
eval $ISTIO_DASHBOARD_SECRET

# label the configmap so it is used by Grafana  
kubectl label -n monitoring --overwrite cm prometheus-oper-istio-dashboards grafana_dashboard=1

popd

Restart the Grafana pod and you should see the Dashboards in Grafana:

image