In the HTTP settings of Grafana you set Access to Proxy , which means that Grafana wants to access Prometheus. Since Kubernetes uses an overlay network, it is a different IP.

There are two ways of solving this: