Introduction
After providing a brief introduction to Istio in Part 1, this blog post will center on a detailed exploration of how Envoy intercepts all traffic, whether sourced or destined for the application container.
Our Application
To keep things straightforward, I’ve opted for the BookInfo application—a commonly featured example in Istio’s getting started guide. This application comprises four microservices: Product Page, Reviews, Details, and Ratings. While Reviews has multiple versions, we’ll keep our focus on v1.
For a visual overview, check out the diagram below, sourced from Istio’s documentation. It vividly illustrates the key components and their interactions.More details can be found here.
Product Page to Reviews Communication
In the upcoming sections, we will only focus on the communication between productspage and the reviews service particularly reviews-v1. Note that the reviews service has multiple versions as shown in the below figure and the configuration is done in such a way that traffic has to go to the reviews-v1 subset. Again, the objective is to be able to understand how does this communication flow end-to-end.
From ProductsPage Container to SideCar
Let’s start with the productspage container as it is the source of the request (Reference Figure1).
DNS request that is destined to CoreDNS to resolve reviews.default.svc.cluster.local.
The response comes back as 10.107.148.159 which is the Kubernetes service IP for the reviews service.
It is worth noting that one of the endpoints of that service is 10.0.0.38 (reviews-v1).
Now, the intriguing question arises: How does the packet from 10.0.1.136 (Product Page) to 10.0.0.38 (Reviews-v1) navigate its way through the Envoy proxy?
IPTables is the secret sauce behind getting the packet redirected to Envoy. As the packet exits the Product Page container, it encounters the IPTables chains listed below. Remember, our packet has a source of 10.0.1.136 and a destination of 10.0.0.38.
We will start with the OUTPUT chain which will send us to the ISTIO_OUTPUT Chain.
If you look at the ISTIO_OUTPUT chain the 1st rule doesn’t match as the source should be 127.0.0.6. The rules beneath generally match the UID 1337 which is traffic sourced by the Envoy sidecar thus these rules won’t match. In addition, there are rules that have an out interface of loopback which will not match. Thus, the last rule in the ISTIO_OUTPUT chain will match sending us towards the ISTIO_REDIRECT Chain.
Within the ISTIO_REDIRECT chain, we have a REDIRECT rule which redirects traffic to port 15001 i.e. it will change the destination address to localhost (127.0.0.1) and destination port to 15001.
The output below also proves that the Envoy proxy is actually listening on port 15001.
Up to this point, we’ve observed the packet departing the Product Page container and successfully reaching Envoy through IPTables redirection
Envoy Processing
In this section, our attention shifts to the processing of the packet by the productpage sidecar i.e. the Envoy proxy co-residing with the productpage container. The below outputs can be summarized as follows:
We examine listeners on port 15001, identifying the destination as PassthroughCluster. The PassthroughCluster’s type is ORIGINAL_DST, indicating that Envoy will revert the packet’s destination to its pre-IPTables redirection state, 10.0.0.38:9080.
Our attention then turns to Envoy routes, specifically focusing on reviews.default.svc.cluster.local:9080. The route view reveals the cluster as ‘outbound|9080|v1|reviews.default.svc.cluster.local’. Again, this is a result of the intended configuration where we want traffic that hits the reviews service to go to reviews-v1.
Finally, we need to figure out the specific endpoint within the Cluster which is the reviews-v1 pod address 10.0.0.38.
Lastly, inspecting the productpage Envoy logs, we observe that the request originates from 10.0.0.136 and is destined for 10.0.0.38:9080.
IPTables on Reviews-v1 Pod
As the packet from 10.0.0.136 (productspage) headed for 10.0.0.38 (reviews-v1) with a destination port of 9080 reaches the second node and enters the reviews-v1 pod network namespace, a familiar journey unfolds. Much like our previous process, the packet undergoes redirection to the Envoy sidecar, and once again, IPtables comes to the rescue.
We will start with the PREROUTING chain as this is an incoming packet. The PREROUTING chain will directly send us to the ISTIO_INBOUND chain.
If you look at the ISTIO_INBOUND chain the 1st 4 rules do not match as the destination port is not 15008, 15090, 15021, and 15020 (Destination Port: 9080). This takes us to the ISTIO_IN_REDIRECT chain.
Within the ISTIO_IN_REDIRECT chain, we have a REDIRECT rule which redirects traffic to port 15006 i.e. it will change the destination address to localhost (127.0.0.1) and destination port to 15006. Again you can refer to the netstat output to see that Envoy listens for port 15006.
Envoy Processing
Please note that this is the Envoy proxy (sidecar) of the reviews pod.
Upon inspecting the Envoy proxy within the reviews pod, a quick confirmation reveals a listener on Port 15006, aligning with expectations. This listener efficiently directs traffic to the Cluster ‘inbound|9080||.’ Examining the cluster further, denoted as ‘inbound|9080||,’ reveals its type as ‘ORIGINAL_DST,’ signifying that Envoy will revert the destination from the redirected address (127.0.0.1:15006) to the original one (10.0.0.38:9080).
Another interesting finding here is that the sourceAddress will be changed to 127.0.0.6. Thus, our packet will have a source as 127.0.0.6 & a destination as 10.0.0.38:9080. This can be seen in the below output.
Finally, the below figure puts it all together.
