Karim El Jamali

Dynamic Credentials with Hashicorp Vault

Karim El Jamali — Sat, 17 Aug 2024 16:47:01 GMT

The Hassle of Managing Secrets

It was around 3:15 AM when my pager buzzed for the first time ever, jolting me awake from sleep. At that time, I was working as a Technical Account Manager at AWS, and the situation was urgenta developer had mistakenly checked in AWS credentials to GitHub. Recently, I started going through an excellent course by Bryan Krausen on HashiCorp Vault, which connects directly to that issue and inspired me to write this post.

You can think of Hashicorp Vault as a super secret safe for your digital secrets. At a high level it can store your secrets safely, and request dynamic secrets which will be the focus of today's post, with granular levels of access control and a detailed audit trail. You can think about it as your one stop shop for all things credentials and secrets or even a front-end or a service by which all clients (users/developers/machines) can request access to those secrets.

Figure 1. Hassle of Managing Secrets

Dynamic Secrets

You can think of Vault as an API Gateway servicing all your credential and secret requests, bringing a unified experience & a common methodology on secrets management. But what are Dynamic Secrets?

Dynamic secrets are short-lived keys or credentials. Instead of giving out a permanent password or API key, Vault generates or requests a new one whenever you need it, and it expires automatically after a set period. A few reasons this might be very beneficial:

Better Security: This means that if someone gets hold of the key, it won't be useful for long, keeping your data safer. You can have credentials that last for a few minutes which in case leaked will have a much more limited damage as most likely they wouldn't be usable shortly. When employees leave the company, you don't have to worry about any static long lived credentials that they might still have to access your systems. In addition, all secrets are safely encrypted via multiple chains of encryption keys.
On Demand Credentials: Credentials are only there when you need them. You need to assume a particular AWS role, you ask Vault to generate dynamic short lived credentials to AWS to do a particular task.

High Level Overview

At a high level Vault works as follows:

Clients (Machines/Users) first need to authenticate to Vault. This is a critical step as it determines what the secrets the client can access or request. This authentication can happen via multiple methods some of which are shown in Figure 2. In addition, this authentication process can happen via CLI or UI. In this example, we will use Okta as our authentication method.
When Okta successfully authenticates the client, it returns back the Okta Group in which the user resides (e.g IT, SecOps..etc). Vault will make use of this group information and assign this user a policy. This policy basically authorizes the user or machine to authenticate for instance to AWS (simplified example here).
Now that Vault has identified the group and policy the user belongs to, it returns back a Token that is mapped to a policy that we discussed in step 2. The whole objective of this whole authentication process is to be able to get a token that is constrained by an associated policy.
Finally, the client can request credentials against a particular AWS Role i.e. AssumeRole will be requested by Vault against the particular AWS Role to receive Dynamic or time-bound AWS credentials (AWS Access Key, Secret Key, Session Token). AWS is just an example here, it could be a full Multi-Cloud environment, local secrets, Kubernetes, databases and much more. The full list of supported secret engines can be found here.

Figure 2. Vault High Level Architecture

Let's Dive Deeper

Setup Configuration

The objective of this section is to explain the setup from a configuration & object standpoint. In this section we are assuming the following:

Prerequisites:

There are two AWS Accounts Prod & Dev that users need access to.
For simplification purposes, within the Prod as well as Dev accounts there are two IAM roles. S3:readonly provides read only access to S3 storage and S3:readwrite allows full S3 access. This configuration is fully done within AWS. Enterprise setups would have a way larger number of accounts & roles but the same logic and object relationships continues to hold true.

These are the configuration steps:

First and foremost, Vault needs to have access to the AWS accounts to be able to request dynamic credentials from within these accounts. This can be done via credentials (Access Key, Secret Key) or via an instance Profile if the Vault server itself runs within AWS. The IAM permissions used by Vault are documented here.
Customers will generally have an Identity Provider (IdP) which in our case is Okta. Within Okta, most customers would group users depending on the role or function within the organization. It is important that the groups within Okta that would require Vault interaction be mapped to groups within Vault.
Each group within Vault (1:1 mapping of Okta/IdP group) has an associated Vault Policy. This Policy for instance would allow a particular group to access particular paths (Yes, everything in Vault is a path) and eventually assume particular Vault roles. Our policy in Figure3, allows access to prod/s3-readonly & dev/s3-readonly roles.
The Vault roles are indeed a 1:1 mapping of the roles within the different AWS accounts.

Figure 3. Setup Configuration

Object Relationships

Figure 4 tries to simplify the object relationships.

A Vault Group needs to have an associated Vault Policy. Recall that in our setup configuration, these Vault groups within the Vault Server are a one to one mapping of the external IdP groups (Okta). The vault policy can in turn provide access to one or more paths (roles) where these roles are again a one to one mapping of the roles configured within AWS.

Figure 4. Object Relationships

Putting it All Together

In this section, we will tackle it end-to-end from the user's perspective again touching on some of the concepts we explained for an end-to-end understanding.

Adam is a user belonging to Group: s3-readonly in Okta.
Adam tries to authenticate to Vault by using the Okta authentication method supplying his credentials (CLI/UI). Okta authenticates Adam successfully and returns back the group name Group: s3-readonly.
As mentioned the groups in the IdP that need Vault access should have corresponding vault roles (Group: s3-readonly).
The vault group s3-readonly is associated with a policy that provides access to multiple paths (roles) in our example prod/s3-readonly & dev/s3-readonly. Note that each of those roles within Vault have corresponding IAM roles in the prod and dev accounts.
Now that authentication is complete and Vault maps the user to a particular policy, Vault issues a token and provides it to Adam.
With the token & correct permissions, Adam goes ahead and requests credentials against the Vault group prod/s3-readonly.
Since the vault policy allows Adam to access the Vault role prod/s3-readonly, Vault with its own credentials requests AWS STS for temporary credentials against the IAM role s3-readonly within the Prod account.
Vault will return the credentials to Adam consisting of Access Key, Secret Key and Session token.
Adam can now issue the command "aws s3 ls" to list the buckets within S3 with the received credentials.

Figure 5. Putting it All Together

Lab Setup

This is the lab setup we will use for demonstration. For Vault to be able to communicate with the AWS Account Prod, I associated the Vault EC2 instance with an IAM role via an Instance Profile.

Figure 6. Lab Setup

The below code snippet shows the Vault group within Okta "s3-ro-group-role" that maps to an Okta group name mapping it to a Vault Policy "s3-ro-access-role-policy".

# Creating the group within Vault and mapping it to a policyvault write auth/okta/groups/s3-ro-group-role \      policies="s3-ro-access-role-policy"#Creating the policy s3-ro-access-role-policyvault policy write s3-ro-access-role-policy s3-ro-access-policy.hcl

Figure 7. Creating Auth Group within Vault & mapping it to a Vault Policy

The below figure shows the details of the Vault policy.

#Contents of the Policy[ec2-user@ip-10-0-5-142 ~]$ vault policy read s3-ro-access-role-policy#The below path allows access to request credentials (read capability in #Vault) for the role s3-ro-access-sts. Note the path has credspath "aws/creds/s3-ro-access-sts" {  capabilities = ["read"]}#The below path allows access to read the configuration of the role s3-ro-access-stspath "aws/roles/s3-ro-access-sts" {  capabilities = ["read"]}# If you want to allow users to renew the leases for the same role# Renew leases means rotating the credentialspath "sys/leases/lookup/aws/creds/s3-ro-access-sts" {  capabilities = ["read"]}#Ability to revoke the leasepath "sys/leases/revoke/aws/creds/s3-ro-access-sts" {  capabilities = ["update"]}#Ability to list all available roles, note this doesn't allow issuing credentials# towards these rolespath "aws/roles/*" {  capabilities = ["list"]}

Figure 8. Vault Policy s3-ro-access-role-policy

The code snippet in Figure 9 shows the mapping between the Vault role "s3-ro-access-sts" and the IAM role of the same name. Note that the ttl (default_sts_ttl) is set to 15 min and the max_sts_ttl (max ttl) is 24 hours. This means that that credentials generated against this rule will live for 15 minutes but can be renewed or rotated where applicable until the max_sts_ttl is reached. Once max_sts_ttl is reached, there is no possibility to renew the credential.

# Finally this is a mapping between the Vault role s3-ro-access-sts &# the IAM Role ARN. Account number details are hidden # default_sts_ttl is 15 min means that credentials will live for a period of 15 min# max_sts_ttl=24h means you can request credential renewal where applicablevault write aws/roles/s3-ro-access-sts \    role_arns="arn:aws:iam::XXXXXXXXXXXX:role/s3-ro-access-sts" \    credential_type=assumed_role \    default_sts_ttl=15m    max_sts_ttl=24h#reading the configuration of the role with the same details[ec2-user@ip-10-0-5-142 ~]$ vault read aws/roles/s3-ro-access-stsKey                         Value---                         -----credential_type             assumed_roledefault_sts_ttl             15mexternal_id                 n/aiam_groups                  <nil>iam_tags                    <nil>max_sts_ttl                 24hmfa_serial_number           n/apermissions_boundary_arn    n/apolicy_arns                 <nil>policy_document             n/arole_arns                   [arn:aws:iam::871253670082:role/s3-ro-access-sts]session_tags                <nil>user_path                   n/a

Figure 9. Mapping Vault Roles to IAM Roles in AWS

Now let's try to have a client authenticate.

# Login with user credentials and receive the Vault token. # At this stage you only logged into to Vault successfully and have a token# with an associated policy (s3-ro-access-role-policy)vault login -method=okta \    username="david@gmail.com" \    password="XXXXXXXXXX"Success! You are now authenticated. The token information displayed belowis already stored in the token helper. You do NOT need to run "vault login"again. Future Vault requests will automatically use this token.Key                    Value---                    -----token                  hvs.CAESIMniUaK0hhXKCGQaCQMkIR-P8K4UkKdQDAybYTWPQRLKGh4KHGh2cy5CcHRid01ETFhENmk5bWlCZGFoUk43ZTEtoken_accessor         jEUNtHfL7PGTFxIAOzxyJgNytoken_duration         768htoken_renewable        truetoken_policies         ["default" "s3-ro-access-role-policy"]identity_policies      []policies               ["default" "s3-ro-access-role-policy"] #policy associated with the vault tokentoken_meta_policies    s3-ro-access-role-policytoken_meta_username    david@gmail.com#Now david our user has requested credentials against the role s3-ro-access-sts# Since the Vault policy allows it the request is successful#david receives short-lived credentials (access key, secret key and session_token)#note that there is no need for me to remove the credentials from the output because#their lifetime is 15 minvault read aws/creds/s3-ro-access-stsKey                Value---                -----lease_id           aws/creds/s3-ro-access-sts/sfB1tR8zvGSLqlxyxS6UATFxlease_duration     15mlease_renewable    falseaccess_key         ASIA4VWWD7DBJJFME3HOarn                arn:aws:sts::871253670082:assumed-role/s3-ro-access-sts/vault-root-s3-ro-access-sts-1723777259-eAMrdCkR95w7Bv38YBbcsecret_key         ch3xhCWqtz5rO2S5nnuj505Rjajy0WYxg9kDZrGqsecurity_token     FwoGZXIvYXdzEPz//////////wEaDH3IMV5gEXgYudFS9iLfAfytgc1Ajloy83I+nyqFP92v++K60uGInLXQsWtztYEyCdRtNsGXfaumjU/Fuea2ZB6x3mLq7u/D8T5WQqU4/AHmecy0Gl2lGdsP6EFivLCVejJDVbgprDEVTZzDsBwIPpXsoMDmB9+9bxAxsVTEtifpqnrcG5WkuSne2OfrYEu6AwvDxp7HdhpgqXgabsxfT7zfO3WFyzEFutmM3N8yjyc001kROOxFxJojzDBAvo7PejN5tf2wjlWXdi8ZnxqjhSGaIcAn1rVCjpWM3DGn7wTUhzW97GMT2eaUsiBrw1so64H7tQYyLXiVIc3xZ3j1HbueByMKVUqDkuy885fM6CeIuVfW+KcLjsPahfngTb8O52Wbfg==session_token      FwoGZXIvYXdzEPz//////////wEaDH3IMV5gEXgYudFS9iLfAfytgc1Ajloy83I+nyqFP92v++K60uGInLXQsWtztYEyCdRtNsGXfaumjU/Fuea2ZB6x3mLq7u/D8T5WQqU4/AHmecy0Gl2lGdsP6EFivLCVejJDVbgprDEVTZzDsBwIPpXsoMDmB9+9bxAxsVTEtifpqnrcG5WkuSne2OfrYEu6AwvDxp7HdhpgqXgabsxfT7zfO3WFyzEFutmM3N8yjyc001kROOxFxJojzDBAvo7PejN5tf2wjlWXdi8ZnxqjhSGaIcAn1rVCjpWM3DGn7wTUhzW97GMT2eaUsiBrw1so64H7tQYyLXiVIc3xZ3j1HbueByMKVUqDkuy885fM6CeIuVfW+KcLjsPahfngTb8O52Wbfg==ttl                14m59s

Figure 10. Client authenticates to Vault, receives token & requests AWS credentials

#Prior to getting credentials failing to list the buckets in S3% aws s3 lsAn error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.#Inputting the credentials% export AWS_ACCESS_KEY_ID="ASIA4VWWD7DBJJFME3HO"export AWS_SECRET_ACCESS_KEY="ch3xhCWqtz5rO2S5nnuj505Rjajy0WYxg9kDZrGq"export AWS_SESSION_TOKEN="FwoGZXIvYXdzEPz//////////wEaDH3IMV5gEXgYudFS9iLfAfytgc1Ajloy83I+nyqFP92v++K60uGInLXQsWtztYEyCdRtNsGXfaumjU/Fuea2ZB6x3mLq7u/D8T5WQqU4/AHmecy0Gl2lGdsP6EFivLCVejJDVbgprDEVTZzDsBwIPpXsoMDmB9+9bxAxsVTEtifpqnrcG5WkuSne2OfrYEu6AwvDxp7HdhpgqXgabsxfT7zfO3WFyzEFutmM3N8yjyc001kROOxFxJojzDBAvo7PejN5tf2wjlWXdi8ZnxqjhSGaIcAn1rVCjpWM3DGn7wTUhzW97GMT2eaUsiBrw1so64H7tQYyLXiVIc3xZ3j1HbueByMKVUqDkuy885fM6CeIuVfW+KcLjsPahfngTb8O52Wbfg=="#Succeeded in listing the buckets in S3% aws s3 ls2023-11-01 13:57:10 kops-configuration-test

Figure 11. Checking Credentials

# Here you can see the leases (credentials requested via Vault) # Notice we only requested credentials for the role s3-ro-access-sts[ec2-user@ip-10-0-5-142 ~]$ vault list sys/leases/lookup/aws/creds/Keys----s3-ro-access-sts/#Here you can see the lease id sfB1tR8zvGSLqlxyxS6UATFx[ec2-user@ip-10-0-5-142 ~]$ vault list sys/leases/lookup/aws/creds/s3-ro-access-sts/Keys----sfB1tR8zvGSLqlxyxS6UATFx#Here you can see details of the lease. Just want to show the ttl 6m57s #this is the time after which these credentials will expire[ec2-user@ip-10-0-5-142 ~]$ vault lease lookup aws/creds/s3-ro-access-sts/sfB1tR8zvGSLqlxyxS6UATFxKey             Value---             -----expire_time     2024-08-16T03:15:59.000032318Zid              aws/creds/s3-ro-access-sts/sfB1tR8zvGSLqlxyxS6UATFxissue_time      2024-08-16T03:00:59.177372978Zlast_renewal    <nil>renewable       falsettl             6m57s#Another run of the same command now showing the ttl is now 28s[ec2-user@ip-10-0-5-142 ~]$ vault lease lookup aws/creds/s3-ro-access-sts/sfB1tR8zvGSLqlxyxS6UATFxKey             Value---             -----expire_time     2024-08-16T03:15:59.000032318Zid              aws/creds/s3-ro-access-sts/sfB1tR8zvGSLqlxyxS6UATFxissue_time      2024-08-16T03:00:59.177372978Zlast_renewal    <nil>renewable       falsettl             28s#We can no longer find the lease (ttl is expired)[ec2-user@ip-10-0-5-142 ~]$ vault lease lookup aws/creds/s3-ro-access-sts/sfB1tR8zvGSLqlxyxS6UATFxerror looking up lease id aws/creds/s3-ro-access-sts/sfB1tR8zvGSLqlxyxS6UATFx: Error making API request.URL: PUT http://127.0.0.1:8200/v1/sys/leases/lookupCode: 400. Errors:* invalid lease#Trying again to list the buckets in S3 and getting an ExpiredToken error% aws s3 lsAn error occurred (ExpiredToken) when calling the ListBuckets operation: The provided token has expired.

Figure12. Details of the lease & TTL

Audit Logs

Audit logs are a necessity especially when it comes to your critical assets and attempts to access them.

This is how you can enable Audit Logs for Vault in a lab environment.

vault audit enable file file_path=/var/log/vault_audit.log

This is a sample log message that shows the user david authenticating and getting a token (client_token field) which is redacted in this output that is associated with the policy s3-ro-access-role-policy.

{  "auth": {    "client_token": "REDACTED",    "display_name": "okta-david@gmail.com",    "metadata": {      "policies": "s3-ro-access-role-policy",      "username": "david@gmail.com"    },    "policies": ["default", "s3-ro-access-role-policy"],    "token_ttl": 2764800,    "token_type": "service"  },  "request": {    "id": "REDACTED",    "mount_point": "auth/okta/",    "operation": "update",    "path": "auth/okta/login/david@gmail.com"  },  "response": {    "auth": {      "client_token": "REDACTED",      "display_name": "okta-david@gmail.com",      "metadata": {        "policies": "s3-ro-access-role-policy",        "username": "david@gmail.com"      },      "policies": ["default", "s3-ro-access-role-policy"],      "token_ttl": 2764800,      "token_type": "service"    },    "mount_point": "auth/okta/"  },  "type": "response"}

]]>

Intro to Kyverno

Karim El Jamali — Fri, 31 May 2024 02:43:16 GMT

Last week, a new tool called Kyverno captured my interest. I spent some time diving into their documentation to understand the benefits and use cases it addresses.

Kyverno is a policy engine designed specifically for Kubernetes. But what exactly is a policy engine? Simply put, it sets acceptable configuration standards for Kubernetes objects and resources. When these standards are deviated from, the engine can block the deployment, send alerts, or even mutate the configurations to comply with the policies. The main use cases for a policy engine like Kyverno revolve around enhancing security, ensuring compliance, automating tasks, and improving governance within Kubernetes clusters.

Basics of Admission Controllers

An admission controller is a component within Kubernetes that intercepts requests to the Kubernetes API server before the object is persisted, but after the request has been authenticated and authorized. As shown in the figure below, when a request to create a deployment is sent to the API server, it first undergoes authentication and authorization. Afterward, it is passed to the Admission Controller, depicted as a police officer in the diagram. In this scenario, the Admission Controller is configured to ensure that any deployment must have a minimum of two replicas to maintain high availability (HA).

Thus, the best way to think about an admission controller is as a police officer that examines requests sent to the API server to ensure they align with the organization's intent. It acts as a controller or watchdog, persistently ensuring conformance to organizational policies and intents.

Built in Admission Controllers

It's worth noting that Kubernetes includes several built-in admission controllers.. You can find a list of these controllers here.

Before delving further, it's important to differentiate between the two general types of admission controllers:

Validating Admission Controllers: These controllers simply allow or reject requests.
Mutating Admission Controllers: These controllers have the ability to modify objects related to the requests they receive.

Let's take the example of LimitRanger, an in-built admission controller or plugin. A LimitRange defines policies to govern resource allocations for pods or other applicable resources. More details can be found here. It's important to note that LimitRange functions as both a validating and a mutating admission controller.

To try it out, the first step is to add it to the list of admission controllers or plugins using the "--enable-admission-plugins" flag.

apiVersion: v1kind: Podmetadata:  annotations:    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 172.18.0.3:6443  creationTimestamp: null  labels:    component: kube-apiserver    tier: control-plane  name: kube-apiserver  namespace: kube-systemspec:  containers:  - command:    - kube-apiserver    - --advertise-address=172.18.0.3    - --allow-privileged=true    - --authorization-mode=Node,RBAC    - --client-ca-file=/etc/kubernetes/pki/ca.crt    - --enable-admission-plugins=NodeRestriction,LimitRanger

Now, let's define the LimitRange object where we configure:

Minimum and maximum CPUs for all pods within the cluster.
Default limits applied when a user doesn't explicitly configure the resources for a pod.

This configuration ensures that all pods adhere to the specified CPU limits, promoting efficient resource utilization across the cluster. Additionally, it provides a safety net by automatically assigning default limits when users do not specify resource constraints for their pods.

apiVersion: v1kind: LimitRangemetadata:  name: cpu-resource-constraintspec:  limits:  - default: # this section defines default limits      cpu: 500m    defaultRequest: # this section defines default requests      cpu: 500m    max: # max and min define the limit range      cpu: 600m    min:      cpu: 100m    type: Container

Below is an example of a Pod configuration that requests 700m of CPU resources, which exceeds the defined limit of 500m. When attempting to apply this configuration, you'll immediately encounter an error, showcasing the validating behavior of the admission controller.

apiVersion: v1kind: Podmetadata:  name: example-conflict-with-limitrange-cpuspec:  containers:  - name: demo    image: registry.k8s.io/pause:2.0    resources:      requests:        cpu: 700m

The Pod "example-conflict-with-limitrange-cpu" is invalid: spec.containers[0].resources.requests: Invalid value: "700m": must be less than or equal to cpu limit of 500m

Now, let's attempt to create another pod without specifying any resource requests or limits.

kubectl run nginx --image nginx

We can observe that the default requests and limits have been automatically applied to the pod, courtesy of the admission controller. This exemplifies a mutating behavior, wherein the original request has been modified to include the necessary resource requests and limits.

kubectl get pods nginx -o yaml spec:  containers:  - image: nginx    imagePullPolicy: Always    name: nginx    resources:      limits:        cpu: 500m      requests:        cpu: 500m

External Admission Controllers

In the previous section, we delved into the inherent admission controllers within Kubernetes. However, these built-in controllers often lack the configurability or flexibility needed to accommodate diverse organizational requirements.

Below is a diagram (Source: Kyverno documentation) illustrating the stages of an API request to a Kubernetes API Server, highlighting the crucial steps involving Mutating and Validating admissions.

Here's a concise overview of the mentioned components:

Webhook Controller: This controller is responsible for managing webhooks, which play a pivotal role in programming the Kubernetes API server to forward API requests to Kyverno. Webhooks are essential building blocks that enable seamless integration between Kubernetes and Kyverno, facilitating the enforcement of policies and configurations.
Policy Engine: Kyverno's policy engine serves as the cornerstone component tasked with evaluating and enforcing policies on Kubernetes resources. It empowers administrators to define policies as Kubernetes resources within Kyverno, enabling them to manage, validate, and mutate other Kubernetes resources effectively. By processing these policies, the policy engine ensures that the cluster complies with desired configurations and standards, fostering consistency and adherence to organizational policies.
Cert Renewer: This component focuses on managing TLS (Transport Layer Security) communications between the Kubernetes API Server and the Kyverno webhook service. By handling TLS certificates, the Cert Renewer ensures secure transmission of data between these components, safeguarding the integrity and confidentiality of communications.

The ValidatingWebhookConfiguration plays a crucial role in configuring the Kubernetes API server to route requests to Kyverno. Here's how it works:

Webhooks Section: This section defines where to redirect matching API requests. It typically references a Kubernetes Service and specifies a particular path to send the requests for validation.
Rules Section: The rules section allows you to specify the resources and operations that you want to send to the admission controller for validation. This granularity enables precise control over which requests are subject to validation.
Namespace Selector: This field enables you to focus admission control on specific namespaces where stricter governance is required or where you want to apply policies gradually. It provides flexibility in applying policies across different parts of the cluster.
Failure Policy: This field determines how the Kubernetes API server responds when it receives a negative admission response from the admission controller. The options are "Fail" and "Ignore". "Fail" immediately rejects the intended operation upon receiving a negative admission response, ensuring strict enforcement of policies. On the other hand, "Ignore" provides more leniency by allowing the operation to proceed despite policy violations. This can be useful when you want to identify and understand gaps before fully enforcing policies.

apiVersion: admissionregistration.k8s.io/v1kind:     ValidatingWebhookConfigurationmetadata:  name: kyverno-validating-webhookwebhooks:  - name: validate.kyverno.svc    clientConfig:      service:        name: kyverno-svc        namespace: kyverno        path: "/validate"      caBundle: <ca-bundle> # Base64 encoded CA certificate    rules:      - operations: ["CREATE", "UPDATE"]        apiGroups: ["*"]        apiVersions: ["*"]        resources: ["*"]    failurePolicy: Fail    matchPolicy: Equivalent    namespaceSelector: {}    objectSelector: {}    sideEffects: None    timeoutSeconds: 10    admissionReviewVersions: ["v1"]

Sample Policies

This example policy ensures that each configured pod must have the "app" label.

apiVersion: kyverno.io/v1kind: ClusterPolicymetadata:  name: check-label-appspec:  validationFailureAction: Audit  background: false  validationFailureActionOverrides:    - action: Enforce     # Action to apply      namespaces:       # List of affected namespaces        - default  rules:    - name: check-label-app      match:        any:        - resources:            kinds:            - Pod      validate:        message: "The label `app` is required."        pattern:          metadata:            labels:              app: "?*"

Explanation:

ClusterPolicy: as the name suggests is a policy that applies Cluster wide vs policy which is scoped to the namespace.
background: When set to true, this field ensures that the policy is applied retroactively to existing resources upon creation or update. This means that the policy will be evaluated against resources that were created before the policy was implemented.
validationFailureAction: This field determines the default action to take when a validation rule fails. In this case, it is set to "audit", meaning that failed validations will be recorded, but resource creation will continue.
validationFailureActionOverrides: This section allows you to override the default validation failure action for specific namespaces. In the example, validation failures in the default namespace will be enforced, overriding the default action set in validationFailureAction.
rules: This section defines the rules of the policy. In this example, there is one rule named "check-app-label" that applies to Pods. The rule specifies that Pods must have an 'app' label.

Attempting to create an nginx pod within the blue namespace succeeds with a warning as per the policy.

kubectl -n blue run nginx --image=nginxWarning: policy check-label-app.check-label-app: validation error: The label `app` is required. rule check-label-app failed at path /metadata/labels/app/pod/nginx created

Attempting the same operation within the default namespace is denied again as per policy.

k.eljamali@LLJXN1XJ9X admission % kubectl run nginx --image=nginx Error from server: admission webhook "validate.kyverno.svc-fail" denied the request: resource Pod/default/nginx was blocked due to the following policies check-label-app:  check-label-app: 'validation error: The label `app` is required. rule check-label-app    failed at path /metadata/labels/app/'

Here's the example policy enforcing a minimum level of high availability for deployments:

apiVersion: kyverno.io/v1kind: ClusterPolicymetadata:  name: validatespec:  validationFailureAction: Enforce  rules:    - name: validate-replica-count      match:        any:        - resources:            kinds:            - Deployment      validate:        message: "Replica count for a Deployment must be greater than or equal to 2."        pattern:          spec:            replicas: ">=2"

Cilium Policies

After reviewing several examples, I've considered implementing Kyverno for admission control of Cilium Network Policies. For that matter I found an interesting repository here that helped me get started.

After making little modifications to various examples on GitHub, I crafted a policy aimed at whitelisting specific namespaces where Cilium policies can be configured. Any attempts to create Cilium policies in namespaces not included in the whitelist will be denied. I think this is pretty useful to ensure no mishaps can happen as part of Network Policy configuration and helps with locking things down one namespace at a time.

apiVersion: kyverno.io/v1kind: ClusterPolicymetadata:  name: limit-ciliumnetworkpolicy-scopespec:  validationFailureAction: enforce  background: false  rules:  - name: limit-scope    match:      any:      - resources:          kinds:          - CiliumNetworkPolicy    validate:      message: "CiliumNetworkPolicy can only be applied to specific namespaces"      anyPattern:      - metadata:            namespace: blue      - metadata:            namespace: default

I attempt to use the below Network Policy.

apiVersion: "cilium.io/v2"kind: CiliumNetworkPolicymetadata:  name: "isolate-ns1"  namespace: greenspec:  endpointSelector:    matchLabels:      {}  ingress:  - fromEndpoints:    - matchLabels:        {}

As expected, the request is blocked by the admission controller.

kubectl apply -f mypolicy.yaml Error from server: error when creating "mypolicy.yaml": admission webhook "validate.kyverno.svc-fail" denied the request: resource CiliumNetworkPolicy/green/isolate-ns1 was blocked due to the following policies limit-ciliumnetworkpolicy-scope:  limit-scope: 'validation error: CiliumNetworkPolicy can only be applied to specific    namespaces. rule limit-scope[0] failed at path /metadata/namespace/ rule limit-scope[1]    failed at path /metadata/namespace/'

The example below provides more insight on Kyverno's capabilities whereby we leverage a ClusterPolicy whose objective is to ensure that pods to be created need to have the label "app.kubernetes.io/name".

apiVersion: kyverno.io/v1kind: ClusterPolicymetadata:  name: require-labels  annotations:    policies.kyverno.io/title: Require Labels    policies.kyverno.io/category: Best Practices    policies.kyverno.io/minversion: 1.6.0    policies.kyverno.io/severity: medium    policies.kyverno.io/subject: Pod, Label    policies.kyverno.io/description: >-      Define and use labels that identify semantic attributes of your application or Deployment.      A common set of labels allows tools to work collaboratively, describing objects in a common manner that      all tools can understand. The recommended labels describe applications in a way that can be      queried. This policy validates that the label `app.kubernetes.io/name` is specified with some value.spec:  validationFailureAction: Enforce  background: true  rules:  - name: check-for-labels    match:      any:      - resources:          kinds:          - Pod    validate:      message: "The label `app.kubernetes.io/name` is required."      pattern:        metadata:          labels:            app.kubernetes.io/name: "?*"

Now, I try to create a deployment (not a Pod) & interestingly enough the admission controller reacts in the right manner not allowing the deployment to happen.

kubectl create deployment kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1error: failed to create deployment: admission webhook "validate.kyverno.svc-fail" denied the request: resource Deployment/default/kubernetes-bootcamp was blocked due to the following policies require-labels:  autogen-check-for-labels: 'validation error: The label `app.kubernetes.io/name`    is required. rule autogen-check-for-labels failed at path /spec/template/metadata/labels/app.kubernetes.io/name/'

Conclusion

Kyverno and more generally admission controllers can help ensure better security, compliance and governance of Kubernetes Clusters. On top of this, there is a good number of readily available policies making it easier to consume. I even tried asking ChatGPT to generate a Kyverno policy:

please craft a kyverno policy that would allow cilium network policies to only touch pods that have environment: dev as not to disrupt production

apiVersion: kyverno.io/v1kind: ClusterPolicymetadata:  name: restrict-cilium-policyspec:  background: true  validationFailureAction: enforce  rules:  - name: restrict-cilium-network-policy    match:      resources:        kinds:        - CiliumNetworkPolicy    validate:      message: "Cilium Network Policies can only target pods with the label 'environment: dev'."      pattern:        spec:          endpointSelector:            matchLabels:              environment: dev

It might be worthwhile if you are getting started to deploy it to have the validationFailureAction as Audit as this would provide context on how objects being created deviate from the norm and have the background set to true as this will ensure that your existing resources are scanned in terms of adherence to organizational policies.ValidateActionOverride will be useful to enforce policies on namespaces one by one to ensure a smooth adoption.

Finally, a good one to mention is Policy Reports that provide insights into the compliance status of resources in the Kubernetes cluster. Though I have noticed that some resources don't show up in the reports, it is still very useful when background is set to true as this will give an indication of the compliance state of all resources within Kubernetes.

kubectl get policyreports.wgpolicyk8s.io -ANAMESPACE            NAME                                   KIND         NAME                                             PASS   FAIL   WARN   ERROR   SKIP   AGEcilium-monitoring    6228a955-195b-4795-83ee-5c1a662bf532   Deployment   prometheus                                       0      1      0      0       0      8dcilium-monitoring    81242176-f6ac-4331-8be3-bdedccc32f67   Pod          grafana-6f4755f98c-4cxzx                         0      1      0      0       0      8dcilium-monitoring    bbe6c1f9-a91f-4628-b524-bf04c2603c0a   ReplicaSet   grafana-6f4755f98c                               0      1      0      0       0      8dcilium-monitoring    c70e65cc-800a-4986-90ab-c52078189f29   ReplicaSet   prometheus-67fdcf4796                            0      1      0      0       0      8dcilium-monitoring    d7006776-21e2-4b86-a684-d6cf173f2f2e   Pod          prometheus-67fdcf4796-gj4kb                      0      1      0      0       0      8dcilium-monitoring    dd37f6a5-0bd7-43f8-a7d5-337f089df95f   Deployment   grafana                                          0      1      0      0       0      8dkube-system          0064fbf9-80ce-4517-934e-5b34d98e88c5   DaemonSet    cilium                                           0      1      0      0       0      8dkube-system          0be0cbb5-25b0-448e-922c-074e510d269c   Pod          cilium-zx692                                     0      1      0      0       0      8dkube-system          0d9c47b0-2b55-4752-ae1f-1d22ab63e9c9   Pod          cilium-operator-5d64788c99-g2f48                 0      1      0      0       0      8dkube-system          14ae51f5-ba2f-4942-aa35-d93e94d2d76f   Pod          kube-scheduler-kind-control-plane                0      1      0      0       0      8dkube-system          23a370a8-ef0e-4d1c-aa19-ffab703b5f0d   DaemonSet    kube-proxy                                       0      1      0      0       0      8dkube-system          35e9706f-7a88-4932-a78c-37745a397abe   Pod          coredns-76f75df574-xvxns                         0      1      0      0       0      8dkube-system          501429e0-4f00-4f30-9cc8-9eb633d9a0e5   ReplicaSet   coredns-76f75df574                               0      1      0      0       0      8dkube-system          6911c7d8-4128-470d-89ef-6fc969e62b05   Pod          kube-controller-manager-kind-control-plane       0      1      0      0       0      8dkube-system          69a7f09e-a753-4dd9-8fe2-e5eb70e3691d   Pod          kube-proxy-jvllh                                 0      1      0      0       0      8dkube-system          8c6c30a5-1ef6-4bd1-9978-381b0aeca611   Deployment   cilium-operator                                  0      1      0      0       0      8dkube-system          9cd0150a-34fd-49df-837b-d2f19ea575dd   ReplicaSet   cilium-operator-5d64788c99                       0      1      0      0       0      8dkube-system          9ecef8f0-c9b1-4989-9fa6-15b07f8de581   Deployment   coredns                                          0      1      0      0       0      8dkube-system          a8482fb9-10bc-4819-8230-17a776d1079f   Pod          cilium-kfrnt                                     0      1      0      0       0      8dkube-system          be23c6c2-8ad5-4a86-adc1-de4180dc8d81   Pod          kube-proxy-g4gnt                                 0      1      0      0       0      8dkube-system          cc4021e0-2360-403b-b013-fad70807f04b   Pod          coredns-76f75df574-7zdjj                         0      1      0      0       0      8dkube-system          dba5f38a-e8e5-46d0-9e4a-ec269ae9ddb4   Pod          cilium-bztgh                                     0      1      0      0       0      8dkube-system          e0146bde-f48c-4663-997f-019e24402bcd   Pod          cilium-operator-5d64788c99-hdlsz                 0      1      0      0       0      8dkube-system          eadc9e7d-90dd-42c1-9dbe-62b3822d668a   Pod          kube-proxy-fn567                                 0      1      0      0       0      8dkube-system          f3d97e60-2a0f-4985-9644-a98e98541103   Pod          etcd-kind-control-plane                          0      1      0      0       0      8dkube-system          ff45c31d-ca5d-43c5-90da-bb0f113b03b3   Pod          kube-proxy-6jfqw                                 0      1      0      0       0      8dkyverno              0840b270-f6a2-499c-b1d7-a37a716ef440   Pod          kyverno-background-controller-84b5ff7875-fkd2x   0      1      0      0       0      8dkyverno              13f74dc4-4e27-4a78-b34c-cc9ae3e7b57c   CronJob      kyverno-cleanup-admission-reports                0      1      0      0       0      8dkyverno              27bf13f4-2bc8-4dae-8e0c-2a29d57438c0   Deployment   kyverno-reports-controller                       0      1      0      0       0      8dkyverno              29bcbffc-d253-4d8e-97d6-98043bf9600b   CronJob      kyverno-cleanup-cluster-admission-reports        0      1      0      0       0      8dkyverno              532c51d8-843e-42ae-82d2-d5a8f6f9b62e   Pod          kyverno-cleanup-controller-574fc8c6b6-kpbbg      0      1      0      0       0      8dkyverno              6fea3745-3b96-46c9-9c62-59cc22f75722   Deployment   kyverno-admission-controller                     0      1      0      0       0      8dkyverno              80e5c9d7-9677-4a71-a4c3-89370afd665e   ReplicaSet   kyverno-cleanup-controller-574fc8c6b6            0      1      0      0       0      8dkyverno              94048b9d-8a52-4f70-bbc4-cb814b8ee671   Pod          kyverno-reports-controller-5557c85bb4-8jpm6      0      1      0      0       0      8dkyverno              a0948eaa-9332-488c-a7a9-cb9c3ee4b045   ReplicaSet   kyverno-admission-controller-6797d4bd6b          0      1      0      0       0      8dkyverno              ab1c5601-072a-4782-b08a-d9c2e9ae3d92   Deployment   kyverno-cleanup-controller                       0      1      0      0       0      8dkyverno              ead76fa0-bfec-4385-9a00-4b26293c069b   Pod          kyverno-admission-controller-6797d4bd6b-l2kwf    0      1      0      0       0      8dkyverno              f47b4093-46fd-4eb7-abe3-0e431a98d23a   ReplicaSet   kyverno-background-controller-84b5ff7875         0      1      0      0       0      8dlocal-path-storage   13e4514a-18e1-484d-aa39-c591f34c4d56   Pod          local-path-provisioner-7577fdbbfb-6hb2v          0      1      0      0       0      8dlocal-path-storage   45119b4b-a214-4d6c-b06a-7c8a2c4af402   ReplicaSet   local-path-provisioner-7577fdbbfb                0      1      0      0       0      8dlocal-path-storage   adc91343-2e69-4059-aebb-1e4a50f80a2b   Deployment   local-path-provisioner                           0      1      0      0       0      8d

]]>

Cilium: Network Policies

Karim El Jamali — Wed, 01 May 2024 12:50:43 GMT

Introduction to Network Policies

In this blog episode, we're diving into the world of Kubernetes Network Policies. But before we get there, let's talk about the main headache these policies aim to fix. So, without Network Policies, Kubernetes lets Pods communicate freely with each other, no matter if they are on the same or different node or within the same namespace or in different namespaces. But here's the catch: while it might seem convenient for Pods to enable free communication among pods, it's like leaving your front door wide open in a not-so-great neighbourhood. Sure, it's all fun and games until someone you don't want barges in. And in Kubernetes land, that means inviting all sorts of security risks.


Figure1. Behavior without Network Policies

Thus, in a nutshell Kubernetes Network policies can help in isolating namespaces from one another in the sense that workloads belonging to "env1" to communicate with workloads in "env2". Another very common use of Network Policies is enforcing Zero Trust policies around workloads ensuring least privilege access & that a compromise in one of your workloads can't spread laterally to affect other workloads. Note that policies can govern East-West Traffic within Kubernetes but can also extend to govern the communication patterns with other non-Kubernetes workloads (IP Addresses).

It's essential to understand that while these policies result in isolation at the Dataplane level (e.g., Firewall), they do not imply any form of Virtual Routing and Forwarding (VRF) or Segmentation from a networking perspective.


Figure2. The Need for Kubernetes Network Policies

More on Network Policies Concepts


Figure3. Deeper Dive on Network Policies

CiliumNetwork Policy structure can be found below and it's basic sections are:

Selector needs to define which endpoint(s) the policy is subject to.
Direction of the Policy: Ingress/Egress
Labels are heavily used in Kubernetes & Network Policies as they allow:
- Decoupling from IP addresses especially recalling that Pods are ephemeral
- Being able to apply policies on a number of Pods at the same time. Below example for instance shows that all 3 pods share the same label (app=reviews).
- Any new upcoming Pods that are being instantiated can automatically be captured by the Network Policy.
- Examples in the above diagram are role:frontend, & role:db.
In Cilium L3 Policies, most use-cases are neatly outlined in Figure 3. I'd like to highlight L3 DNS-based policies, which simplify allowing egress access to specific domains from particular Pods or groups of Pods.

Let's break it down: Instead of manually resolving the IP addresses for, say, nginx.com, and then configuring access accordingly, L3 DNS-based policies take a smarter approach. When a Pod sends a DNS request for a domain like nginx.com, Cilium steps in. It relays the request to a DNS proxy, giving us visibility over the DNS request and response.

Here's where the magic happens: the response to the DNS request becomes an IP address that we control communication to. This means we can seamlessly manage and restrict access to desired domains without the hassle of manual IP management. It's like having your own traffic cop directing Pod communications in real-time, all thanks to Cilium's intelligent handling of DNS

On Cilium L7 Policies ,the DNS policy here is a bit different where it basically leverages a DNS Proxy to accept/reject DNS requests.
HTTP Policy within L7 policies, allows for example more granular control on incoming traffic by allowing it from a group of pods only when it targets a particular URI path.

k.eljamali@LLJXN1XJ9X Network Policies % kubectl get pods -l app=reviewsNAME                          READY   STATUS    RESTARTS   AGEreviews-v1-5b5d6494f4-dwvpn   1/1     Running   0          4d14hreviews-v2-5b667bcbf8-kjwhn   1/1     Running   0          4d14hreviews-v3-5b9bd44f4-qjfth    1/1     Running   0          4d14h

Figure4. Kubernetes Pods Label Example

          apiVersion: "cilium.io/v2"          kind: CiliumNetworkPolicy          metadata:            name: "productpage-deny-all"          spec:            endpointSelector:              matchLabels:                app: productpage            egress:            - {}

Figure5. Sample Cilium Network Policy

Cilium Policy Enforcement Mode: Default

Cilium offers different modes for Policy Enforcement: Default, Always, and Never. Let's focus on describing the behavior under the Default mode:

No Network Policies Attached: For endpoints without any Network Policies attached, Egress and Ingress traffic are allowed without restriction.
Network Policy Applied to Endpoint in Egress Direction:When a Network Policy is applied to an endpoint in the egress direction, only the specific egress traffic explicitly mentioned in the policy will be allowed. All other egress traffic will be denied.
An empty egress policy implies that no egress traffic is allowed, resulting in the denial of all egress traffic.
Ingress traffic remains unaffected by egress policies.

Stateful Network Policies

It's crucial to grasp the concept that Network Policies operate as stateful constructs. For instance, when considering permitting communication from a client Pod to an nginx Pod, we only need two policies:

Egress policy on the Client Pod to facilitate communication with the nginx Pod.Ingress policy on the nginx Pod to allow communication from the client Pod.Remarkably, explicit network policies for return traffic are unnecessary. This is because the return path of a flow, such as the response from the nginx Pod to the client Pod, is inherently part of the same outgoing flow permitted by the client to the nginx Pod. Therefore, no incoming policy is required on the client side for this return traffic, as it is already encompassed within the outgoing flow allowance.


Figure6. Stateful Network Policies

Scenario

To keep things straightforward, Ive opted for the BookInfo applicationa commonly featured example in Istios getting started guide. This application comprises four microservices: Product Page, Reviews, Details, and Ratings. While Reviews has multiple versions, well keep our focus on v1.

For a visual overview, check out the diagram below, sourced from Istios documentation. It vividly illustrates the key components and their interactions.More details can be found here.


Figure 7. BookInfo Application

It is worth noting that all of this setup is running locally on my laptop as a kind cluster. More details can be found here.


Figure 8. Lab Setup

Application Discovery

Hubble is a fully distributed networking and security observability platform. It is built on top of Cilium and eBPF to enable deep visibility into the communication and behavior of services as well as the networking infrastructure in a completely transparent manner. (Source: Cilium Documentation)

We will look at identities in the upcoming section but for now you can see:

Figure9. shows the traffic from the internal-client pod to productpage pod
Figure10. shows all the backends that the productpage actually communicates with reviews throughout its versions (v1,v2,v3).

Effective crafting of Network Policies hinges on attaining comprehensive visibility into the communication patterns among Pods and entities. This visibility enables a thorough understanding of how these components interact within the Kubernetes cluster, facilitating the creation of policies that accurately reflect and enforce security requirements.

k.eljamali@LLJXN1XJ9X Network Policies % hubble observe --to-identity 9889 --last 5 --from-identity 23217 Apr 27 22:05:46.867: default/internal-client:47838 (ID:23217) -> default/productpage-v1-675fc69cf-8zzxh:9080 (ID:9889) to-endpoint FORWARDED (TCP Flags: ACK, FIN)Apr 27 22:09:16.221: default/internal-client:42856 (ID:23217) -> default/productpage-v1-675fc69cf-8zzxh:9080 (ID:9889) to-endpoint FORWARDED (TCP Flags: SYN)Apr 27 22:09:16.221: default/internal-client:42856 (ID:23217) -> default/productpage-v1-675fc69cf-8zzxh:9080 (ID:9889) to-endpoint FORWARDED (TCP Flags: ACK)Apr 27 22:09:16.221: default/internal-client:42856 (ID:23217) -> default/productpage-v1-675fc69cf-8zzxh:9080 (ID:9889) to-endpoint FORWARDED (TCP Flags: ACK, PSH)Apr 27 22:09:21.344: default/internal-client:42856 (ID:23217) -> default/productpage-v1-675fc69cf-8zzxh:9080 (ID:9889) to-endpoint FORWARDED (TCP Flags: ACK, FIN)

Figure 9. Traffic from Client to Productpage

k.eljamali@LLJXN1XJ9X Network Policies % hubble observe --from-identity 9889 --last 5                      Apr 27 22:05:46.813: default/productpage-v1-675fc69cf-8zzxh:35336 (ID:9889) -> default/reviews-v2-5b667bcbf8-kjwhn:9080 (ID:10139) to-endpoint FORWARDED (TCP Flags: ACK, FIN)Apr 27 22:05:46.814: default/productpage-v1-675fc69cf-8zzxh:35336 (ID:9889) -> default/reviews-v2-5b667bcbf8-kjwhn:9080 (ID:10139) to-endpoint FORWARDED (TCP Flags: ACK)Apr 27 22:05:46.822: default/productpage-v1-675fc69cf-8zzxh:43297 (ID:9889) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) to-endpoint FORWARDED (UDP)Apr 27 22:05:46.859: default/productpage-v1-675fc69cf-8zzxh:43712 (ID:9889) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) to-endpoint FORWARDED (UDP)Apr 27 22:05:46.863: default/productpage-v1-675fc69cf-8zzxh:35352 (ID:9889) -> default/reviews-v1-5b5d6494f4-dwvpn:9080 (ID:55669) to-endpoint FORWARDED (TCP Flags: ACK)Apr 27 22:05:46.863: default/productpage-v1-675fc69cf-8zzxh:35352 (ID:9889) -> default/reviews-v1-5b5d6494f4-dwvpn:9080 (ID:55669) to-endpoint FORWARDED (TCP Flags: ACK, PSH)Apr 27 22:05:46.866: default/productpage-v1-675fc69cf-8zzxh:35352 (ID:9889) -> default/reviews-v1-5b5d6494f4-dwvpn:9080 (ID:55669) to-endpoint FORWARDED (TCP Flags: ACK, FIN)Apr 27 22:05:46.867: default/productpage-v1-675fc69cf-8zzxh:35352 (ID:9889) -> default/reviews-v1-5b5d6494f4-dwvpn:9080 (ID:55669) to-endpoint FORWARDED (TCP Flags: ACK)Apr 27 22:09:16.230: default/productpage-v1-675fc69cf-8zzxh:50168 (ID:9889) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) to-endpoint FORWARDED (UDP)Apr 27 22:09:21.260: default/productpage-v1-675fc69cf-8zzxh:42803 (ID:9889) -> kube-system/coredns-76f75df574-rw29s:53 (ID:17513) to-endpoint FORWARDED (UDP)Apr 27 22:09:21.276: default/productpage-v1-675fc69cf-8zzxh:33718 (ID:9889) -> default/reviews-v3-5b9bd44f4-qjfth:9080 (ID:1786) to-endpoint FORWARDED (TCP Flags: ACK, PSH)Apr 27 22:09:21.336: default/productpage-v1-675fc69cf-8zzxh:33718 (ID:9889) -> default/reviews-v3-5b9bd44f4-qjfth:9080 (ID:1786) to-endpoint FORWARDED (TCP Flags: ACK, FIN)Apr 27 22:09:21.336: default/productpage-v1-675fc69cf-8zzxh:33718 (ID:9889) -> default/reviews-v3-5b9bd44f4-qjfth:9080 (ID:1786) to-endpoint FORWARDED (TCP Flags: ACK)Apr 27 22:09:21.342: default/internal-client:42856 (ID:23217) <- default/productpage-v1-675fc69cf-8zzxh:9080 (ID:9889) to-endpoint FORWARDED (TCP Flags: ACK, PSH)Apr 27 22:09:21.343: default/internal-client:42856 (ID:23217) <- default/productpage-v1-675fc69cf-8zzxh:9080 (ID:9889) to-endpoint FORWARDED (TCP Flags: ACK, FIN)

Figure10. Traffic from Productpage to Reviews, Ratings & Details Pods

Security Identity

As discussed, Kubernetes & Network Policies are heavily dependent on labels. Basically, for every unique combination of labels that would target a set of Pods, Cilium would create unique security identities.

Thus, when traffic for instance needs to go from frontend to backend.

On the host having the web pod, Cilium figures out the security identity that maps to the source which is 81 in our example.
Cilium would ensure that the security identity information (81) is encoded into the packet it sends to the backend.
Thus, you can think about Network Policies as controlling access based on source and destination security identities.
Upon receiving the packet, we know that the source identity is 81 and the backend identity is 77 and need to check if there are policies that allow Identity 81


Figure11. Cilium Security Identities

This output shows all the existing pods and their existing security identities.

NAME                             SECURITY IDENTITY   ENDPOINT STATE   IPV4           IPV6details-v1-698d88b-89gnc         285                 ready            10.244.2.140   internal-client                  23217               ready            10.244.2.172   productpage-v1-675fc69cf-8zzxh   9889                ready            10.244.2.148   ratings-v1-6484c4d9bb-8nsbj      5778                ready            10.244.3.241   reviews-v1-5b5d6494f4-dwvpn      55669               ready            10.244.1.170   reviews-v2-5b667bcbf8-kjwhn      10139               ready            10.244.3.87    reviews-v3-5b9bd44f4-qjfth       1786                ready            10.244.2.191

Figure12. Endpoints & their corresponding identities

Egress Policy: Deny-All

This is an example of an empty egress network policy that is attached to the productpage pod. This would block all outgoing traffic from productpage to any other pod or entity.

apiVersion: "cilium.io/v2"kind: CiliumNetworkPolicymetadata:  name: "productpage-deny-all"spec:  endpointSelector:    matchLabels:      app: productpage  egress:  - {}

Figure13. Empty Egress Network Policy

Refer back to figure 12 where the pods and their respective identities are listed. Please note that Hubble can also use pod names directly with from-pod and to-pod.

Figure 14 shows that traffic continues to flow from the client to the productpage which is expected as the policy on the productpage is an egress only policy.

k.eljamali@LLJXN1XJ9X Network Policies % hubble observe --to-identity 9889 --last 5 --from-identity 23217Apr 27 22:17:28.254: default/internal-client:51160 (ID:23217) -> default/productpage-v1-675fc69cf-8zzxh:9080 (ID:9889) to-endpoint FORWARDED (TCP Flags: ACK)Apr 27 22:17:28.255: default/internal-client:51160 (ID:23217) -> default/productpage-v1-675fc69cf-8zzxh:9080 (ID:9889) to-endpoint FORWARDED (TCP Flags: ACK, FIN)Apr 27 22:17:28.280: default/internal-client:46532 (ID:23217) -> default/productpage-v1-675fc69cf-8zzxh:9080 (ID:9889) to-endpoint FORWARDED (TCP Flags: SYN)Apr 27 22:17:28.281: default/internal-client:46532 (ID:23217) -> default/productpage-v1-675fc69cf-8zzxh:9080 (ID:9889) to-endpoint FORWARDED (TCP Flags: ACK)Apr 27 22:17:28.281: default/internal-client:46532 (ID:23217) -> default/productpage-v1-675fc69cf-8zzxh:9080 (ID:9889) to-endpoint FORWARDED (TCP Flags: ACK, PSH)

Figure14. Using Hubble to observe traffic between internal-client & productpage

Figure 15, shows that traffic from productpage to coredns is blocked. The reason this happens is when productpage wants to communicate with any of its backend services (reviews, details..etc) it sends a DNS request to resolve the IP address of the service name (reviews for example). For more details refer to the Kubernetes Services blog post here.

hubble observe --from-identity 9889 --since 2mApr 27 22:17:28.285: default/productpage-v1-675fc69cf-8zzxh:40789 (ID:9889) <> kube-system/coredns-76f75df574-rw29s:53 (ID:17513) policy-verdict:none EGRESS DENIED (UDP)Apr 27 22:17:28.285: default/productpage-v1-675fc69cf-8zzxh:40789 (ID:9889) <> kube-system/coredns-76f75df574-rw29s:53 (ID:17513) Policy denied DROPPED (UDP)Apr 27 22:17:28.285: default/productpage-v1-675fc69cf-8zzxh:40789 (ID:9889) <> kube-system/coredns-76f75df574-rw29s:53 (ID:17513) policy-verdict:none EGRESS DENIED (UDP)Apr 27 22:17:28.285: default/productpage-v1-675fc69cf-8zzxh:40789 (ID:9889) <> kube-system/coredns-76f75df574-rw29s:53 (ID:17513) Policy denied DROPPED (UDP)Apr 27 22:17:33.290: default/productpage-v1-675fc69cf-8zzxh:40789 (ID:9889) <> kube-system/coredns-76f75df574-rw29s:53 (ID:17513) policy-verdict:none EGRESS DENIED (UDP)Apr 27 22:17:33.290: default/productpage-v1-675fc69cf-8zzxh:40789 (ID:9889) <> kube-system/coredns-76f75df574-rw29s:53 (ID:17513) Policy denied DROPPED (UDP)

Figure15. Traffic from productpage to coredns is blocked

Output from internal-client is actually inline with our findings whereby we get a response from productpage however it clearly mentions that reviews are inaccessible as a result of the egress policy.

    </div>    <div class="col-md-6">      <h4 class="text-center text-primary">Error fetching product reviews!</h4>      <p>Sorry, product reviews are currently unavailable for this book.</p>

Figure16. Output from Internal Client

Allow Traffic to DNS

We have concluded our earlier section, whereby traffic to the coredns pod was being blocked. This policy allows product page to get to coredns on port 53.

apiVersion: "cilium.io/v2"kind: CiliumNetworkPolicymetadata:  name: "allow-to-dns"spec:  endpointSelector:    matchLabels:      app: productpage  egress:    - toEndpoints:      - matchLabels:          "k8s:io.kubernetes.pod.namespace": kube-system          "k8s:k8s-app": kube-dns      toPorts:        - ports:           - port: "53"             protocol: ANY

Figure17. Allow Traffic from productpage to coredns

k.eljamali@LLJXN1XJ9X Network Policies % hubble observe --from-identity 9889 --last 1 minApr 28 00:36:17.388: default/productpage-v1-675fc69cf-8zzxh:49263 (ID:9889) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) to-endpoint FORWARDED (UDP)Apr 28 00:36:17.407: default/productpage-v1-675fc69cf-8zzxh:60156 (ID:9889) -> default/reviews-v1-5b5d6494f4-dwvpn:9080 (ID:55669) to-endpoint FORWARDED (TCP Flags: ACK)Apr 28 00:36:17.407: default/internal-client:40690 (ID:23217) <- default/productpage-v1-675fc69cf-8zzxh:9080 (ID:9889) to-endpoint FORWARDED (TCP Flags: ACK, FIN)

Figure18. Traffic Forwarded from productpage to coredns

The DNS rules being added here can be used as an L7 policy to allow DNS requests to particular domains (like an Allowlist) via a DNS proxy as shown in Figure 3. The other value that comes with DNS rules is that now we have added a DNS proxy into the picture, we can have more visibility on DNS traffic.

apiVersion: "cilium.io/v2"kind: CiliumNetworkPolicymetadata:  name: "allow-to-dns"spec:  endpointSelector:    matchLabels:      app: productpage  egress:    - toEndpoints:      - matchLabels:          "k8s:io.kubernetes.pod.namespace": kube-system          "k8s:k8s-app": kube-dns      toPorts:        - ports:           - port: "53"             protocol: ANY          rules:            dns:              - matchPattern: "*"

Figure19. Add DNS Rules to the NetworkPolicy

k.eljamali@LLJXN1XJ9X Network Policies % hubble observe   --protocol dns Apr 28 00:49:07.340: default/productpage-v1-675fc69cf-8zzxh:50114 (ID:9889) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-request proxy FORWARDED (DNS Query details.default.svc.cluster.local. AAAA)Apr 28 00:49:07.340: default/productpage-v1-675fc69cf-8zzxh:50114 (ID:9889) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-request proxy FORWARDED (DNS Query details.default.svc.cluster.local. A)Apr 28 00:49:07.342: default/productpage-v1-675fc69cf-8zzxh:50114 (ID:9889) <- kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-response proxy FORWARDED (DNS Answer "10.96.222.2" TTL: 30 (Proxy details.default.svc.cluster.local. A))Apr 28 00:49:07.342: default/productpage-v1-675fc69cf-8zzxh:50114 (ID:9889) <- kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-response proxy FORWARDED (DNS Answer  TTL: 4294967295 (Proxy details.default.svc.cluster.local. AAAA))Apr 28 00:49:07.345: default/productpage-v1-675fc69cf-8zzxh:35429 (ID:9889) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-request proxy FORWARDED (DNS Query reviews.default.svc.cluster.local. AAAA)Apr 28 00:49:07.345: default/productpage-v1-675fc69cf-8zzxh:35429 (ID:9889) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-request proxy FORWARDED (DNS Query reviews.default.svc.cluster.local. A)Apr 28 00:49:07.346: default/productpage-v1-675fc69cf-8zzxh:35429 (ID:9889) <- kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-response proxy FORWARDED (DNS Answer "10.96.179.42" TTL: 10 (Proxy reviews.default.svc.cluster.local. A))Apr 28 00:49:07.347: default/productpage-v1-675fc69cf-8zzxh:35429 (ID:9889) <- kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-response proxy FORWARDED (DNS Answer  TTL: 4294967295 (Proxy reviews.default.svc.cluster.local. AAAA))

Figure20. DNS Traffic Visibility

Allow Traffic to Backend Pods

Recall that at this stage, the DNS resolution works however we still need to add the relevant rules from productpage to reviews & details. Bear in mind that for other pods that are not selected as part of the policy all ingress/egress traffic continues to flow (e.g. Traffic between reviews & ratings)

apiVersion: "cilium.io/v2"kind: CiliumNetworkPolicymetadata:  name: "from-product-to-reviews-details"spec:  endpointSelector:    matchLabels:      app: productpage  egress:  - toEndpoints:    - matchLabels:        app: reviews  - toEndpoints:    - matchLabels:        app: details

Figure21. Allow from productpage to reviews & details

k.eljamali@LLJXN1XJ9X Network Policies % hubble observe --from-identity 9889 --last 4 minApr 28 00:38:03.715: default/productpage-v1-675fc69cf-8zzxh:37378 (ID:9889) -> default/reviews-v1-5b5d6494f4-dwvpn:9080 (ID:55669) to-endpoint FORWARDED (TCP Flags: ACK, PSH)Apr 28 00:38:03.724: default/productpage-v1-675fc69cf-8zzxh:37378 (ID:9889) -> default/reviews-v1-5b5d6494f4-dwvpn:9080 (ID:55669) to-endpoint FORWARDED (TCP Flags: ACK, FIN)Apr 28 00:38:03.725: default/productpage-v1-675fc69cf-8zzxh:37378 (ID:9889) -> default/reviews-v1-5b5d6494f4-dwvpn:9080 (ID:55669) to-endpoint FORWARDED (TCP Flags: ACK)Apr 28 00:38:03.740: default/productpage-v1-675fc69cf-8zzxh:58725 (ID:9889) -> kube-system/coredns-76f75df574-rw29s:53 (ID:17513) to-endpoint FORWARDED (UDP)Apr 28 00:38:08.810: default/productpage-v1-675fc69cf-8zzxh:37394 (ID:9889) -> default/reviews-v2-5b667bcbf8-kjwhn:9080 (ID:10139) to-endpoint FORWARDED (TCP Flags: ACK, FIN)Apr 28 00:38:08.810: default/productpage-v1-675fc69cf-8zzxh:37394 (ID:9889) -> default/reviews-v2-5b667bcbf8-kjwhn:9080 (ID:10139) to-endpoint FORWARDED (TCP Flags: ACK)Apr 28 00:38:08.867: default/productpage-v1-675fc69cf-8zzxh:34917 (ID:9889) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) to-endpoint FORWARDED (UDP)Apr 28 00:38:08.879: default/productpage-v1-675fc69cf-8zzxh:51600 (ID:9889) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) to-endpoint FORWARDED (UDP)Apr 28 00:38:08.911: default/productpage-v1-675fc69cf-8zzxh:37402 (ID:9889) -> default/reviews-v3-5b9bd44f4-qjfth:9080 (ID:1786) to-endpoint FORWARDED (TCP Flags: ACK, FIN)Apr 28 00:38:08.911: default/productpage-v1-675fc69cf-8zzxh:37402 (ID:9889) -> default/reviews-v3-5b9bd44f4-qjfth:9080 (ID:1786) to-endpoint FORWARDED (TCP Flags: ACK)

Figure22. Hubble Visbility for traffic from product page & reviews versions

Internet Egress Policy

This is an Internet Egress use-case where the objective is to limit the domains that the internal-client pod can connect to on the Internet.

Notice you can still see the same DNS policy, however we are adding the toFQNDs construct in this policy. This works as follows:

Traffic from internal-client to coredns is allowed.
When the internal-client wants to connect to the public URL "ifconfig.me", it first needs to resolve its IP Address.
Since, we already have visibility to DNS traffic with DNS Proxy, we can see the DNS response containing the IP Address of "ifconfig.me".
toFQDN is basically a representation of that IP Address i.e. as if we are putting a policy to the IP address that is a result of DNS resolution.

The reason this approach is interesting is that we don't need to deal with changing addresses & we can just rely on DNS names.

apiVersion: "cilium.io/v2"kind: CiliumNetworkPolicymetadata:  name: "from-internal-client-to-fqdn"spec:  endpointSelector:    matchLabels:      app: internal-client  egress:    - toEndpoints:      - matchLabels:          "k8s:io.kubernetes.pod.namespace": kube-system          "k8s:k8s-app": kube-dns      toPorts:        - ports:           - port: "53"             protocol: ANY          rules:            dns:              - matchPattern: "*"    - toFQDNs:        - matchName: "ifconfig.me"

Figure23. Allow Traffic to Public FQDN ifconfig.me

k.eljamali@LLJXN1XJ9X Network Policies % hubble observe --pod internal-client --protocol dns Apr 28 01:03:39.378: default/internal-client:60700 (ID:23217) -> kube-system/coredns-76f75df574-rw29s:53 (ID:17513) dns-request proxy FORWARDED (DNS Query ifconfig.me. A)Apr 28 01:03:39.378: default/internal-client:60700 (ID:23217) -> kube-system/coredns-76f75df574-rw29s:53 (ID:17513) dns-request proxy FORWARDED (DNS Query ifconfig.me. AAAA)Apr 28 01:03:39.423: default/internal-client:60700 (ID:23217) <- kube-system/coredns-76f75df574-rw29s:53 (ID:17513) dns-response proxy FORWARDED (DNS Answer "2600:1901:0:bbc3::" TTL: 30 (Proxy ifconfig.me. AAAA))Apr 28 01:03:39.424: default/internal-client:60700 (ID:23217) <- kube-system/coredns-76f75df574-rw29s:53 (ID:17513) dns-response proxy FORWARDED (DNS Answer "34.117.118.44" TTL: 30 (Proxy ifconfig.me. A))Apr 28 01:04:16.604: default/internal-client:57590 (ID:23217) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-request proxy FORWARDED (DNS Query cnn.com.default.svc.cluster.local. AAAA)Apr 28 01:04:16.604: default/internal-client:57590 (ID:23217) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-request proxy FORWARDED (DNS Query cnn.com.default.svc.cluster.local. A)Apr 28 01:04:16.607: default/internal-client:57590 (ID:23217) <- kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-response proxy FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy cnn.com.default.svc.cluster.local. AAAA))Apr 28 01:04:16.607: default/internal-client:57590 (ID:23217) <- kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-response proxy FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy cnn.com.default.svc.cluster.local. A))Apr 28 01:04:16.610: default/internal-client:57590 (ID:23217) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-request proxy FORWARDED (DNS Query cnn.com.svc.cluster.local. A)Apr 28 01:04:16.610: default/internal-client:57590 (ID:23217) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-request proxy FORWARDED (DNS Query cnn.com.svc.cluster.local. AAAA)Apr 28 01:04:16.611: default/internal-client:57590 (ID:23217) <- kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-response proxy FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy cnn.com.svc.cluster.local. A))Apr 28 01:04:16.611: default/internal-client:57590 (ID:23217) <- kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-response proxy FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy cnn.com.svc.cluster.local. AAAA))Apr 28 01:04:16.611: default/internal-client:57590 (ID:23217) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-request proxy FORWARDED (DNS Query cnn.com.cluster.local. AAAA)Apr 28 01:04:16.612: default/internal-client:57590 (ID:23217) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-request proxy FORWARDED (DNS Query cnn.com.cluster.local. A)Apr 28 01:04:16.612: default/internal-client:57590 (ID:23217) <- kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-response proxy FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy cnn.com.cluster.local. AAAA))Apr 28 01:04:16.612: default/internal-client:57590 (ID:23217) <- kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-response proxy FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy cnn.com.cluster.local. A))Apr 28 01:04:16.612: default/internal-client:57590 (ID:23217) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-request proxy FORWARDED (DNS Query cnn.com. AAAA)Apr 28 01:04:16.612: default/internal-client:57590 (ID:23217) -> kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-request proxy FORWARDED (DNS Query cnn.com. A)Apr 28 01:04:16.652: default/internal-client:57590 (ID:23217) <- kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-response proxy FORWARDED (DNS Answer "151.101.3.5,151.101.131.5,151.101.67.5,151.101.195.5" TTL: 30 (Proxy cnn.com. A))Apr 28 01:04:16.656: default/internal-client:57590 (ID:23217) <- kube-system/coredns-76f75df574-sgbcv:53 (ID:17513) dns-response proxy FORWARDED (DNS Answer "2a04:4e42:c00::773,2a04:4e42:a00::773,2a04:4e42::773,2a04:4e42:800::773,2a04:4e42:600::773,2a04:4e42:e00::773,2a04:4e42:200::773,2a04:4e42:400::773" TTL: 30 (Proxy cnn.com. AAAA))

Figure24. Visibility to DNS Request & Responses for Public Domains

Here you can see traffic to "ifconfig.me" forwarded whereas traffic for other domains being denied.

Apr 28 01:04:16.656: default/internal-client:54006 (ID:23217) <> cnn.com:80 (world) policy-verdict:none EGRESS DENIED (TCP Flags: SYN)Apr 28 01:04:16.656: default/internal-client:54006 (ID:23217) <> cnn.com:80 (world) Policy denied DROPPED (TCP Flags: SYN)Apr 28 01:04:17.699: default/internal-client:54006 (ID:23217) <> cnn.com:80 (world) policy-verdict:none EGRESS DENIED (TCP Flags: SYN)Apr 28 01:04:17.699: default/internal-client:54006 (ID:23217) <> cnn.com:80 (world) Policy denied DROPPED (TCP Flags: SYN)Apr 28 01:05:20.563: default/internal-client:54006 (ID:23217) <> 151.101.3.5:80 (world) policy-verdict:none EGRESS DENIED (TCP Flags: SYN)Apr 28 01:05:20.563: default/internal-client:54006 (ID:23217) <> 151.101.3.5:80 (world) Policy denied DROPPED (TCP Flags: SYN)Apr 28 01:08:27.832: default/internal-client:54862 (ID:23217) -> ifconfig.me:80 (ID:16777219) policy-verdict:L3-Only EGRESS ALLOWED (TCP Flags: SYN)Apr 28 01:08:27.833: default/internal-client:54862 (ID:23217) -> ifconfig.me:80 (ID:16777219) to-stack FORWARDED (TCP Flags: SYN)Apr 28 01:08:27.854: default/internal-client:54862 (ID:23217) -> ifconfig.me:80 (ID:16777219) to-stack FORWARDED (TCP Flags: ACK)Apr 28 01:08:27.854: default/internal-client:54862 (ID:23217) -> ifconfig.me:80 (ID:16777219) to-stack FORWARDED (TCP Flags: ACK, PSH)Apr 28 01:08:27.894: default/internal-client:54862 (ID:23217) -> ifconfig.me:80 (ID:16777219) to-stack FORWARDED (TCP Flags: ACK, FIN)Apr 28 01:08:27.894: default/internal-client:54862 (ID:23217) -> ifconfig.me:80 (ID:16777219) to-stack FORWARDED (TCP Flags: ACK)

Figure25. Allow Traffic to Public FQDN ifconfig.me

Troubleshooting

In this section, I wanted to go a bit deeper on a flow productpage to reviews-v1 pod. I have added the figure again for convenience. You can clearly see that productpage and reviews-v1 are on different nodes.

In addition, in this setup VXLAN tunnels to encapsulate traffic between the nodes as per the output of Figure 27.


Figure26. Revisit Pod Placement

k.eljamali@LLJXN1XJ9X Network Policies % cilium config view | grep tunnelrouting-mode                                      tunneltunnel-protocol                                   vxlan

Figure27. Tunnel Mode with VXLAN

It is mind boggling to see all this information within Hubble on a particular flow.

Figure 28 shows all the information about source & destination in terms of labels & identities and we can clearly see the traffic being forwarded on the overlay (VXLAN Tunnel). Note that the verdict clearly shows that the traffic is forwarded.

{  "flow": {    "time": "2024-04-27T21:36:04.373296596Z",    "uuid": "3ce41f60-b203-4e43-ab34-c89f631f7cd2",    "verdict": "FORWARDED",    "ethernet": {      "source": "9e:f7:72:78:39:c7",      "destination": "d6:55:a3:91:69:37"    },    "IP": {      "source": "10.244.2.148",      "destination": "10.244.1.170",      "ipVersion": "IPv4"    },    "l4": {      "TCP": {        "source_port": 47812,        "destination_port": 9080,        "flags": {          "ACK": true        }      }    },    "source": {      "ID": 1881,      "identity": 9889,      "namespace": "default",      "labels": [        "k8s:app=productpage",        "k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default",        "k8s:io.cilium.k8s.policy.cluster=kind-kind",        "k8s:io.cilium.k8s.policy.serviceaccount=bookinfo-productpage",        "k8s:io.kubernetes.pod.namespace=default",        "k8s:version=v1"      ],      "pod_name": "productpage-v1-675fc69cf-8zzxh",      "workloads": [        {          "name": "productpage-v1",          "kind": "Deployment"        }      ]    },    "destination": {      "identity": 55669,      "namespace": "default",      "labels": [        "k8s:app=reviews",        "k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default",        "k8s:io.cilium.k8s.policy.cluster=kind-kind",        "k8s:io.cilium.k8s.policy.serviceaccount=bookinfo-reviews",        "k8s:io.kubernetes.pod.namespace=default",        "k8s:version=v1"      ],      "pod_name": "reviews-v1-5b5d6494f4-dwvpn"    },    "Type": "L3_L4",    "node_name": "kind-kind/kind-worker3",    "event_type": {      "type": 4,      "sub_type": 4    },    "traffic_direction": "EGRESS",    "trace_observation_point": "TO_OVERLAY",    "is_reply": false,    " interface": {      "index": 6,      "name": "cilium_vxlan"    },    "Summary": "TCP Flags: ACK"  },  "node_name": "kind-kind/kind-worker3",  "time": "2024-04-27T21:36:04.373296596Z"}

Figure28. Egress Traffic from Product Forwarded over VXLAN Tunnel

Figure29 shows the incoming traffic from kind-worker1's perspective, where the traffic gets forwarded to the interface connecting reviews-v1 pod. Note that the verdict clearly shows that the traffic is forwarded.

{  "flow": {    "time": "2024-04-27T21:36:04.373343054Z",    "uuid": "bf9df4cf-44b2-479a-bd41-9d232d5aacf6",    "verdict": "FORWARDED",    "ethernet": {      "source": "72:e6:23:94:38:bb",      "destination": "02:9e:93:c9:74:57"    },    "IP": {      "source": "10.244.2.148",      "destination": "10.244.1.170",      "ipVersion": "IPv4"    },    "l4": {      "TCP": {        "source_port": 47812,        "destination_port": 9080,        "flags": {          "ACK": true        }      }    },    "source": {      "identity": 9889,      "namespace": "default",      "labels": [        "k8s:app=productpage",        "k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default",        "k8s:io.cilium.k8s.policy.cluster=kind-kind",        "k8s:io.cilium.k8s.policy.serviceaccount=bookinfo-productpage",        "k8s:io.kubernetes.pod.namespace=default",        "k8s:version=v1"      ],      "pod_name": "productpage-v1-675fc69cf-8zzxh"    },    "destination": {      "ID": 2039,      "identity": 55669,      "namespace": "default",      "labels": [        "k8s:app=reviews",        "k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default",        "k8s:io.cilium.k8s.policy.cluster=kind-kind",        "k8s:io.cilium.k8s.policy.serviceaccount=bookinfo-reviews",        "k8s:io.kubernetes.pod.namespace=default",        "k8s:version=v1"      ],      "pod_name": "reviews-v1-5b5d6494f4-dwvpn",      "workloads": [        {          "name": "reviews-v1",          "kind": "Deployment"        }      ]    },    "Type": "L3_L4",    "node_name": "kind-kind/kind-worker",    "event_type": {      "type": 4    },    "traffic_direction": "INGRESS",    "trace_observation_point": "TO_ENDPOINT",    "is_reply": false,    "interface": {      "index": 15,      "name": "lxc240530c746ff"    },    "Summary": "TCP Flags: ACK"  },  "node_name": "kind-kind/kind-worker",  "time": "2024-04-27T21:36:04.373343054Z"}

Figure29. Egress Traffic from Reviews-v1 with Policy Enforcement

]]>

PODCAST: Kubernetes Networking Essentials

Karim El Jamali — Thu, 04 Apr 2024 04:54:27 GMT

I recently was invited by Packet Pushers to chat on the topic of Kubernetes Networking Essentials as part of their Heavy Networking Podcast.

You can listen to the talk here.

]]>

Introduction to AWS VPC Lattice

Karim El Jamali — Mon, 26 Feb 2024 02:45:53 GMT

From Networking to Service Networking

Well, we have always talked about Networking in the context of L2, and L3 Protocols, SDN Fabrics and as workloads moved to the Cloud we started seeing Cloud Networking which varies between Public Clouds. So what is this service networking?

I think the best way to describe service networking is networking for application owners or developers. It is an abstraction of all the intricacies of the L3/L4 networking with primal focus on the application or service you are providing. Another reason Service Networking became a thing is the advent of micro-services. Historically in a monolith all the components run as services on the same hardware or virtual machine making the communicating between components internal to the node. In a micro-services environment each component can live on a different node but we still need to visualize, monitor, optimize and secure the component to component communication.

Now that we have introduced service networking, some of the advantages that service networking brings:

Application Centricity in terms of monitoring and deep visibility
Provides Improved resiliency with Advanced Load Balancing techniques (Blue/Green deployments)
Zero Trust with Service to Service or client to Service Authentication & Authorization
Some solutions provides additional capabilities to inject failures & delays providing better robustness and resiliency of your services and applications

Now that we have introduced service networking, we will move on to focus on AWS VPC Lattice.

VPC Lattice Constructs

VPC Lattice is AWS's answer to service networking. It is made up of 3 basic components:

Service: This is a component or a micro-service within your uber application. Some examples could be: Checkout service, location service, authentication service..etc You can think about a service in Lattice similar to a Load Balancer VIP that abstracts and exposes a number of workloads which could be VMs, K8 Pods..etc A service is composed of 3 different elements:
- Listener: Similar to a LB, the listener defines the protocol, port & hostname(s) the service is listening to.
- Rules: The idea here is when traffic hits the listener you can have host based or path based rules by which you decide where to send this traffic. For example:
  - Path based:
    - acme.com/prod: Routes users to the production stable page
    - acme.com/latest: Routes users to the latest update that is under testing
  - Host based: Ability to publish multiple applications
    - app1.acme.com
    - app2.acme.com

You can also have a combination of Host and Path based routing.

Target Group: Similar to an ALB or NLB Target Group, this references the backend workloads where traffic needs to be forwarded when matching a particular rule.

Service Network: Services can be placed in one or more service networks. On another hand, VPCs (containing clients) to those services can also be associated to a single Service Network. Thus, a service network marries the services that are available within a service network and VPCs that need access to those services.

This figure shows the different target types supported by a VPC Lattice service. The interesting part here is that VPC Lattice works across all compute types (Virtual Machines, Kubernetes & Serverless)

This is an example of a service where for a particular rule (path or host based) we send 90% of the traffic towards the production target group (nginx-production-tag) and 10% of the traffic towards the new release (nginx-new-tg).

The association of services and VPCs to a service network provides high level security and scoping (similar to a VRF in networking). In the figure below, based on Service Network A, the only 3 services that can be accessed are S1, S2 and S3 and only VPC-1, VPC-2 and VPC-3 can access them. This means based on the example that VPC-1 can't access service S4 and VPC-4 can't access services S1,S2, or S3.

The VPC Lattice Advantage

I will use this section to explain the architecture of VPC Lattice and highlight its unique points.

Parallel Network: One of the major benefits of VPC Lattice is that it runs as a parallel network under the hood i.e. there isn't any change required for the existing network. You can see the figure below where there is a TGW connecting two VPCs and there is another path that goes through VPC Lattice. Another advantage of VPC Lattice is that it connects services across VPCs and Accounts without requiring any underlying networking changes via this parallel network.
VPC Lattice leverages DNS: So now that we have the two different paths, the decision on which path to take depends on DNS. When a service is created within VPC lattice, it gets allocated a DNS Name or you can specify your own name for that matter. The resolution of the DNS name maps to the Proxy represented by the Lattice icon on the top path. Any other traffic or DNS resolution continues to run in the same manner. It is worth noting that VPC lattice or these proxies operate within the link local range of 169.254.171.0/24 so that they don't interfere with your existing address space.
Solves overlapping address Problem: VPC Lattice by default will SNAT to the "Proxy IP" within the 169.254.171.0/24 range when sending the traffic to the service thus the service will always see the request coming from that range.

VPC Lattice Security

There are many aspects of VPC Lattice Security that can be highlighted:

Service to Service vs Network Connectivity: When you connect two VPCs (A&B) via TGW or VPC peering for instance, all workloads in VPC-A can communicate with VPC-B unless you are using some form of firewalling (Security Groups, Network Firewall..etc).VPC lattice improves on this model by narrowing down the list of services that can communicate within the service network as well as the list of VPCs that need to communicate with those limited set of services.
Security Groups: It is interesting that VPC Lattice respects the security groups of the ENIs across the path from the client to the server (member of the service target group). In addition, when going to VPC Lattice there is also an associated Security Group that can also be used to enforce which workloads within the VPC can access the services within the service network. In reality, I couldn't see the Lattice ENI but it just helps to put a mental model around it.

Authentication & Authorization: The Auth type (Authentication and Authorization) in VPC Lattice can be set to None or AWS IAM. None basically doesn't take into account authenticating and authorizing the client. The idea with AWS IAM is that when a client makes a request to a service regardless if the client is another service or a non service, the client needs to have an identity. VPC Lattice leverages Sigv4 which is the process to add authentication information to requests and is heavily leveraged in AWS for instance when you send a request to AWS services (e.g. S3), before the request is fulfilled AWS has to figure out the user identity, check the Identity Policy attached to the user and the resource policy on the S3 bucket and based on all of this either accept or reject the request.
VPC Lattice follows the same logic where it needs to identify the identity of a client, if it has the identity based policies to Invoke Lattice, in addition to the resource based policies which in Lattice are Service Network Policy & Service Policy. The Service Network Policy gets associated with the Service Network affecting the level of acceptable access to all services within the service network and is generally a coarse grained Authorization policy where for instance you can allow traffic from particular VPCs or Organization ID and that all requests have to be authenticated. The Service policy is associated with a particular service and provides fine grained least privilege access to the service.
- This is not the easiest part of VPC Lattice due to:
  - IAM Dependency: IAM is not the the easiest AWS service to wrap your head around at least for me.
  - Application Changes: The client or client application will need to be modified to send a Sigv4 request for Lattice to allow or deny the request.

Closing Thoughts

In this blog post, I tried to introduce the need for VPC Lattice, its constructs, as well as its architecture and benefits. We ended this blog post by touching upon VPC Lattice Security.

I will be working on a part 2 on VPC Lattice where I will run a small demo, discuss Kubernetes Integration and external access to VPC Lattice.

]]>

Kubernetes Gateway API

Karim El Jamali — Mon, 12 Feb 2024 19:15:44 GMT

So, we ended our previous post talking about some of the downsides of the Load Balancer service in terms of security, manageability and increased costs. So this weekend, I set myself up to understand the Kubernetes Gateway API (not an API Gateway) which is a consolidated ingress solution to get traffic to your Kubernetes Clusters.

It's important to note the terminology here. Traditionally, Kubernetes users have been familiar with Ingress, the widely-used method for creating a single entry point to expose services within a cluster, primarily targeting HTTP/HTTPs workloads. However, there's a new player in town: the Gateway API.

While Ingress acts like a monolith, requiring modifications to a single object as more applications are exposed, the Gateway API takes a more versatile approach. It covers a broader spectrum of protocols, including HTTP, HTTPS, and gRPC. What sets it apart is not just the protocol support but also the improved role-based API, providing a more flexible and sophisticated means of managing ingress.

Role Based API

ACME Corp has different roles that help manage their Kubernetes infrastructure and deploy their required applications and services on top.

Choosing the Provider: John handles the decision on which Infrastructure Providers to bring in and the high level settings of that provider. For instance, John favors the NGINX solution. For that matter John has to define the GatewayClass object which is a cluster scoped resource tied to the provider (NGINX). At a high level, this is John saying "We will use NGINX as the platform of choice to handle Ingress Traffic." There might be cases for various reasons where multiple GatewayClasses (Infrastructure Providers) can be used simultaneously.
Configuring the Proxy: Karim handles the NGINX proxy configuration via the Gateway Object. It is important to think about the proxy or Load Balancer Instance as a shared resource across multiple teams, thus some settings will be common and defined under the Gateway Object whereas others will be configured under Routes (HTTPRoutes, TLSRoutes). Options defined under the Gateway are:
- Specifying the class of Gateway it belongs to (GatewayClass specified by John)
- Specifying the hostname (acme.com), port, and protocol the proxy is listening to as well as TLS parameters.
- Allowing particular routes to attach to the Load Balancer.
Route Configuration: Chris has a particular ticketing Application that he intends to publish. He would like to use the FQDN ticket.acme.com and as soon as this URI is seen, he would like this routed to the ClusterIP service within Kubernetes which in turn will target his application workloads. A deeper explanation on ClusterIP and how it works can be found here.
Another advantage that the RoleBased API brings into the picture is a better RBAC implementation where each of those users or teams can gain access to the needed objects & thus sets better governance on publishing applications externally.
The final advantage of Gateway API we will touch upon is portability i.e. the ability to switch implementation from one vendor to another without requiring change. Ingress on the other hand was heavily dependent on vendor specific annotations which made it harder to switch from one implementation to another.

Source: https://gateway-api.sigs.k8s.io/

But how does it Work?

In this section, we will use NGINX's implementation of Gateway API referenced as NGINX Gateway Fabric as an example but the logic holds true mostly for other vendor implementations. There are a few components that make this work:

Custom Resource Definitions (CRDs): Kubernetes by default has a set of out of the box objects but can be extended with the use of Custom Resource Definitions (CRDs). In a nutshell, Gateway API is delivered as a set of CRDs extending the capability of Kubernetes to be able to manage the end-to-end lifecycle of third party objects which in our case are referencing the 3rd party NGINX Load Balancers.
We need to install the vendor implementation i.e. the NGINX Gateway Fabric Pod and all its associated objects (Deployments, ReplicaSets, Service Account, ClusterRoleBindings..etc. A key piece to understand is that the NGINX Gateway Fabric Pod (NGF Pod) consists of two different containers: NGF and NGINX. You can think of NGF as a component that listens to API Server configuration changes pertaining to the 3 main objects within GatewayAPI (GatewayClass, Gateway, Routes here encompasses HTTPRoutes/TLSRoutes) and then programs the NGINX container accordingly. Thus, NGF acts as a translator between the API Server and the native NGINX container configuration. For more details on installation, you can refer to NGINX Gateway Fabric docs here.

Scenarios

Simple Routing Configuration

Traffic Flow

This is the simplest of all scenarios:

Client resolves the DNS name of the NLB to an IP address
Traffic from the client goes towards the NLB's Public address to be DNATTed by the IGW.
An important consideration to mention here is that this NLB is part of a Load Balancer service in AWS that instantiated this NLB and we use to provide Public Access to the Nginx Gateway Fabric Pod. More details on the NLB service can be found here.
NGINX is configured via the HTTP Route to send all traffic destined to any path to the nginx service backend which in turn sends it to the NGINX pod. It is worth noting that the nginx service & pod are running NGINX web servers and could be any other workload for that purpose.

Configuration Snippets

It is important to understand that the route object references the Gateway and the Gateway references the GatewayClass. You can find the comments within each of the outputs.

apiVersion: v1items:- apiVersion: gateway.networking.k8s.io/v1  kind: GatewayClass  metadata:    labels:      app.kubernetes.io/instance: nginx-gateway       app.kubernetes.io/name: nginx-gateway      app.kubernetes.io/version: 1.1.0    name: nginx  spec:    controllerName: gateway.nginx.org/nginx-gateway-controller #this is a fixed domain set by the vendor and the path can vary specifically if you want to run multiple implementations

apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata:  name: mygateway  namespace: defaultspec:  gatewayClassName: nginx #tying the Gateway to the Class basically stating this is an NGINX Gateway. Similar to IngressClass  listeners:  - allowedRoutes:   #By default we are only allowing Routes to use this GW if they come from the same namespace      namespaces:    #which in our case is the default namespace object for the gateway        from: Same    name: http     #Listening on TCP port 80 for HTTP. Can be locked down to particular domains also.    port: 80    protocol: HTTP  #Can be used to filter TLS Routes

apiVersion: gateway.networking.k8s.io/v1kind: HTTPRoutemetadata:  name: nginx-default  namespace: defaultspec:  parentRefs: #Tying the Route to the Gateway  - group: gateway.networking.k8s.io    kind: Gateway    name: mygateway  rules:  - backendRefs: #Backend Service to send the traffic to    - group: ""      kind: Service      name: nginx      port: 80      weight: 1    matches: #Any traffic     - path:        type: PathPrefix        value: /

In a nutshell the objects tie as follows:

Route (HTTPRoute/TLSRoute) ties to the Gateway
The Gateway has an allowedRoutes which is like a filter on which routes to accept which by default allows routes within the same namespace as the Gateway Object but can be configured to allow routes from other namespaces. The Gateway on the other hand ties to the GatewayClass. On another note, Gateways leverage listeners as another method of filtering Routes i.e. if a gateway for instance listens for HTTP traffic, any TLS route will not be attached to the gateway. In addition, if the Gateway is listening to a particular hostname (*.acme.com) it will automatically ignore any routes that are not part of the ACME domain. The reason these filtering capabilities are important is because they set some regulations/boundaries on publishing applications that are agreed upon between the developers and infrastructure owners.
The GatewayClass ties to the Controller Name. You might want to make variations there when if you have a case where you need dual implementations of the NGINX Gateway Fabric for instance where one pod handles for example production workloads and the other handles development workloads.

Leveraging Paths

Traffic Flow

Following the same logic from the simple routing configuration but just noting the differences. NGINX is configured via the HTTP Route as follows:

When traffic matches /blue send traffic to the blue nginx deployment
When traffic matches /green send traffic to the green nginx deployment
Again this example can become more advanced by listening to Host headers within the HTTP and then within these host headers we can use paths where the objective is to publish many applications behind the NGINX Proxy for an enterprise deployment. For example:
- App1: app1.acme.com and within you can have app1.acme.com/internal app1.acme.com/customer
- App2: app2.acme.com and within you can have app2.acme.com/internal app2.acme.com/customer

Configuration Snippets

The difference in this scenario is that the HTTPRoute objects don't reside in the same namespace as the Gateway object thus we need to configure the gateway to allow routes in the blue and green namespaces to hook to it. This is done by the allowedRoutes block under the Gateway Object referencing a particular label for the namespace which in our example is "access-gateway:true" and ensuring that both namespaces blue and green have that label on. This step will ensure that the Gateway object residing in the default namespace will accept HTTPRoutes in the blue and green namespaces.

apiVersion: v1items:- apiVersion: gateway.networking.k8s.io/v1  kind: Gateway  metadata:    name: mygateway    namespace: default #gateway object resides in the default namespace  spec:    gatewayClassName: nginx    listeners:    - allowedRoutes:    #In this scenario, the two services we expose         namespaces:     #live in a different namespace thus we need to allow the HTTP routes          from: Selector #in those namespaces to hook to the Gateway else they will be floating          selector:      #this is done via labels on those namespaces that are accepted by the gateway object            matchLabels:              access-gateway: "true"      name: http      port: 80      protocol: HTTP

NAME              STATUS   AGE     LABELSblue              Active   2d1h    access-gateway=true,kubernetes.io/metadata.name=bluegreen             Active   2d1h    access-gateway=true,kubernetes.io/metadata.name=green

In this case the route is a bit different as we need to differentiate /blue and /green paths. We are not showing the green route for brevity.

apiVersion: v1items:- apiVersion: gateway.networking.k8s.io/v1  kind: HTTPRoute  metadata:    name: blue-nginx    namespace: blue  spec:    parentRefs:    - group: gateway.networking.k8s.io      kind: Gateway      name: mygateway      namespace: default    rules:    - backendRefs:      - group: ""        kind: Service        name: blue-nginx-deployment        port: 80        weight: 1      matches:      - path:          type: PathPrefix          value: /blue

In order to differentiate the NGINX web server deployments, this is a simple configmap that can be attached to the NGINX deployment so that NGINX in different namespaces will be easily distinguishable in your browser.

apiVersion: v1data:  nginx.conf: |    server {    listen 80;    location / {        default_type text/html;        set $nginx_ip $server_addr;        set $client_ip $remote_addr;        return 200 '<html>                      <head>                        <style>                          body {                            background-color: blue;                            color: white;                            font-size: 72px;                            display: flex;                            align-items: center;                            justify-content: center;                            height: 100vh;                            margin: 0;                          }                        </style>                      </head>                      <body>                        <div style="text-align: center;">                          <p>Server Name: Blue NGINX</p>                          <p>Server IP Address: $nginx_ip</p>                          <p>Client IP Address: $client_ip</p>                        </div>                      </body>                    </html>';                }        }kind: ConfigMapmetadata:  name: blue-nginx-config  namespace: blue

Dual Ingress Implementation

In this setup, we extend the model by adding another implementation of the GatewayAPI from Kong where this time we have two ingress points to our Kubernetes Cluster one with the NGINX Gateway Fabric and the other from Kong and we publish some applications via NGINX and the others via Kong. The NGINX setup is left intact in this scenario.

Traffic Flow

For the echo service it will be accessible via Kong which in turn will have its own NLB for external access from outside the Kubernetes Cluster.

Configuration Snippets

We will need to create the GatewayClass and Gateway for Kong similar to what we have done for the NGINX Fabric Controller. Note that the HTTPRoute has a reference to the Kong Gateway.

apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata:  name: kongspec:  controllerName: konghq.com/kic-gateway-controller #this is again a domain name specified by the vendor

apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata:  name: kong  namespace: defaultspec:  gatewayClassName: kong           #by default we only allow routes from the same namespace (default)  listeners:  - allowedRoutes:      namespaces:        from: Same    name: proxy    port: 80    protocol: HTTP

apiVersion: gateway.networking.k8s.io/v1kind: HTTPRoutemetadata:  annotations:    konghq.com/strip-path: "true"  name: echo  namespace: defaultspec:  parentRefs: #referencing the Kong gateway  - group: gateway.networking.k8s.io    kind: Gateway    name: kong  rules:  - backendRefs:    - group: ""      kind: Service      name: echo      port: 1027 #this is the port the echo service is running on      weight: 1    matches:    - path:        type: PathPrefix        value: /echo

References:

NGINX Gateway Fabric documentation: https://docs.nginx.com/nginx-gateway-fabric/

Kong Documentation: https://docs.konghq.com/kubernetes-ingress-controller/3.0.x/gateway-api/

]]>

Kubernetes Services: NodePort & LoadBalancer

Karim El Jamali — Sat, 27 Jan 2024 02:47:42 GMT

Introduction

In the last blog post found here, we focused on the inner workings of Kubernetes ClusterIP service. Kubernetes ClusterIP exposes pods within a Kubernetes Cluster. In this blog post, we focus on exposing Pods to clients and services outside the Kubernetes Cluster focusing on two service types Kubernetes NodePort & Kubernetes Load Balancer services.

It is worth noting that the scenarios we will discuss in this post have been built on AWS however the same concepts & logic should follow for other environments.

Kubernetes NodePort

The simplest way to expose a service externally is by using the NodePort service whereby all the nodes listen by default on a TCP port in the range 30000-32767.

The flow works as follows:

The source being the client sends the request to an Elastic IP address (In this example we are assuming 1.1.1.1 & 1.1.1.2 corresponding to worker nodes 1 & 2).
The IGW will do a static NAT translating those Public addresses to their corresponding private addresses. We are assuming in the figure that traffic will hit Worker Node 1 thus the source remains to be the Client and the destination becomes eth0's address on Worker Node 1 (192.168.45.167).
Based on the NodePort service configuration, the port the node is listening to is 30730 thus when traffic hits worker node 1 on port 30730, this will be DNATTed to the nginx-node-port address (192.168.22.71).
For traffic to return in the same path, an SNAT also occurs changing the source address from the Client to Worker Node 1's eth0 address (192.168.45.167).

It is also worth noting that a NodePort service also creates a ClusterIP that can be accessible within the Kubernetes Cluster which is depicted as 10.100.199.193 in the figure below.

Name:                     nginx-node-portNamespace:                defaultLabels:                   run=nginx-node-portAnnotations:              <none>Selector:                 run=nginx-node-portType:                     NodePortIP Family Policy:         SingleStackIP Families:              IPv4IP:                       10.100.199.193IPs:                      10.100.199.193Port:                     <unset>  80/TCPTargetPort:               80/TCPNodePort:                 <unset>  30730/TCPEndpoints:                192.168.22.71:80Session Affinity:         NoneExternal Traffic Policy:  ClusterEvents:                   <none>

IPTables Inner Workings

In this section, we'll explore the IPTable inner workings that govern the functionality of a NodePort Service in a Kubernetes cluster.

PREROUTING Chain: Our journey begins with the PREROUTING chain, which directs us towards the KUBE-SERVICES chain.

KUBE-SERVICES: Within KUBE-SERVICES, the policy KUBE-SVC-3KJW6L7XGSFMVIPO is instantiated as a ClusterIP. This does not impact traffic from outside the cluster.

KUBE-NODEPORTS: Next, we reach KUBE-NODEPORTS, where the first rule, matches traffic to tcp with a destination port of 30730 redirecting us to KUBE-EXT-3KJW6L7XGSFMVIPO which in turn redirects us to KUBE-SVC-3KJW6L7XGSFMVIPO

KUBE-SVC-3KJW6L7XGSFMVIPO: This redirects us to KUBE-SEP-KS7GFULRFICP7QTF.

KUBE-SEP-KS7GFULRFICP7QTF: Here, a DNAT rule performs destination NAT to the Pod address 192.168.22.71.

For those interested in the SNAT for the reverse direction, you can find its details below following this pattern:

KUBE-POSTROUTING
AWS-SNAT-CHAIN-0
AWS-SNAT-CHAIN-1 SNAT takes place with an address of 192.168.26.112. Note that this IPTables output is extracted from Worker Node 2 and represents eth0's address.

[ec2-user@ip-192-168-26-112 ~]$ sudo iptables -t nat -vLChain PREROUTING (policy ACCEPT 36 packets, 2160 bytes) pkts bytes target     prot opt in     out     source               destination          2208  147K KUBE-SERVICES  all  --  any    any     anywhere             anywhere             /* kubernetes service portals */ Chain KUBE-SERVICES (2 references) pkts bytes target     prot opt in     out     source               destination             0     0 KUBE-SVC-3KJW6L7XGSFMVIPO  tcp  --  any    any     anywhere             ip-10-100-199-193.ec2.internal  /* default/nginx-node-port cluster IP */ tcp dpt:http   52  3120 KUBE-NODEPORTS  all  --  any    any     anywhere             anywhere             /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCALChain KUBE-NODEPORTS (1 references)pkts bytes target     prot opt in     out     source               destination            0     0 KUBE-EXT-3KJW6L7XGSFMVIPO  tcp  --  any    any     anywhere             anywhere             /* default/nginx-node-port */ tcp dpt:30730Chain KUBE-EXT-3KJW6L7XGSFMVIPO (1 references)pkts bytes target     prot opt in     out     source               destination            0     0 KUBE-MARK-MASQ  all  --  any    any     anywhere             anywhere             /* masquerade traffic for default/nginx-node-port external destinations */   0     0 KUBE-SVC-3KJW6L7XGSFMVIPO  all  --  any    any     anywhere             anywhere            Chain KUBE-SVC-3KJW6L7XGSFMVIPO (2 references)pkts bytes target     prot opt in     out     source               destination            0     0 KUBE-SEP-KS7GFULRFICP7QTF  all  --  any    any     anywhere             anywhere             /* default/nginx-node-port -> 192.168.22.71:80 */Chain KUBE-SEP-KS7GFULRFICP7QTF (1 references) pkts bytes target     prot opt in     out     source               destination             0     0 KUBE-MARK-MASQ  all  --  any    any     ip-192-168-22-71.ec2.internal  anywhere             /* default/nginx-node-port */    0     0 DNAT       tcp  --  any    any     anywhere             anywhere             /* default/nginx-node-port */ tcp to:192.168.22.71:80Chain POSTROUTING (policy ACCEPT 246 packets, 16109 bytes)pkts bytes target     prot opt in     out     source               destination         82418 5146K KUBE-POSTROUTING  all  --  any    any     anywhere             anywhere             /* kubernetes postrouting rules */81338 5079K AWS-SNAT-CHAIN-0  all  --  any    any     anywhere             anywhere             /* AWS SNAT CHAIN */Chain AWS-SNAT-CHAIN-0 (1 references)pkts bytes target     prot opt in     out     source               destination         59065 3589K AWS-SNAT-CHAIN-1  all  --  any    any     anywhere            !ip-192-168-0-0.ec2.internal/16  /* AWS SNAT CHAIN */Chain AWS-SNAT-CHAIN-1 (1 references)pkts bytes target     prot opt in     out     source               destination         51947 3162K SNAT       all  --  any    !vlan+  anywhere             anywhere             /* AWS, SNAT */ ADDRTYPE match dst-type !LOCAL to:192.168.26.112 random-fully

Kubernetes Load Balancer Service

Load Balancer Provisioning

Load Balancer Service is another type of Kubernetes Service whereby we orchestrate Load Balancers that are outside the Kubernetes Cluster and have them serve as ingress points to a group of pods within our Kubernetes Cluster.

This generally requires a controller (Pod) that listens to all Kubernetes API Server activities that involve Kubernetes Load Balancer service. When the controller sees a service of type load balancer created, it creates an external Load Balancer that is mapped to that service. In AWS, this is depicted by the AWS Load Balancer Controller that also needs to have IAM permissions to create the Load Balancers within AWS.

The below are logs taken from the AWS Load Balancer controller pod and show that the controller is creating resources such as Load Balancers, Target Groups and Listeners which are again objects pertaining to Load Balancers in AWS. Please note that other Cloud Providers have similar functionality in terms of controllers that can orchestrate their Load Balancer offerings.

[cloudshell-user@ip-10-138-162-38 ~]$ kubectl logs -n kube-system aws-load-balancer-controller-5749768698-8jslx{"level":"info","ts":"2024-01-23T19:24:55Z","msg":"setting service loadBalancerClass","service":"nginx1","loadBalancerClass":"service.k8s.aws/nlb"}{"level":"info","ts":"2024-01-23T19:24:55Z","logger":"controllers.service","msg":"successfully built model","model":"{\"id\":\"default/nginx1\",\"resources\":{\"AWS::ElasticLoadBalancingV2::Listener\":{\"80\":{\"spec\":{\"loadBalancerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::LoadBalancer/LoadBalancer/status/loadBalancerARN\"},\"port\":80,\"protocol\":\"TCP\",\"defaultActions\":[{\"type\":\"forward\",\"forwardConfig\":{\"targetGroups\":[{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/default/nginx1:80/status/targetGroupARN\"}}]}}]}}},\"AWS::ElasticLoadBalancingV2::LoadBalancer\":{\"LoadBalancer\":{\"spec\":{\"name\":\"k8s-default-nginx1-36dbca87e3\",\"type\":\"network\",\"scheme\":\"internet-facing\",\"ipAddressType\":\"ipv4\",\"subnetMapping\":[{\"subnetID\":\"subnet-0bb01040718d4acb6\"},{\"subnetID\":\"subnet-0c11a935b51724fa2\"}]}}},\"AWS::ElasticLoadBalancingV2::TargetGroup\":{\"default/nginx1:80\":{\"spec\":{\"name\":\"k8s-default-nginx1-25a492f569\",\"targetType\":\"ip\",\"port\":80,\"protocol\":\"TCP\",\"ipAddressType\":\"ipv4\",\"healthCheckConfig\":{\"port\":\"traffic-port\",\"protocol\":\"TCP\",\"intervalSeconds\":10,\"timeoutSeconds\":10,\"healthyThresholdCount\":3,\"unhealthyThresholdCount\":3},\"targetGroupAttributes\":[{\"key\":\"proxy_protocol_v2.enabled\",\"value\":\"false\"}]}}},\"K8S::ElasticLoadBalancingV2::TargetGroupBinding\":{\"default/nginx1:80\":{\"spec\":{\"template\":{\"metadata\":{\"name\":\"k8s-default-nginx1-25a492f569\",\"namespace\":\"default\",\"creationTimestamp\":null},\"spec\":{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/default/nginx1:80/status/targetGroupARN\"},\"targetType\":\"ip\",\"serviceRef\":{\"name\":\"nginx1\",\"port\":80},\"networking\":{\"ingress\":[{\"from\":[{\"ipBlock\":{\"cidr\":\"192.168.0.0/19\"}},{\"ipBlock\":{\"cidr\":\"192.168.32.0/19\"}}],\"ports\":[{\"protocol\":\"TCP\",\"port\":80}]}]},\"ipAddressType\":\"ipv4\"}}}}}}}"}{"level":"info","ts":"2024-01-23T19:24:56Z","logger":"controllers.service","msg":"creating targetGroup","stackID":"default/nginx1","resourceID":"default/nginx1:80"}{"level":"info","ts":"2024-01-23T19:24:56Z","logger":"controllers.service","msg":"created targetGroup","stackID":"default/nginx1","resourceID":"default/nginx1:80","arn":"arn:aws:elasticloadbalancing:us-east-1:87125367xxxx:targetgroup/k8s-default-nginx1-25a492f569/ad12b7f6eaf1ee7d"}{"level":"info","ts":"2024-01-23T19:24:56Z","logger":"controllers.service","msg":"creating loadBalancer","stackID":"default/nginx1","resourceID":"LoadBalancer"}{"level":"info","ts":"2024-01-23T19:24:57Z","logger":"controllers.service","msg":"created loadBalancer","stackID":"default/nginx1","resourceID":"LoadBalancer","arn":"arn:aws:elasticloadbalancing:us-east-1:87125367xxxx:loadbalancer/net/k8s-default-nginx1-36dbca87e3/7ff79bdbac47fb35"}{"level":"info","ts":"2024-01-23T19:24:57Z","logger":"controllers.service","msg":"creating listener","stackID":"default/nginx1","resourceID":"80"}{"level":"info","ts":"2024-01-23T19:24:57Z","logger":"controllers.service","msg":"created listener","stackID":"default/nginx1","resourceID":"80","arn":"arn:aws:elasticloadbalancing:us-east-1:87125367xxxx:listener/net/k8s-default-nginx1-36dbca87e3/7ff79bdbac47fb35/ead861bf3e4bcbb5"}{"level":"info","ts":"2024-01-23T19:24:57Z","logger":"controllers.service","msg":"creating targetGroupBinding","stackID":"default/nginx1","resourceID":"default/nginx1:80"}{"level":"info","ts":"2024-01-23T19:24:57Z","logger":"controllers.service","msg":"created targetGroupBinding","stackID":"default/nginx1","resourceID":"default/nginx1:80","targetGroupBinding":{"namespace":"default","name":"k8s-default-nginx1-25a492f569"}}{"level":"info","ts":"2024-01-23T19:24:57Z","logger":"controllers.service","msg":"successfully deployed model","service":{"namespace":"default","name":"nginx1"}}{"level":"info","ts":"2024-01-23T19:24:57Z","msg":"authorizing securityGroup ingress","securityGroupID":"sg-02cd536f30a36abed","permission":[{"FromPort":80,"IpProtocol":"tcp","IpRanges":[{"CidrIp":"192.168.0.0/19","Description":"elbv2.k8s.aws/targetGroupBinding=shared"}],"Ipv6Ranges":null,"PrefixListIds":null,"ToPort":80,"UserIdGroupPairs":null},{"FromPort":80,"IpProtocol":"tcp","IpRanges":[{"CidrIp":"192.168.32.0/19","Description":"elbv2.k8s.aws/targetGroupBinding=shared"}],"Ipv6Ranges":null,"PrefixListIds":null,"ToPort":80,"UserIdGroupPairs":null}]}{"level":"info","ts":"2024-01-23T19:24:57Z","msg":"authorized securityGroup ingress","securityGroupID":"sg-02cd536f30a36abed"}{"level":"info","ts":"2024-01-23T19:24:58Z","msg":"registering targets","arn":"arn:aws:elasticloadbalancing:us-east-1:87125367xxxx:targetgroup/k8s-default-nginx1-25a492f569/ad12b7f6eaf1ee7d","targets":[{"AvailabilityZone":null,"Id":"192.168.39.147","Port":80}]}

Instance Target

For Instance Target, we are configuring the Network Load Balancer to send the traffic to one of the two worker nodes which again have a NodePort configuration by which they listen to port 31729 in our example.

Thus, the follow works as follows:

The client resolves the FQDN name of the Network Load Balancer. Note that NLBs are generally implemented across Availability Zones for redundancy. As soon as we resolve the NLB's FQDN, the client will sends the packet towards one of the two NLBs Public address
The Internet Gateway at the edge of the VPC does a static 1:1 NAT changing the Public address of the chosen NLB to the private address 192.168.41.233 (Left NLB)
The left NLB has two targets which are the eth0 of Worker Nodes 1 & 2 that are both listening on TCP Port 31729. In our example, we assume the traffic goes to Worker Node 1. Thus till this point the source is the client IP and the destination is Worker Node 1's eth0 192.168.45.167.
Now that the packet has hit Worker Node#1, again similar to the behavior we have seen with NodePort, a DNAT will occur translating the destination tot he Pod address 192.168.22.71 and an SNAT will change the client address to 192.168.45.167.

The way we can tune the NLB's behavior is via Annotations set on the service, specifically the setting:

service.beta.kubernetes.io/aws-load-balancer-nlb-target-type

Name:                     nginx1Namespace:                defaultLabels:                   run=nginx1Annotations:              service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance                          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing                          service.beta.kubernetes.io/aws-load-balancer-type: externalSelector:                 run=nginx1Type:                     LoadBalancerIP Family Policy:         SingleStackIP Families:              IPv4IP:                       10.100.69.226IPs:                      10.100.69.226LoadBalancer Ingress:     k8s-default-nginx1-36dbca87e3-7ff79bdbac47fb35.elb.us-east-1.amazonaws.comPort:                     <unset>  80/TCPTargetPort:               80/TCPNodePort:                 <unset>  31729/TCPEndpoints:                192.168.39.147:80Session Affinity:         NoneExternal Traffic Policy:  ClusterEvents:                   <none>

In the figures below, we can also see the AWS outputs showing that the targets are the two worker node instances and the NodePort configured is 31729.

The two images here show packet captures from both the Node & Pod perspective where the Node actually sees the client IP (Public Address) in this example and the Pod sees the Node IPs as the source 192.168.45.167 & 192.168.26.112.

IP Target

This is again another iteration of configuring the NLB whereby in this mode, the NLB directly sends the packet to the Pod instead of sending it to the Node. This is possible in our example because we are using the AWS VPC CNI whereby a Pod behaves just like a VM in the sense that it gets an IP address from the VPC range & thus can be reached directly from components within the VPC.

This is again possible by setting the below annotation.

service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

Name:                     nginx1Namespace:                defaultLabels:                   run=nginx1Annotations:              service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip                          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing                          service.beta.kubernetes.io/aws-load-balancer-type: externalSelector:                 run=nginx1Type:                     LoadBalancerIP Family Policy:         SingleStackIP Families:              IPv4IP:                       10.100.69.226IPs:                      10.100.69.226LoadBalancer Ingress:     k8s-default-nginx1-36dbca87e3-7ff79bdbac47fb35.elb.us-east-1.amazonaws.comPort:                     <unset>  80/TCPTargetPort:               80/TCPNodePort:                 <unset>  31729/TCPEndpoints:                192.168.39.147:80Session Affinity:         NoneExternal Traffic Policy:  ClusterEvents:                   <none>

Similar to what we have seen above, this is the AWS output showing the target being the Pod Address 192.168.39.147.

Again, here we show the packet captures on both the Node and the Pod. Notice that even the Node sees the traffic as sourced from the LB and destined to the Pod IP thus there is no need for NodePort treatment at the Worker Nodes.

Drawbacks of Load Balancer Service

The main drawback of the Load Balancer Service is a 1:1 mapping between the Load Balancer and the service (group of pods) that you want to expose. This has multiple implications including increased costs, more management overhead, public addresses, and the need to secure all the Load Balancers with WAF, API Security, Anti-DDoS solutions. If one of these services goes uncovered or unprotected there could be security breaches leading to major losses.

The diagram below explains the problem where for each service, the AWS LB Controller will instantiate an NLB.

The below diagram emphasizes the same challenge whereby we can see our Kubernetes Cluster (here depicted as a single node) and all the different entry points that map to the different services.

]]>

Kubernetes Services: ClusterIP

Karim El Jamali — Fri, 12 Jan 2024 20:09:35 GMT

Brief Introduction

Kubernetes Services serve as an abstraction layer for one or more pods operating behind them. If you're thinking Load Balancers or proxies, you are on the right path.

In essence, pods are transient resources within Kubernetes, typically instantiated through Deployments. These Deployments leverage controllers to ensure a consistent minimum number of pods is always active. Referring to the below figure, there's a Pod named "tmp-shell" that needs to communicate with an nginx pod. "Tmp-shell" doesn't care about the specific nginx pod it connects to. Even if the actual pods in the nginx service change, "tmp-shell" isn't bothered by these changes. Kubernetes Services provide this abstraction. In our example, the nginx service hides the details of individual nginx pods from "tmp-shell." So, "tmp-shell" connects to the nginx service, and the service then directs the request to one of the available endpoints or targets.

One lingering question is: How does "tmp-shell" discover the service? The service has both an IP address and a DNS name. Consequently, "tmp-shell" initiates a DNS request to CoreDNS, the Kubernetes DNS Service, seeking to resolve the DNS name associated with the nginx service. Upon receiving a response from CoreDNS, "tmp-shell" can establish a connection to the nginx service using its IP address.

There are multiple service types in Kubernetes, those that expose Pods internally within the Kubernetes Cluster and those that expose Pods externally (Outside the Kubernetes Cluster). In this blog post, we will mainly focus on ClusterIP service that exposes pods within the Kubernetes cluster just like the example we have been discussing between tmp-shell and nginx where all of these pods reside within the same Kubernetes cluster.

Where do Services Live?

Services are implemented through iptable rules on each node to manage traffic effectively. Whenever a service is created or updated, these iptable rules are adjusted to ensure that traffic is correctly directed to the designated pods.

ClusterIP services are instantiated on all nodes, meaning iptable rules are programmed across the entire cluster. Think of it like an Anycast Address, extending its presence across all nodes in the cluster. In the figure below, the nginx service is created with 3 supporting pods nginx-1, nginx-2 and nginx-3. All nodes have an instantiation of the service depicted as IPtables configuration, and the service shares a common virtual address, exemplified as 10.96.76.14 in our case.

When Client-1 on Worker Node-1 sends a request to the nginx service, the service must decide which of the four backing pods to route the request to. With four pods, each one handles 25% of the traffic. Importantly, even on nodes without nginx pods, the service is still instantiated, as observed in the case of Worker Node 3.

Traffic Flow

In this section, we delve into the traffic flow between the "tmp-shell" pod and the nginx pods, highlighting key points:

Virtual Ethernet Pairs: "tmp-shell" connects to the host through virtual Ethernet pairs, where one end is labeled as eth0 on the pod, and the other end is an interface named lxc01 for "tmp-shell." This pattern applies consistently across all pods in the cluster. You can think about it like a virtual wire connecting the Pod to the host.
Service Name Resolution: To communicate with the nginx service, "tmp-shell" needs to resolve the IP address mapped to the service name. The naming convention for services is <servicename>.<namespace>.svc.cluster.local assuming the default domain name. In our example, using the default namespace, the nginx service's FQDN is "nginx.default.svc.cluster.local." Subsequently, a packet is sent from "tmp-shell" (10.0.0.8) to the nginx service address (10.96.76.14) upon receiving the DNS response.
Destination NAT (DNAT): iptables perform a Destination NAT, converting the nginx service address (10.96.76.14) to the nginx pod address. This is demonstrated in the example with a single endpoint or backing pod for the nginx service, though real-world scenarios may involve multiple pods.
Pod-to-Pod Routing Patterns: There are two high-level patterns for pod-to-pod routing across nodes:
- Native Routing: Pod addresses traverse the network between nodes unchanged. In this scenario, the underlying network must be aware of Pod addresses, necessitating proper routing. However, the absence of NAT offers better visibility into pod-to-pod communication.
- Overlays: Regardless of the tunneling mechanism, packets leave the node with an outer envelope indicating source/destination worker nodes. The underlying network views these packets as originating from one worker node and destined for another. Beyond this point, the nginx pod receives the packet with the source address as "tmp-shell" (10.0.0.8).
Response Handling: Upon reaching Worker Node 1, the packet's source is 10.0.1.79, and the destination is 10.0.0.8. However, "tmp-shell" won't accept this packet as the original packet it sent was from itself (10.0.0.8) to the nginx service (10.96.76.14). iptables intervenes again, performing Source NAT (SNAT) of 10.0.1.79 (nginx address) to the nginx service (10.96.76.14), aligning with the original packet flow.

This detailed explanation clarifies the steps in how "tmp-shell" communicates with the nginx pods throughout the Kubernetes cluster.

Here you can find the packet captures that align with the explanation on how it works.

The iptables rules operate as follows:

PREROUTING Chain:
- Initiates the process, directing traffic to the KUBE-SERVICES chain.
KUBE-SERVICES Chain:
- Checks if the destination matches the Cluster IP (10.109.91.212) of the nginx service.
- If matched, redirects the traffic to the KUBE-SVC-2CMXP7HKUVJN7L6M chain.
KUBE-SVC-2CMXP7HKUVJN7L6M Chain:
- Directs traffic to the KUBE-SEP-XZRG5L6J3WL7H422 chain, as indicated by the comment.
KUBE-SEP-XZRG5L6J3WL7H422 Chain:
- Utilizes DNAT (Destination Network Address Translation) in its final rule.
- This rule modifies the destination of the traffic, rerouting it to the nginx pod.

These iptables rules collectively ensure that traffic directed to the specified nginx service IP undergoes the necessary transformations, ultimately reaching the appropriate nginx pod.

Quick Note on IPTables:

KUBE-SERVICES: This is a container for all Kubernetes Services configured on the cluster.

KUBE-SVC-*: This is an entry for a particular service. e.g. nginx service

KUBE-SEP-*: In IPTables, when you see KUBE-SEP-* this is an entry for a particular endpoint

Thus in our example, you can see one KUBE-SVC-* and one KUBE-SEP-* as we have a single service (nginx) and a single endpoint (10.0.1.79)

-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES-A KUBE-SERVICES -d 10.109.91.212/32 -p tcp -m comment --comment "default/nginx cluster IP" -m tcp --dport 80 -j KUBE-SVC-2CMXP7HKUVJN7L6M-A KUBE-SVC-2CMXP7HKUVJN7L6M -m comment --comment "default/nginx -> 10.0.1.79:80" -j KUBE-SEP-XZRG5L6J3WL7H422-A KUBE-SEP-XZRG5L6J3WL7H422 -p tcp -m comment --comment "default/nginx" -m tcp -j DNAT --to-destination 10.0.1.79:80

The below IPtables example, show what happens when we add another endpoint to back the nginx service thus we have two endpoints. You can now see that we still have one KUBE-SVC-* (one service) and 2 KUBE-SEP-* (2 Endpoints) and the service sends to the particular endpoint with a probability of 0.5 (50%) thus equally distributing the load across both endpoints.

Endpoint 1: 10.0.1.79

Endpoint 2: 10.0.0.224

-A KUBE-SVC-2CMXP7HKUVJN7L6M -m comment --comment "default/nginx -> 10.0.0.224:80" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-HZXSYA2C4E7MQSH3-A KUBE-SVC-2CMXP7HKUVJN7L6M -m comment --comment "default/nginx -> 10.0.1.79:80" -j KUBE-SEP-XZRG5L6J3WL7H422-A KUBE-SEP-XZRG5L6J3WL7H422 -p tcp -m comment --comment "default/nginx" -m tcp -j DNAT --to-destination 10.0.1.79:80-A KUBE-SEP-HZXSYA2C4E7MQSH3 -p tcp -m comment --comment "default/nginx" -m tcp -j DNAT --to-destination 10.0.0.224:80

How are these IPTables Programmed?

Whenever any changes happen for the Kubernetes Services or Endpoints, API Server shares these changes with Kube-proxy which in turn programs the IPTables accordingly.

]]>

Istio Networking Part 2

Karim El Jamali — Mon, 27 Nov 2023 05:00:00 GMT

Introduction

After providing a brief introduction to Istio in Part 1, this blog post will center on a detailed exploration of how Envoy intercepts all traffic, whether sourced or destined for the application container.

Our Application

For a visual overview, check out the diagram below, sourced from Istios documentation. It vividly illustrates the key components and their interactions.More details can be found here.

Product Page to Reviews Communication

In the upcoming sections, we will only focus on the communication between productspage and the reviews service particularly reviews-v1. Note that the reviews service has multiple versions as shown in the below figure and the configuration is done in such a way that traffic has to go to the reviews-v1 subset. Again, the objective is to be able to understand how does this communication flow end-to-end.

From ProductsPage Container to SideCar

Lets start with the productspage container as it is the source of the request (Reference Figure1).

DNS request that is destined to CoreDNS to resolve reviews.default.svc.cluster.local.
The response comes back as 10.107.148.159 which is the Kubernetes service IP for the reviews service.
It is worth noting that one of the endpoints of that service is 10.0.0.38 (reviews-v1).

Now, the intriguing question arises: How does the packet from 10.0.1.136 (Product Page) to 10.0.0.38 (Reviews-v1) navigate its way through the Envoy proxy?

IPTables is the secret sauce behind getting the packet redirected to Envoy. As the packet exits the Product Page container, it encounters the IPTables chains listed below. Remember, our packet has a source of 10.0.1.136 and a destination of 10.0.0.38.

We will start with the OUTPUT chain which will send us to the ISTIO_OUTPUT Chain.
If you look at the ISTIO_OUTPUT chain the 1st rule doesnt match as the source should be 127.0.0.6. The rules beneath generally match the UID 1337 which is traffic sourced by the Envoy sidecar thus these rules wont match. In addition, there are rules that have an out interface of loopback which will not match. Thus, the last rule in the ISTIO_OUTPUT chain will match sending us towards the ISTIO_REDIRECT Chain.
Within the ISTIO_REDIRECT chain, we have a REDIRECT rule which redirects traffic to port 15001 i.e. it will change the destination address to localhost (127.0.0.1) and destination port to 15001.

The output below also proves that the Envoy proxy is actually listening on port 15001.

Up to this point, weve observed the packet departing the Product Page container and successfully reaching Envoy through IPTables redirection

Envoy Processing

In this section, our attention shifts to the processing of the packet by the productpage sidecar i.e. the Envoy proxy co-residing with the productpage container. The below outputs can be summarized as follows:

We examine listeners on port 15001, identifying the destination as PassthroughCluster. The PassthroughClusters type is ORIGINAL_DST, indicating that Envoy will revert the packets destination to its pre-IPTables redirection state, 10.0.0.38:9080.
Our attention then turns to Envoy routes, specifically focusing on reviews.default.svc.cluster.local:9080. The route view reveals the cluster as outbound|9080|v1|reviews.default.svc.cluster.local. Again, this is a result of the intended configuration where we want traffic that hits the reviews service to go to reviews-v1.
Finally, we need to figure out the specific endpoint within the Cluster which is the reviews-v1 pod address 10.0.0.38.

Lastly, inspecting the productpage Envoy logs, we observe that the request originates from 10.0.0.136 and is destined for 10.0.0.38:9080.

IPTables on Reviews-v1 Pod

As the packet from 10.0.0.136 (productspage) headed for 10.0.0.38 (reviews-v1) with a destination port of 9080 reaches the second node and enters the reviews-v1 pod network namespace, a familiar journey unfolds. Much like our previous process, the packet undergoes redirection to the Envoy sidecar, and once again, IPtables comes to the rescue.

We will start with the PREROUTING chain as this is an incoming packet. The PREROUTING chain will directly send us to the ISTIO_INBOUND chain.
If you look at the ISTIO_INBOUND chain the 1st 4 rules do not match as the destination port is not 15008, 15090, 15021, and 15020 (Destination Port: 9080). This takes us to the ISTIO_IN_REDIRECT chain.
Within the ISTIO_IN_REDIRECT chain, we have a REDIRECT rule which redirects traffic to port 15006 i.e. it will change the destination address to localhost (127.0.0.1) and destination port to 15006. Again you can refer to the netstat output to see that Envoy listens for port 15006.

Envoy Processing

Please note that this is the Envoy proxy (sidecar) of the reviews pod.

Upon inspecting the Envoy proxy within the reviews pod, a quick confirmation reveals a listener on Port 15006, aligning with expectations. This listener efficiently directs traffic to the Cluster inbound|9080||. Examining the cluster further, denoted as inbound|9080||, reveals its type as ORIGINAL_DST, signifying that Envoy will revert the destination from the redirected address (127.0.0.1:15006) to the original one (10.0.0.38:9080).

Another interesting finding here is that the sourceAddress will be changed to 127.0.0.6. Thus, our packet will have a source as 127.0.0.6 & a destination as 10.0.0.38:9080. This can be seen in the below output.

Finally, the below figure puts it all together.

]]>

Istio Networking Part 1

Karim El Jamali — Sun, 26 Nov 2023 17:00:00 GMT

About the Series

In this two-part series, we embark on a journey into the realm of Istio, a powerful service mesh that has become integral to modern cloud-native applications. In this first installment, our focus will be on understanding the fundamental need for a service mesh, exploring the high-level architecture of Istio, and ending with a brief explanation of Envoy.

In the upcoming part, we will deep dive on a how the Envoy sidecar intercepts all traffic sourced or destined to the application container.

From Monoliths to Microservices

Unless you have been hiding under a rock, you would have heard about the shift from Monoliths to Microservices. Actually, this topic even gets revisited when migrating workloads to the cloud. While working as a Technical Account Manager (TAM) for AWS in the past, we worked with customers to come up with the optimum migration strategy for each workload or component. If this is a topic of interest you can find some details here.

Figure2. Microservices distributed across multiple nodes

Now that we have shifted to running microservices across multiple nodes, a few arising challenges come to the picture:

Network Performance: The performance of the underlying network becomes pivotal in determining the quality of our applications user experience.
Security: With microservices communicating across networks, encryption becomes paramount to safeguard data in transit.
Observability and Traceability: This is another key concern in microservices where a transaction goes across these different components and we lose the visibility of an end-to-end user journey. The solution to this problem is distributed tracing . In addition, we will always need to have component level metrics & alerts.
Identity & Zero Trust: With the multitude of components and services running in a distributed fashion, we will need to strive to ensure that none of these components gets compromised and even if any of those components were to get compromised, we need to ensure that it cannot be used as a pivot to attack other critical services. This requires a solid identity foundation whereby each service has an identity and a role. Thus, any service to service call has to be authenticated & authorized with least privilege principle in mind. Advanced Load Balancing: Each of the components depicted as part of an application will have a number of replicas for purposes of scaling and high availability, thus at a minimum we will need to distribute load across all endpoints of a particular service in an efficient manner. This encompasses requests from external users or customers as well as requests between those microservices. Microservices allows pushing newer versions of individual components (billing, product details, ..etc) more frequently. This is another critical use-case for load balancing where for instance you can have 95% of requests target the stable version and only 5% target the updates versions and keep adjusting the weights based on your testing. Another possibility is sending your internal authenticated users to the new version and keeping external customers on the production/stable version.
Resiliency: As per the Principles of Chaos Engineering website, Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the systems capability to withstand turbulent conditions in production. Distributed complex systems are always prone to fail however our objective should be to stop the overall failure of the application or service we provide customers and end-users. Thus, we need tooling that will help application owners test their application in chaotic conditions whereby for instance a component responds with errors or delay a response and see the effect on the end-to-end user journey.
Manageability: Just like Kubernetes offers a single API to manage all resources, it is a must that all those capabilities (1-6) be centrally orchestrated.

These challenges birthed the need for a service mesh, with Istio emerging as the de facto solution. It might be interesting to note that prior to service mesh, attempts were made to solve those challenges with software libraries like Stubby and Hystrix.

Istio Architecture

At a high level Istios architecture has 2 main components:

Envoy proxies (dataplane): This is the data plane component of the solution. The Envoy proxy runs as a sidecar i.e. it runs alongside the main service in the pod to provide additional functionality which in our case is security, observability, resiliency and load balancing.
Istio Control Plane (istiod): So we have added those Envoy proxies, but we still need a central management & operations to manage all of those proxies at scale. Istiod or the Istio Control Plane actually fills this gap whereby it manages all of the configurations of the Envoy proxies. One of the reasons that Envoy was chosen as the sidecar for Istio deployment is its inherent ability to be dynamically configured via APIs. As a side note, when running Istio with Kubernetes, Istio automatically detects the services and their endpoints in that particular cluster and makes an entry for each service within its internal registry.

Figure3. Istio Architecture

Figure4. Istio Data Plane

It is also worth noting that there is another mode of Istio named Ambient Mesh that removes the need of sidecars from the dataplane. More on this can be found here.

Quick Word on Envoy

I will end up this part by a very high level overview of how Envoy works. If you have done any work on Load Balancers, Envoy is pretty similar from a grand scheme of things.

Client or the requester is referred to as downstream.
The Server is referred to as upstream.
Envoy just like any other Load Balancer, needs to have a listener whereby Envoy is waiting for requests that match a particular address or any address and a particular port. An example could be 0.0.0.0:80 whereby the listener will accept requests on any interface with a destination port of 80. Any request that doesnt hit the listener port is ignored by Envoy.
Once Envoy receives the packet on the listener, it subjects it to a Filter Chain i.e. a set of filters which determine how to deal with the incoming packet. You might want an L3/L4 Load Balancer and thus might use the TCP proxy filter or need forwarding based on L7 HTTP semantics with HTTP Connection Manager. TLS inspector is another commonly used filter to detect if the transport is plaintext or TLS and if TLS can detect Server Name Indication (SNI) and/or Application Layer Protocol Negotiation (ALPN). This filter chain, assuming the request is accepted, will most likely finalize by sending the request to the Clusters (Upstream components).
Clusters can be envisioned as collections of endpoints or targets. In a highly available application, a Cluster could comprise multiple Nginx web servers. Once a specific cluster is chosen, the remaining decision lies in selecting an endpoint within the cluster, typically governed by a load balancing policy.

Figure5. Envoy Constructs: Listeners, Filters, Clusters & Endpoints

In the upcoming part, well embark on a deep dive into the intricate workings of how Envoys proxy intercepts all traffic sourced and destined for the container.

]]>

Aviatrix Network Segmentation

Karim El Jamali — Wed, 16 Feb 2022 05:00:00 GMT

Background on Network Segmentation

Network Segmentation (a.k.a VRFs) is a common theme in large enterprises and service providers.

Service Providers use segmentation at their Provider Edge (PE) routers whereby those PE(s) connect to multiple Customer Edge (CE) routers while maintaining separate routing domains for each of those customers.

Enterprise environments generally have a multitude of services (Data, IP Telephony, IoT, CCTV, IPTVetc) whereby each service can run on its own dedicated infrastructure or have them all share a common infrastructure. In addition, multiple applications running on-premises and cloud have strict compliance and security requirements on segmentation & isolation. One example of compliance requirement is The Payment Card Industry Data Security Standard (PCI DSS) which sets a widely accepted set of policies and procedures intended to optimize the security of credit, debit and cash card transactions and protect cardholders against misuse of their personal information. PCI DSS strongly recommends network segmentation or isolating the cardholder data environment from the rest of the network. On another note, the majority of customers run development and production environments and it is imperative for security, operations and compliance reasons to keep both environments isolated. Finally, overlapping addresses as a result of mergers & acquisitions (M&A) or poor planning are a common reality and at many times are dealt with via network segmentation and/or NAT.

One of the concerns of a common routing domain is a large blast radius whereby any problem involving routing injection or deletion could result in outages across all of these services. In addition, a common routing domain allows attackers to target specific IoT devices with a lower security posture and use those IoT devices to sneak into the crown jewels within the environment. Since most of these services operate independently of one another, a proper security policy would be to isolate them from one another thus improving the environments security posture.

Cloud Network Segmentation

Let us start by looking at the below diagram. The diagram shows:

2 CSPs (AWS, Azure) and an on-prem environment.
3 Virtual Machines (dev, prod and shared) in each of the spoke VPCs.

If you would like to better understand the architecture, you can have a look here.

Without network segmentation, we operate in a common routing domain i.e. there is a common routing table for dev, prod and shared services allowing routing reachability between all 3 workloads. This however, doesnt always equate to data plane reachability as there are security constructs that can be leveraged to block this communication (Security Groups, Network Security Groups, NACLs, Firewalls).

Common Routing Domain Explained

The below diagram dives a bit further into the common routing domain (i.e. no segmentation). I have removed the workloads for simplicity, thus we have two spoke gateways (dev-1 and prod-1) connected to a common transit (transit-1). Note that in this setup, the transit has a common/default routing table that gets shared with the dev-1 and prod-1 spokes allowing any-to-any routing reachability.

Segmenting the Network

The below diagram segments the infrastructure to two network domains (Dev/Prod). Please note the following:

Transit routers will have all the routing tables (similar to a PE router) or Network Domains (Dev/Prod)
Upon attaching a spoke gateway (e.g. prod-1) to the transit gateway (transit-1) the transit gateway will associate the spoke gateway with a Network Domain (Prod) limiting its view of the world to the Prod search space or routing table. In this example, when traffic reaches the prod-1 GW from the prod instances they will be subject to the prod-1 routing table thus no communication is possible between prod & dev workloads.
It is important to realize that that this segmentation takes place at the transit layer i.e. any connection to the transit where the transit is enabled for segmentation can be associated or placed in a network domain limiting its search space.

Extending Segmentation to other Regions and Clouds

The below figure expands the same logic to another transit. Two important points to recall here:

All the segmentation we are talking about here is CSP agnostic and can extend to multiple regions, account and CSPs (AWS, Azure, GCP..etc)
All the management and control plane is fully automated by the controller thus the controller fully manages all the gateway and CSP routing without any manual intervention.

Connection Policy

Up till the last section, we have been able to isolate dev/prod workloads across all our multi-cloud architecture. In reality, customers tend to have shared services such as NTP, DNS, Syslog, Netflow Collectors..etc that service both their production and development environments. Thus, we need to ensure that both dev/prod remain isolated but able to access shared services. A connection policy is a way to leak routes between Network Domains to allow access to shared services as depicted in the figure below. Notice that the routing tables for both dev-1 and shared-svcs spoke gateways now have the routes to each other thus allowing communications from a routing plane. The same logic can be extended to prod, thus prod/dev cant communicate but they can communicate with shared-svcs.

Extending Segmentation to On-Premises

Now that we have extended our segmentation to all our cloud infrastructure, we would like to extend it to on-premises. In the below diagram the on-premises edge (on-prem RTR) has a Site2Cloud (S2C) connection to the transit. You can find more details about S2C here but for the purposes of our discussion, you can think of it like an IPSec tunnel with BGP running on top for route-exchange. In this setup, we would like to put the on-prem RTRs connection to the transit in a network domain (Prod) so that it can only access the prod workloads across the different CSPs.

Extending Multiple Segments from On-Prem to Cloud

In the last section, we extended the Network Domain (prod) to on-premises. The below figure takes this a step further whereby we would like to extend all the segments from on-premises to Cloud thus enabling an end-to-end segmentation strategy. Manual VRF configuration and SDN Fabrics are both viable examples of implementing on-premises segmentation, and our goal is to extend those segments into our Multi-Cloud Network Architecture.

In the diagram the underlay RTR connection to the VGW is an IPSec tunnel but the same holds true for Direct Connect (DX). Please note the following:

On the on-premises side, I have put two routers to clarify the example, however this could be done with a single router.
Think of the underlay RTR as building connectivity between the overlay RTR and the transit.
Underlay RTR in this example is running an IPSec tunnel with BGP to AWSs VGW thus sending all the routes including the two loopbacks that are originally advertised by the overlay RTR. Now transit-1 has reachability to Lo0 and Lo1 and overlay RTR has reachability to 10.100.0.0/16. Thus, we now have a connectivity between the overlay RTR and the transit-1.
Now that VGW has received the routes, we leverage route propagation in order to ensure these routes are advertised to transit-1 GW routing table. This is depicted in the diagram with the orange VPC router and its associated route table where all these routes are propagated. In addition, by default the transit-1 router has a default route pointing to the VPC router and thus able to reach the loopbacks advertised by the overlay router (100.100.100.1/32 & 100.100.100.2/32)

Now that we have built the underlay connectivity between the overlay RTR, and transit-1 we can now build the GRE tunnels where one tunnel is sourced from lo0 and has a destination the private IP address of the transit that belongs to 10.100.0.0/16. This tunnel interface can be configured to be in VRF/Network Domain Dev. The same logic would follow for VRF prod leveraging lo1.

The below diagram simplifies the explanation boiling it down to:

You need an underlay connectivity (sold red line) between the overlay RTR and Transit-1
Once that connectivity is built, you can build multiple tunnels that utilize this underlay and place each of them in its own VRF/Network Domain
The below figure shows two tunnels one for Dev Network Domain/VRF extension and the other one for Prod Network Domain/VRF extension.

Segmentation Made Simple

With all its capabilities and advanced use cases, one of Aviatrixs major values is simplicity where the controller abstracts many of the complexities within Cloud Networking. The below figure shows a network domain (Dev) being created, allowing the Dev Domain to connect to Shared-Svcs Domain and associating the Dev Domain with a spoke (aws-us-east-1-dev-1)

Check it Out

If you are keen on trying, I have created a github repository here that builds a lab infrastructure that you can use to test segmentation and other cool features and get acquainted with the platform. Sandbox Starter Tool is another great way to get acquainted with the platform. More information on this can be found here.

]]>