Use Kubernetes primitives to find the problem in Cloud Foundry applications
Debugging cf-for-k8s
Three weeks ago the cf-for-k8s repository has been open-sourced. It allows you to deploy Cloud Foundry components in Kubernetes and use cf push against that deployment. There are already several projects that enable deploying Cloud Foundry to Kubernetes — KubeCF, its previous version SCF, several Kubernetes CPIs for Bosh. However, they all leverage BOSH in one way or another, cf-for is the first native project that uses everything built from scratch.
The project is just started and has minimal functionality that enables teams to integrate. Right now it only allows users to use Docker push, but there is work ongoing to introduce kpack as a building mechanism.
Deploying cf-for-k8s is relatively simple, you can follow the official readme. However, problems might happen. What to do when the application doesn’t work? To do that you need to know a deployment flow.
How does cf push work in cf-for-k8s?
(This section will be missing some pieces soon when features are getting added to the cf-for-k8s).
The first step is staging. During that step, the application is prepared to run. In case of source code it is compiled and put on Docker image. When you are pushing the Docker image, Eirini should extract the port on which application is running, so it can generate the health check, but right now this happening in CAPI. We are working on a story to fix it. The important part that during the staging process CAPI will make a call to Eirini and wait for a callback(but not right now).
The next step is deploying. CAPI sends the request to Eirini with the application information. Then Eirini creates a stateful set which creates a pod.
In the parallel, the routesync component makes sure that application is routable. It creates a service in Kubernetes and a virtual service for Istio. As soon as application starts it is routable.
Debugging
So imagine, you have pushed the application but you can’t access it.
Now when you know what is happening you can debug the issue.
First, you can check if routesync worked fine — check if you have a service and a virtual service: kubectl get virtualservices.networking.istio.io,services -n cf-workloads . If they are not, you should check the logs for the routesync component kubectl -n cf-system logs cfroutesync-6b6844444-xrdm --all-containers
Then you can check if the pods are created: kubectl get pods -n cf-workloads. If there are no pods, you can also check statefulsets kubectl get statefulsets -n cf-workloads .
If there are stateful sets, but not pods there might be several reasons and you can find them if you run kubectl describe statefulset -n cf-workloads and find out the reason why. There might be not enough resources or something wrong with the security policy.
If there are no stateful sets, then Eirini didn’t work, so check the logs. You should see the logs with all requests from CAPI something like :
{"timestamp":"2020-02-15T14:38:35.767557362Z","level":"debug","source":"handler","message":"handler.get-app.requested","data":{"guid":"a2448075-715c-40a9-aead-7e8230cdb709","session":"3","version":"2d45687b-d7cb-4850-9b6a-f8152a1a1dcd"}}If you don’t see that message in logs, then there is an issue at CAPI end and you can try to check CAPI logs for the issue. In this case, the issue was with the wrong certificate. I know that because I found the issue in logs:
{"timestamp":1581544219.7816372,"message":"Request failed: 503: {\"description\"=>\"Instances information unavailable: hostname \\\"eirini.cf-system.svc.cluster.local\\\" does not match the server certificate\", \"error_code\"=>\"CF-InstancesUnavailable\", \"code\"=>220002, \"test_mode_info\"=>{\"description\"=>\"Instances information unavailable: hostname \\\"eirini.cf-system.svc.cluster.local\\\" does not match the server certificate\", \"error_code\"=>\"CF-InstancesUnavailable\", The debugging process is one of the main reasons why Cloud Foundry moves to Kubernetes. You can easily debug everything with the same command, you can get everything using the same patterns and you can find the missing part.
