Use Go Debugger’s Delve with Kubernetes and CSI PowerFlex
Wed, 15 Mar 2023 14:41:14 -0000
|Read Time: 0 minutes
Some time ago, I faced a bug where it was important to understand the precise workflow.
One of the beauties of open source is that the user can also take the pilot seat!
In this post, we will see how to compile the Dell CSI driver for PowerFlex with a debugger, configure the driver to allow remote debugging, and attach an IDE.
Compilation
Base image
First, it is important to know that Dell and RedHat are partners, and all CSI/CSM containers are certified by RedHat.
This comes with a couple of constraints, one being that all containers use the Red Hat UBI Minimal image as a base image and, to be certified, extra packages must come from a Red Hat official repo.
CSI PowerFlex needs the e4fsprogs package to format file systems in ext4, and that package is missing from the default UBI repo. To install it, you have these options:
- If you build the image from a registered and subscribed RHEL host, the repos of the server are automatically accessible from the UBI image. This only works with podman build.
- If you have a Red Hat Satellite subscription, you can update the Dockerfile to point to that repo.
- You can use a third-party repository.
- You go the old way and compile the package yourself (the source of that package is in UBI source-code repo).
Here we’ll use an Oracle Linux mirror, which allows us to access binary-compatible packages without the need for registration or payment of a Satellite subscription.
The Oracle Linux 8 repo is:
[oracle-linux-8-baseos] name=Oracle Linux 8 - BaseOS baseurl=http://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64 gpgcheck = 0 enabled = 1
And we add it to final image in the Dockerfile with a COPY directive:
# Stage to build the driver image
FROM $BASEIMAGE@${DIGEST} AS final
# install necessary packages
# alphabetical order for easier maintenance
COPY ol-8-base.repo /etc/yum.repos.d/
RUN microdnf update -y && \
...
Delve
There are several debugger options available for Go. You can use the venerable GDB, a native solution like Delve, or an integrated debugger in your favorite IDE.
For our purposes, we prefer to use Delve because it allows us to connect to a remote Kubernetes cluster.
Our Dockerfile employs a multi-staged build approach. The first stage is for building (and named builder) from the Golang image; we can add Delve with the directive:
RUN go install github.com/go-delve/delve/cmd/dlv@latest
And then compile the driver.
On the final image that is our driver, we add the binary as follows:
# copy in the driver COPY --from=builder /go/src/csi-vxflexos / COPY --from=builder /go/bin/dlv /
In the build stage, we download Delve with:
RUN go get github.com/go-delve/delve/cmd/dlv
In the final image we copy the binary with:
COPY --from=builder /go/bin/dlv /
To achieve better results with the debugger, it is important to disable optimizations when compiling the code.
This is done in the Makefile with:
CGO_ENABLED=0 GOOS=linux GO111MODULE=on go build -gcflags "all=-N -l"
After rebuilding the image with make docker and pushing it to your registry, you need to expose the Delve port for the driver container. You can do this by adding the following lines to your Helm chart. We need to add the lines to the driver container of the Controller Deployment.
ports: - containerPort: 40000
Alternatively, you can use the kubectl edit -n powerflex deployment command to modify the Kubernetes deployment directly.
Usage
Assuming that the build has been completed successfully and the driver is deployed on the cluster, we can expose the debugger socket locally by running the following command:
kubectl port-forward -n powerflex pod/csi-powerflex-controller-uid 40000:40000
Next, we can open the project in our favorite IDE and ensure that we are on the same branch that was used to build the driver.
In the following screenshot I used Goland, but VSCode can do remote debugging too.
We can now connect the IDE to that forwarded socket and run the debugger live:
And here is the result of a breakpoint on CreateVolume call:
The full code is here: https://github.com/dell/csi-powerflex/compare/main...coulof:csi-powerflex:v2.5.0-delve.
If you liked this information and need more deep-dive details on Dell CSI and CSM, feel free to reach out at https://dell-iac.slack.com.
Author: Florian Coulombel