Improving the Developer Experience on Kubernetes - The Remote Development Operator
Wed, 03 Jul 2024 14:21:32 -0000
|Read Time: 0 minutes
Introduction
In the software development world, the Developer Experience (DevX) quality can make or break the efficiency and effectiveness of both the individual programmers and the entire team. A well designed DevX has a direct impact on productivity, software quality, innovation, and overall satisfaction. At the same time, it can help keep development costs in check.
The benefits may seem self-evident to most software engineers. However, it can be challenging to convince management to invest in DevX. Interventions may involve multiple aspects of the software development lifecycle, and it’s not always easy to quantify their impact. Thankfully, within the last few years, there have been an increasing number of studies that highlight the importance and also attempt to measure the impact of elevating DevX. At the same time, software-centric companies like Google and Microsoft have been setting the standards by appointing DevX teams and DevX champions within development teams that actively seek areas of improvement.
Understanding DevX
DevX encompasses several critical aspects that contribute to a developer's ability to perform tasks efficiently and effectively:
- Automated environments and self-service—The most important aspect is self-service. Team members should be able to provision new deployments of the application tailored for development or testing. Developers thrive in environments where repetitive setup tasks are automated, and resources are readily available without waiting for manual intervention.
- Performance and reliability—A robust development environment ensures that developers can rely on their tools and infrastructure to perform consistently under various conditions.
- Rapid feedback loops and effective debugging—Quick feedback on changes and efficient debugging tools are crucial for iterative development, helping developers fix issues and iterate on features swiftly.
- Support for preferred tools—The ability to integrate and use choice tools (for example Integrated Development Environments (IDE), debuggers, and version control systems) seamlessly in the development process enhances productivity and comfort.
- Reducing cognitive load—Simplifying daily complex management tasks helps developers focus on solving problems rather than configuration and maintenance tasks of their development environments.
- Documentation and learning resources—Accessible, clear, and concise documentation supports the learning curve and daily developer operations.
- API design and usability—Well-designed tools and interfaces enhance the usability and reduce the time needed to integrate and interact with different systems.
The Challenge of Managing Development Environments
As applications grow in both size and complexity, managing development environments becomes a significant challenge. This increases friction, frustration, wait times, and costs. These hurdles can severely impact productivity, product quality, and overall satisfaction.
Some claim that the solution to this problem is to use containers and follow a microservices architecture. Therefore, each developer only works locally on a single microservice at one time.
In a perfect world, that works great! Every microservice can be developed autonomously because it has well-defined specifications. It comes with unit tests and mock servers to model all possible interactions with the external world.
In practice, things can get messier, especially when dealing with large applications that move fast. The specification might be “agile”, the tests and mocks scheduled for the next sprint, or you may need to perform debugging that requires the entire application to be deployed. Many, if not most developers need that at some point.
In this case, the developer could deploy the development application locally on their computer. This works great, as long as they have adequate resources. But, as applications grow in size and complexity, they may not fit into most developers’ laptops. The only solution is to use a remote environment for development and testing.
Remote Development Approaches
There are different approaches to leverage remote computing resources when developing a complex application.
A popular choice is fully remote development. We use the developer’s laptop as a thin client, and everything happens remotely. The developer uses a web-based IDE to connect to a remote environment where the code is cloned, and the application is deployed. This approach simplifies things since everything runs in fully homogeneous development environments in one single place. But at the same time, it sacrifices flexibility in that the developer can’t select the IDE, shell, and tooling of their choice, or easily personalize their environment.
Another option is to develop and run the microservice that is being worked on locally, while connecting it with a remote instance of the application. This can get quite complex, as the microservice needs to talk to remote components, and remote components need to talk to the microservice. The added latency, especially with databases and other such systems, can easily make the situation unworkable. Network policies and other security constraints can further complicate things. The locally deployed microservice might have a difficult time accessing the information it needs, which may be stored in Kubernetes secrets and ConfigMaps present in the remote environment.
We decided to follow a hybrid approach. We tried to get the best of both worlds and drew a line between development and deployment. The development is purely local. The code is cloned locally, the IDE runs there, and the developer has their shell, scripts, and all the tools they need on their laptop. But their laptop is connected to a remote and pre-deployed instance of the application. On every file saved, changes are synched to the remote environment, where they immediately take effect.
Our approach has several benefits. Developers are free to work with their tools of choice in their personal environment. Multiple developers can leverage the same remote environment, or a single developer can concurrently work on multiple microservices and deploy all of them to the same remote environment. There’s no need to have a distinct way to deploy the microservice locally, instead of deploying it remotely as part of the application. Any dependency on Kubernetes resources, like secrets, volumes, or ConfigMaps, will work out if the box.
The Remote Development Operator
The Remote Development Operator is a new open-source project from Dell Technologies. The operator was developed within Dell ISG Edge to support the development of Dell NativeEdge, and it can be leveraged by any application that runs on Kubernetes.
This Kubernetes operator and Custom Resource Definition (CRD) simplifies the deployment of remote development environments, enabling developers to synchronize their local changes to a remote Kubernetes cluster seamlessly.
We define a new Kubernetes custom resource named DevEnv. Each DevEnv can target one or more Kubernetes deployments or StatefulSets using label selectors. For each DevEnv, the operator creates a DNS record, a PVC for storing the code or binaries, and starts an SSH server that is configured to accept the specified SSH keypairs. Each DevEnv provides a command that can be used to configure the end user's IDE (for example, VSCode). This configuration synchronizes over the SSH local code stored on the developer's device to the PVC that is attached to the DevEnv pod. After the first sync is completed, the DevEnv gets enabled and mounts the code PVC to the target pods.
There are two supported modes of operation:
- Modify mode
In modify mode, the operator will edit the definition of the target deployments or StatefulSets, mounting the code PVC on the target path, which will override the original code or binary that's burned on the image.
2. Clone mode
In clone mode, the operator will launch a new deployment or StatefulSet, with the PVC mounted on the target path, while leaving the original deployment or StatefulSet untouched. A new service and ingress are also generated, to expose a port of the cloned deployment over HTTPS.
When working on a large application with numerous microservices, the clone mode allows multiple developers to work in parallel, leveraging the same deployed application. Still, there may be cases where the cloned DevEnv may affect the original application, for example when working on database schema changes. Developers that leverage the clone mode should coordinate with their peers to ensure they're not breaking each other’s development environments. When in doubt, use the “modify” mode as the safest option. However, it can be much more expensive since it requires a dedicated application deployment for each developer.
Security
For every new DevEnv, the operator performs the following tasks:
- Allocate a new IP address, using the configured network load balancer.
- Create DNS records pointing to it, using the configured external-dns provider.
- Configure a Kubernetes service that exposes an SSH server on the above IP address configured to accept the developer’s credentials.
- Configure an HTTPS service to expose a clone of the service under development only in clone mode.
It is highly recommended to only expose the above within a secure VPN connection. Additionally, we recommend using distinct namespaces or even virtual Kubernetes clusters to allow developers to access the Kubernetes API for their environments, without affecting each other.
Getting Started
You can install the Remote Development Operator on any Kubernetes cluster, as long as it supports the following:
- A load balancer that can dynamically allocate IP addresses for each environment.
- External DNS configured with any of the supported DNS providers to provision records within the selected domain.
- A StorageClass that supports ReadWriteMany mode. This is supported by filesystems like EFS or NFS.
Once the operator is installed, you can create resources of type DevEnv that target existing deployments of the application under development. DevEnvs should be created within the same namespace that contains the target deployments.
For more information, see the README.
Resources
- Developer Velocity: How software excellence fuels business performance
- Elevating the Developer Experience: A Forrester Study on How (and Why) to Improve DevX
- 2017 Software Developer Productivity Survey
- State of the Developer Experience 2024
- DevEx: What Actually Drives Productivity: The developer-centric approach to measuring and improving productivity
- Yes, good DevEx increases productivity. Here is the data.
- What is a developer experience team?
- DevEx Champion Actively Seek Improvements