Originally posted at Loft Labs Blog. © All rights belong to Loft Labs, Inc..

This is one of the posts I have written for Loft Labs, Inc..

Containers and the ecosystem around them changed how engineers deploy, maintain, and troubleshoot workloads. But debugging an application on a Kubernetes cluster can be daunting at times, as you might not find the tools you need in the container. Many engineers build containers with distroless images based on slimmed-to-the-bone distributions where there’s not even a package manager or a shell. In the deep end, some teams use scratch as a base image and only add the files the application needs to run. Some reasons why this is a common practice are:

• To have a smaller attack vector area.
• To have faster-scanning performance.
• Reduced image size.
• To have a faster build and CD/CI cycle.
• To have fewer dependencies.

These stripped-down base images don’t include the tools you would use to troubleshoot an application or its dependencies. This is the perfect opportunity for the Kubernetes ephemeral containers feature to shine. Ephemeral containers allow you to create a container image that includes all the debugging tools you might need. Once there is a need for debugging, you would deploy the ephemeral container into the running pod of your choice.

You can not add a container to a deployed pod; you need to update the spec, and resources are re-created. However, an ephemeral container can be added to an existing pod to allow you to troubleshoot a live issue.

Ephemeral containers is an alpha feature in Kubernetes 1.22, so the official recommendation is not to use it in production environments.