That’s where infrastructure auto tests shine: they validate real behavior in a real cluster and act as living specs for your platform.

In this post, we’ll walk through a practical example: a test that validates Kubernetes Cluster Autoscaler by forcing a scale-up and checking end-to-end results.

What we test: Cluster Autoscaler

This is a simple example to show the approach. The goal: prove that writing infra tests is straightforward and genuinely useful.

Cluster Autoscaler watches pending pods and scales the right node group when capacity is tight.

Why test it? What can go wrong?

• A bad autoscaler release.

• Helm chart breaking change.

• Misconfiguration.

• Incompatible cluster components.

• Access/quotas/IAM issues that block scale-up.

• Networking issues (for external autoscalers).

Our test covers the full path end-to-end:

Pending pods ──> Cluster Autoscaler ──> Node group scales ──> Pods Ready

1. Detect the target node group for the current cluster context.

2. Create a deployment with replicas set to current_nodes + 1.

3. Use pod anti-affinity, so pods spread across nodes, guaranteeing the need for an extra node.

4. Use node affinity/selector, so pods land only on the intended node group.

5. Verify we see the expected pending pod initially; then wait for all pods to be Ready within a timeout.

6. Assert success when all pods are scheduled and ready.

How the test is designed

I wrote the test in Python because it’s easy to read, most engineers already know it, and it’s usually available by default. No extra Python deps needed.

If you prefer Bash, check out the bats framework.

• Node group detection: EKS clusters use label eks.amazonaws.com/nodegroup and expect group main.

• Workload shape: A tiny deployment using registry.k8s.io/pause:3.9 with small CPU/memory requests. We add strict scheduling constraints:

  • Pod anti-affinity (requiredDuringSchedulingIgnoredDuringExecution) to force one pod per node.

  • Node selector/affinity so we only use the target node group.

• Assertions:

  • Immediately after creation, we expect exactly one pod to be Pending.

  • Within the timeout (5 minutes by default), all pods must become Ready.

Running the test

I use mise to run the test. It’s a handy tool to manage dependencies and tasks, so teammates don’t need to learn a custom Python CLI.

Run mise tasks to see what’s available. For the autoscaler test: mise test:cluster-autoscaler.

It’s easy to drop into CI: add kubectl and python to .mise.toml and run mise install up front. More details in my post about mise.

Code examples

Below are minimal snippets from the real test suite. Full code is in the repo.

# cluster_autoscaler/test.py

from utils import BaseInfraTest
# other imports

class ClusterAutoscaler(BaseInfraTest):
    RESOURCE_NAME = "infra-tests-cluster-autoscaler"
    TIMEOUT_SECONDS = 300  # 5 minutes
    DEPLOYMENT_CHECK_DELAY_SECONDS = 5

    def setUp(self):
        # prepare common things that are used in all tests
        super().setUp()

        # set internal variables
        # ...

    def _get_target_nodes(self):
        # Get all nodes and filter for target nodegroup
        # ...

    def test_cluster_autoscaler(self):
        # Get current nodes in target node group
        target_nodes, nodegroup_label_value = self._get_target_nodes()
        current_node_count = len(target_nodes)

        # Calculate required replicas (current nodes + extra to trigger scaling)
        required_replicas = current_node_count + 1

        # Read and prepare deployment manifest
        with open(self.deployment_path, encoding="utf-8") as f:
            manifest = f.read()

        # Apply deployment manifest
        self.command(
            f"kubectl --context {self.current_context} apply -f -",
            input=manifest
            % (
                self.RESOURCE_NAME,  # deployment name
                self.namespace,  # namespace
                self.RESOURCE_NAME,  # app label
                required_replicas,  # replicas count
                self.RESOURCE_NAME,  # selector matchLabels app
                self.RESOURCE_NAME,  # pod template app label
                json.dumps(
                    {self.nodegroup_label_key: nodegroup_label_value}
                ),  # nodeSelector
                self.RESOURCE_NAME,  # pod anti-affinity
            ),
        )

        # Give k8s a moment to process the deployment
        time.sleep(self.DEPLOYMENT_CHECK_DELAY_SECONDS)

        # Check that there is 1 pod in pending state
        pods_cmd = (
            f"kubectl --context {self.current_context}"
            f" --namespace {self.namespace}"
            f" get pods"
            f" -l app={self.RESOURCE_NAME}"
            f" -o jsonpath='{{.items[*].status.phase}}'"
        )
        pod_phases = self.command(pods_cmd).split()
        self.assertEqual(
            pod_phases.count("Pending"),
            1,
            f"Expected 1 pod in pending state, got {pod_phases.count('Pending')}",
        )

        # Wait for all pods to be ready using kubectl wait
        wait_cmd = (
            f"kubectl --context {self.current_context}"
            f" --namespace {self.namespace}"
            f" wait pod"
            f" -l app={self.RESOURCE_NAME}"
            f" --for=condition=Ready"
            f" --timeout={self.TIMEOUT_SECONDS}s"
        )

        self.command(wait_cmd)

Mise configuration:

# .mise.toml
[tools]
python = "3.13.0"
kubectl = "1.31.1"

[tasks.test]
description = "Run all infrastructure tests"
run = "python -m unittest discover -v"

[tasks."test:cluster-autoscaler"]
description = "Run tests for cluster-autoscaler"
run = """
python -m unittest -v cluster_autoscaler.test.ClusterAutoscaler
"""

deployment.yaml is a regular Kubernetes Deployment with %s placeholders that get templated at runtime. A full example is here.

Principles of solid infra tests

• Simple: easy to maintain and add new.

• Small: extract boilerplate into shared utilities.

• Fast: speed matters.

• Wait on conditions: wait for states, not fixed sleeps.

Well-structured tests make it easy to add new ones — even with help from LLMs. Write the test case, then ask to implement it using the existing ones as a reference.

Closing thoughts

Infrastructure tests are your early-warning system for platform regressions. By validating real behavior in real clusters, you turn assumptions into executable specs.

The Cluster Autoscaler test is a concise, low-risk example that catches issues you might otherwise notice only after a late-night surprise.

Apply the same approach to other components to keep changes safe and verified.

YAML files are everywhere in software development — think Kubernetes manifests, Ansible playbooks, or any configuration management system. But when these files contain multiple documents (separated by ---), comparing changes with traditional diff tools becomes a mess.

These tools treat the file as a single blob of text, often producing confusing or outright incorrect diffs. This problem gets worse with large YAML files packed with many documents, making it nearly impossible to spot specific changes.

That’s why I created godiffyaml, a tool designed to solve this exact issue. Whether you’re a developer, DevOps engineer, or system admin, godiffyaml makes diffing multi-document YAML files painless.

Why Standard Diff Tools Struggle

Here’s what makes diffing multi-document YAML tricky with standard tools:

1. Document Order Awareness: If a document is moved within a YAML file, standard diff tools display it as a completely new document. Any changes inside it might be overlooked.

2. Document Awareness: Tools like diff don't recognize YAML's --- separators, so changes across documents get mixed up.

3. Values Notation Awareness: According to the YAML specification, the way values are noted is quite flexible. For example, strings can be quoted or not. However, for diff tools, these differences still appear as changes.

4. Nested Structure Awareness: YAML’s hierarchical structure—nested keys and values—means small changes can drown in a flood of irrelevant differences.

I built godiffyaml to address these pain points head-on, offering a smarter way to compare multi-document YAML files.

Let's explain the issues and how godiffyaml solves them in detail.

Document Order Awareness

A basic YAML file with random Kubernetes manifests:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: default
data:
  DB_HOST: "localhost"
  DB_PORT: "5432"
  CACHE_TTL: "3600"
---
apiVersion: v1
kind: Service
metadata:
  name: frontend-svc
  namespace: default
spec:
  ports:
    - port: 80
      targetPort: 3000
  selector:
    app: frontend

Now reverse the order of the documents: place kind: Service before kind: ConfigMap and then compare the differences.

Can you quickly tell if the ConfigMap has just been moved in the YAML, or if there are additional changes? Imagine having many such "movements" inside; reviewing them would become a nightmare.

With godiffyaml, you can sort YAML files by path, such as .kind, and compare them with any standard tool.

Or just

Document Awareness

Let's change the value of PGHOST from host2 to host-changed in app2-secret within a YAML file containing two secrets, and then compare the original with the modified version.

apiVersion: v1
kind: Secret
metadata:
  name: app1-secret
  namespace: default
stringData:
  PGDATABASE: db1
  PGHOST: host1
  PGPASSWORD: pass1
  PGUSER: user1
---
apiVersion: v1
kind: Secret
metadata:
  name: app2-secret
  namespace: default
stringData:
  PGDATABASE: db2
  PGHOST: host2 # this will be changed
  PGPASSWORD: pass2
  PGUSER: user2

Can you guess which secret has changed? To find its name, you need to expand the context for the difft tool or check the full diff in your favorite diff tool. Now, imagine there are many such changes across several documents.

godiffyaml solves this problem:

See the name in a rectangle? It's templated from the -paths flag: <.kind>_<.metadata.name>.yaml.

Values Notation Awareness

Let’s compare the same YAML files, but the first one will have unquoted strings and the second one will have quoted strings. According to the YAML specification, they are considered the same, but your IDE probably doesn't see it that way.

It's just distracting noise because it's YAML with the same values.

Just like the 'No Document Order Awareness' example above, we can either use godiffyaml sort and compare with a usual tool or simply use godiffyaml diff.

But now let's use the k8s subcommand: godiffyaml k8s is a shortcut for godiffyaml diff -paths apiVersion,kind,metadata.namespace,metadata.name. Notice that the dot at the beginning of each path is optional.

Note that we don’t use the -order flag for the sort subcommand. Details are provided below in the ‘Magic’ section.

Nested Structure Awareness

Typical diff tools show changes for lines, not values. However, it's useful to see specific differences. Difftastic addresses this issue, and godiffyaml uses it behind the scenes.

Difftastic — a structural diff tool that compares files based on their syntax. It’s a great tool that shows differences only for elements that changed, not the lines.

How GoDiffYAML Works

It has two subcommands: sort and diff. Let's break them down first.

Sort subcommand

1. Reads a YAML file.

2. Sorts documents inside by YAML paths if the -order flag is provided.

3. Outputs the YAML file with all documents to stdout.

Diff subcommand

1. Reads YAML files.

2. Saves each document in a temporary directory with names based on path values.

   E.g. for paths=apiVersion,kind,metadata.namespace,metadata.name document name will be like apps_v1_Deployment_default_backend-api.yaml

3. Runs difftastic on the directories to compare them.

💫 Magic 💫

All the magic inside is based on two ideas:

1. By reading and then dumping YAML with the same Go YAML library, we ensure that YAML documents have consistent value notation and key ordering. This applies to both sort and diff modes.

2. By saving files with templated names, we can compare directories with difftastic, which will display the file names. The file names help us identify documents due to the templating.

The structural diff relies entirely on difftastic because I don't want to reinvent the wheel. So, we need difftastic to use the diff or k8s subcommands.

Additionally, any extra flags for godiffyaml are passed to the difftastic backend.

Why You’ll Love GoDiffYAML

• Saves Time: No more digging through messy diffs to find what changed.

• Reduces Errors: Clear, structured output means fewer mistakes when reviewing updates.

• Handles Scale: Works seamlessly with large, complex YAML files.

Wrap-Up

If multi-document YAML files are part of your daily grind, godiffyaml is here to make your life easier. By respecting YAML’s structure and delivering precise, readable diffs, it turns a frustrating task into a breeze. Give it a try — I think you’ll wonder how you ever managed without it!

So go diff that YAML💪

This article explores three beginner-friendly approaches — Claude Desktop, Cursor, and K9S with HolmesGPT — each designed to simplify Kubernetes management for newcomers.

The Model Context Protocol (MCP) is a powerful framework for integrating AI models into development environments. MCP enables seamless communication between AI systems and tools like Kubernetes by providing a standardized way to pass context and commands. With MCP, you can query cluster states, automate resource management, or even troubleshoot issues using natural language. To use MCP effectively with Kubernetes, you’ll need a compatible server setup that bridges your AI tool with kubectl, the Kubernetes command-line interface.

There are many MCPs for Kubernetes, each with different features and programming languages like TypeScript, Go, and Python. They are still being actively developed, so there are some bugs. They are not quite ready for serious everyday use yet.

1. https://github.com/Flux159/mcp-server-kubernetes - TypeScript. This MCP is mentioned in the Anthropic documentation and looks promising. It also has more GitHub stars than the others.

2. https://github.com/rohitg00/kubectl-mcp-server - A lot of declared functionality but currently less stable than others, Python-based.

3. https://github.com/strowk/mcp-k8s-go - Golang-based

4. https://github.com/manusa/kubernetes-mcp-server - Golang-based, but not just a wrapper around kubectl or helm—it doesn't require external dependencies.

5. https://github.com/wenhuwang/mcp-k8s-eye - Golang

In this article, I use the first two to demonstrate basic functionality.

Approach 1: Claude Desktop

In my opinion, Claude Desktop is an excellent solution for communicating with MCP servers today because of its user-friendly interface. I will use mcp-server-kubernetes to link Claude with Kubernetes.

Step 1: Configure Claude Desktop

1. Configure Claude Desktop MCP Settings: go to Settings => Developer menu to add MCP server or just edit ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) with:

{
  "mcpServers": {
    "kubernetes": {
      "command": "npx",
      "args": ["mcp-server-kubernetes"]
    }
  }
}

2. Save and restart Claude.

Step 2: Chat with Your Cluster

Open a chat in Claude and ask something like:

• “List deployments in default k8s ns”

• “Show me nginx logs”

See full example conversation here.

Claude uses mcp-server-kubernetes to handle the technical details, making Kubernetes approachable for beginners by translating plain English into actions.

Approach 2: Cursor IDE

Cursor is a VS Code fork designed to be a more AI-focused code editor. While it may not be as convenient as Claude, if you spend a lot of time in an IDE, it can be useful to have Kubernetes access with natural language right in the same window.

Step 1: Install mcp server

Now I will use kubectl-mcp-server. You can also use the MCP server from the previous approach or any other from the list of MCPs mentioned earlier—they are interchangeable.

pipx install kubectl-mcp-tool

Step 2: Configure Cursor

1. Go to Cursor => Settings => Cursor Settings => MCP.

2. Click "Add new global MCP server".

3. Add:

{
  "mcpServers": {
    "kubernetes": {
      "command": "~/.local/pipx/venvs/kubectl-mcp-tool/bin/python",
      "args": [
        "-m",
        "kubectl_mcp_tool.minimal_wrapper"
      ]
    }
  }
}

Step 3: Interact with Your Cluster

Open chat in Cursor and use prompts like:

• “List all pods in default k8s ns”

• “What is the status of my nginx k8s deployment?”

Approach 3: Using K9s with HolmesGPT

HolmesGPT, from Robusta, is a tool that simplifies Kubernetes troubleshooting. It investigates issues automatically, requiring no prior expertise. Use it via the Robusta SaaS platform or CLI with queries like holmes ask "what pods are unhealthy and why?".

We can integrate it with k9s, a terminal-based UI that allows you to interact with your Kubernetes clusters. It's a popular tool for working with Kubernetes, making it easier to navigate, observe, and manage deployed applications.

Actually, this approach is different from the previous ones. It doesn’t use MCP. However, it is powerful, so I have to mention it.

Step 1: Configuring K9s with HolmesGPT

1. Install HolmesGPT (instructions)

2.   # for mac
     brew tap robusta-dev/homebrew-holmesgpt
     brew install holmesgpt
   

3. Export OpenAI API key, for example with ~/.zshrc: export OPENAI_API_KEY=XXX

   I bet you can easily Google how to get an API key. Certainly, you can use other AIs.

4. Add Holmes as plugin to k9s: edit ~/Library/Application Support/k9s/plugins.yaml (mac). See detailed instructions here.

plugins:
  holmesgpt:
    shortCut: Shift-H
    description: Ask HolmesGPT
    scopes:
      - all
    command: bash
    background: false
    confirm: false
    args:
      - -c
      - |
        holmes ask "why is $NAME of $RESOURCE_NAME in namespace $NAMESPACE not working as expected?"
        echo "Press 'q' to exit"
        while : ; do
        read -n 1 k <&1
        if [[ $k = q ]] ; then
        break
        fi
        done
  custom-holmesgpt:
    shortCut: Shift-Q
    description: Custom HolmesGPT Ask
    scopes:
      - all
    command: bash
    background: false
    confirm: false
    args:
      - -c
      - |
        INSTRUCTIONS="# Edit the line below. Lines starting with '#' will be ignored."
        DEFAULT_ASK_COMMAND="why is $NAME of $RESOURCE_NAME in namespace $NAMESPACE not working as expected?"
        QUESTION_FILE=$(mktemp)

        echo "$INSTRUCTIONS" > "$QUESTION_FILE"
        echo "$DEFAULT_ASK_COMMAND" >> "$QUESTION_FILE"

        # Open the line in the default text editor
        ${EDITOR:-nano} "$QUESTION_FILE"

        # Read the modified line, ignoring lines starting with '#'
        user_input=$(grep -v '^#' "$QUESTION_FILE")
        echo running: holmes ask "\"$user_input\""

        holmes ask "$user_input"
        echo "Press 'q' to exit"
        while : ; do
        read -n 1 k <&1
        if [[ $k = q ]] ; then
        break
        fi
        done

4. Start K9s, select pod and ask Holmes with Shift-H or Shift-Q

HolmesGPT can do much more. Check the repository for more details. Also, there are other tools, like k8sgpt, that can make the life easier. Maybe some of them can be integrated with Lens too.

Why This Matters for Beginners

These tools democratize Kubernetes:

• Claude: A desktop app where anyone can chat with their cluster.

• Cursor: An IDE that lets developers code and manage k8s without expertise.

• K9s: A visual tool with AI to troubleshoot effortlessly.

Needless to say, all of this is not meant for production use. Moreover, MCP servers are still full of bugs and have limited functionality. However, it can still be useful for local development.

Conclusion

Claude Desktop, Cursor, and K9s with HolmesGPT empower beginners to manage Kubernetes clusters with AI, no solid K8s experience required. They turn tasks into simple conversations, making DevOps accessible to all. Try them out and start exploring Kubernetes!

Additionally, automating routine tasks with the codebase is helpful, so some form of task management is also necessary.

Background

As always, there are many tools available to solve these problems. For example, to create a development environment, there are:

1. Language-specific tools

   1. Node.js: nvm / n

   2. Python: venv / pyenv / poetry / conda

   3. Ruby: rbenv

   4. Java: jenv

   5. Infrastructure tools:

      1. tfenv for Terraform

      2. kbenv for kubectl

      3. helmenv for Helm

   6. Environment variables: direnv

2. Language-agnostic tools

   1. asdf - popular all-in-one solution

   2. nix - it’s better to have a lot of free time, lol

   3. or even devcontainers for VS Code: the docker-way

As we can see, there are many tools available 🤯, and there are even more for task management. While we can use make, it's not very user-friendly. So, there are many alternatives to make. Trust me, I've tried most of them

1. task

2. just

3. Earthly - docker inside 🤟

4. bake

5. run - last commit year ago

6. makesure - weird syntax

7. foy - if we love js

8. mmake - make on steroids

9. robo - last commit 2 years ago

10. redo - last commit 3 years ago

11. Or even buck2 / bazel / pants / please if we already have it in the project

I actually really like the first three.

Introducing Mise: Development Tools and Tasks in One App

Mise, inspired by asdf — the multiple runtime version manager, can use asdf’s repository with hundreds of tools as its successor. However, mise goes further:

1. It supports several other "backends" to install tools outside the built-in repository.

2. It has a simpler CLI.

3. Tasks!

Define a Toolset

Mise is set up using a single file called .mise.toml. Here's an example:

# .mise.toml
[tools]
terraform = "1.9"
terramate = "0.9"
pre-commit = "3"
awscli = "2"
"pipx:detect-secrets" = "1.4"
"go:github.com/containerscrew/tftools" = "0.9.0"

I can install it with a single command: mise install, see a demo gif in the next sections. Let me explain the contents.

Our team uses this file for the infrastructure repository with Terraform and Terramate manifests. Terraform requires awscli to work with the AWS API provider.

detect-secrets is a tool used in a pre-commit hook to ensure no secrets are accidentally exposed to git. This tool is installed with pipx, and mise supports this backend out of the box.

tftools is a tool that summarizes changes in Terraform plans. It's useful if you want to review plans for several environments or stacks at once. As we can see, it uses the Go backend to fetch a binary from the repository.

There are many other backends: asdf, cargo, go, npm, pipx, spm, ubi, vfox. Honestly, I'm not sure what spm and vfox are 🙂, but ubi is The Universal Binary Installer, which lets you install any binary from a tool’s GitHub release page.

Need to add more tools? Edit the file or run mise use TOOL_NAME. To see the built-in tool registry, run mise registry. For tools not listed there, we can use the full notation, as I did with tftools above.

Manage Environments

By default, mise installs a tool not system-wide. This means we can have different tool versions for different directories aka projects. If you prefer a system-wide installation, you can configure it by placing the settings in ~/.config/mise/config.toml, making tools and tasks (see below) available in any directory.

Mise supports nested configurations. To find out which configuration provides a tool or task, run mise ls.

For example, you can have different Python versions for different projects. Mise determines which Python version to use based on the nearest .mise.toml file. It even supports automatic virtualenv activation.

You can also set different environment variables for different directories, similar to direnv.

Prepare Dev Env

1. brew install mise or use alternative installation methods

2. Activate mise in your shell. Zsh example:

    echo 'eval "$(~/.local/bin/mise activate zsh)"' >> ~/.zshrc
    # restart shell or `source ~/.zshrc`
   

3. mise install in a dir with .mise.toml file
   or use my VSCode extension

So, add this simple instruction to the repo's README.md, place .mise.toml in the root of the repo, and your teammates can simply run mise install to get the same toolset. Make sure to pin tool versions to ensure consistency. 🙂

Also, mise is cross-platform, so people using Linux or Mac will follow the same instructions. No more juggling with brew, apt, or yum.

Run Tasks

Let's define several tasks to make daily routines easier with that infrastructure repo. To run Terraform, *.tf files should be generated with Terramate. To create a plan, the tf-state must be initialized. To initialize a state, it's best to be logged into AWS. That's a lot to keep track of.

Here's a quick demo:

We can use mise tasks to make the process easier. I've removed the Terramate and SSO details to keep the example brief:

# .mise.toml

[tasks."tf.init"]
alias = "tfi"
description = "`terraform init`"
run = "terraform init"

[tasks."tf.validate"]
alias = "tfv"
depends = ["tf.init"]
description = "`terraform validate`"
run = "terraform validate"

[tasks."tf.plan"]
alias = "tfp"
depends = ["tf.init"]
description = "`terraform plan`"
run = """
#!/usr/bin/env bash

# can use args for this task
terraform plan -out=tfplan $@ 
# ring terminal when complete
tput bel 
"""

[tasks."tf.summarize"]
alias = "tfs"
description = "`tftools summarize` with pre-generated plan file (terraform plan should be generated in advance)"
run = """
#!/usr/bin/env bash

terraform show -json tfplan > plan.json
tftools summarize < plan.json
"""

[tasks."tf.apply"]
alias = "tfa"
description = "`terraform apply` with pre-generated plan file (terraform plan should be generated in advance)"
run = """
#!/usr/bin/env bash

cmd="terraform apply tfplan $@"
read -p "$cmd\n\nAre you sure? (Y/n) " choice
[ "$choice" = "n" ] || ($cmd; tput bel)
"""

[tasks."tf.lock"]
depends = ["tf.init"]
description = "`terraform providers lock` with predefined platform list"
run = "terraform providers lock -platform=linux_amd64 -platform=darwin_amd64 -platform=darwin_arm64 ; tput bel"

[tasks."tf.console"]
alias = "tfc"
depends = ["tf.init"]
description = "`terraform console`"
raw = true
run = "terraform console"

To run plan, use mise run tf.plan. Mise has aliases, so we can use mise run tfp. The shell also supports aliases 🙂. So why not create an alias with alias mr="mise run" and just use mr tfp.

If you use VSCode, check out my VSCode extension to run tasks for your workspace directly from the IDE.

This is a simple, straightforward flow for Terraform. To see the full example I use daily, you can check this gist.

Clean and beautiful syntax, in my opinion.

Makefile is for C developers, while TOML is for humans 😊

List all available tasks, and you'll get a nice table with descriptions:

$ mise tasks

Name          Description                               Source
tf.apply      `terraform apply` with pre-generated pl…  ~/.config/mise/config.toml
tf.console    `terraform console`                       ~/.config/mise/config.toml
tf.init       `terraform init`                          ~/.config/mise/config.toml
tf.lock       `terraform providers lock` with predefi…  ~/.config/mise/config.toml
tf.plan       `terraform plan`                          ~/.config/mise/config.toml
tf.summarize  `tftools summarize` with pre-generated …  ~/.config/mise/config.toml
tf.validate   `terraform validate`                      ~/.config/mise/config.toml

CI/CD Pipelines

Tired of writing boilerplate code to install necessary tools for pipelines? With .mise.toml, why not use mise install in pipelines too? Install mise in advance or use the Docker image jdxcode/mise.

There's no need to track tool versions separately for development environments and CI anymore. You no longer need to remember to update tool versions in different places to prevent any issues. Mise serves as a single source of truth.

If you use a similar task flow in CI as you do locally, you can run the same mise tasks there too. If not, you can use mise profiles to define CI tasks. Bonus: you can easily use it locally to debug pipeline issues.

See the docs for a GitHub Actions example.

Best Practices

1. Place .mise.toml in the root of your Git repository to define tools and tasks for the entire repository.

2. If a repository contains several different projects (like a monorepo), keep common items (like pre-commit tools) in the root config and project-specific items in the project directories.

3. Use your own ~/.config/mise/config.toml for personal automation tasks.

Conclusion

Mise is a unified tool and task management solution that simplifies the process of installing and managing tools for different projects. It supports various backends for tool installation and allows users to define tasks with custom scripts.

Mise can be used in both local development environments and CI/CD pipelines, serving as a single source of truth for tool versions and task flows.

Mise is an excellent alternative to make, tfenv, direnv, and whatever-else-env. Try it!

Let's address this issue and expand the knowledge base to include detailed debugging methods.

Terraform Console

The built-in Terraform feature can make your life much simpler. It's often been avoided by engineers and is not so popular in blogs.

What can you do with the console? Basically, it can be used not only to show info about a resource in a state.

1. Drop state info

When you run terraform console in a Terraform project directory, it tries to use the state information to get actual details. It's useful for getting information about created resources or data objects.

But it sometimes slows down the debugging process. For example, if you place your state in an S3 bucket, it fetches it first. Tools like terraform-repl run terraform console under the hood for every command, making the process really slow. Also, console sets a lock on the state while you work with it.

To work with console without an S3 state, you can comment out the Terraform backend section, remove the .terraform folder, and rerun terraform init.

2. Test your expressions

Often we have to convert data between various formats to meet the requirements of module interfaces or for other cases. And we usually do it blindly, evaluating complicated expressions in our minds only. That leads to unexpected issues in some corner cases.

Just test it:

> coalesce(["", "b"]...)
b

Yes, it's a simple example from the tf docs, but you can use console for much more complicated things.

Did you know that terraform console can be launched not only in a directory with a terraform project? To test expressions, you can start it in any directory.

3. Define your own locals interactively

One of the common drawbacks of terraform console is that it doesn't allow you to set your own variables or locals. To compose a complicated expression, it's handy to use intermediate local variables.

To make it possible, you can use terraform console wrappers such as terraform-repl. It also adds a tab-completion feature.

> local.a=1
> local.a+1
2

By using the local command, you can display all local variables.

There is another tool called tfrepl with similar functionality. However, it doesn't have tab-completion, and it can't show all of your defined locals.

4. Debug your modules

Modules are the way to keep your tf code DRY. But it's a pain in the neck when you debug and refactor it. Use console tools to make it simpler.

If you have default values for your variables defined, just run terraform console or terraform-repl to debug expressions.

If you have no default values, it's safe to place terraform.tfvars just inside the module folder. Terraform will ignore these vars when this module is used somewhere in a parent project.

5. Test your modules

The practice of testing Terraform code is not widely used in the infrastructure world. Actually, I've never seen tests in any single company :)

Personally, I don't see any significant benefit from tests in the way they are meant to be used. See this example from the official docs: the test checks if the bucket's name is created as expected.

Do we really need it? We just use the variable in the bucket name, so it's obvious that nothing will happen to it later!

But we can use tests to make sure that we will not break a module's 'interface' sometime in the future while refactoring. Just write a test and do anything with local expressions and variables.

Stupid simple test example:

# main.tftest.hcl
run "test" {
  command = plan
  assert {
    condition = jsonencode(local.users) == jsonencode({
      "john" = {
        "grants" = null
        "login"  = true
        "member_of" = [
          "admin",
        ]
        "password_enabled" = false
      }
    })
    error_message = "Wrong format of local.users : ${jsonencode(local.users)}"
  }
}

❯ terraform test
main.tftest.hcl... in progress
  run "test"... pass
main.tftest.hcl... tearing down
main.tftest.hcl... pass

Success! 1 passed, 0 failed.

Again, if you already have default values in your variables.tf, you are ready to use tests. If you don't, place terraform.tfvars as stated above.

I don't recommend placing variables inside *.tftest.hcl in simple cases because you will not be able to use those variables with console later. Also, avoid complicated tests. Simple tests are easier to support. So, there is a chance that they will be supported in the future at least.

Testing expressions is the way to be confident that you will have the expected value in your resource property.

Conclusion

I shared a few tricks on debugging Terraform expressions. Do you have any to add? Share them in the comments.

Below, I assume the use of the official Atlantis Helm chart for deployment.

1. Add Terramate binary

Add the following to the Atlantis chart's values.yaml to download the binary and mount it to the Atlantis pod. This allows Atlantis to use it for generating Terraform code.

initContainers:
  - args:
      - >-
        curl -L https://github.com/terramate-io/terramate/releases/download/v${TERRAMATE_VERSION}/terramate_${TERRAMATE_VERSION}_linux_x86_64.tar.gz | tar xz
    command:
      - sh
      - -c
    env:
      - name: TERRAMATE_VERSION
        value: 0.4.5
    image: curlimages/curl
    name: get-terramate
    volumeMounts:
      - mountPath: /home/curl_user/
        name: terramate
    workingDir: /home/curl_user/
  extraVolumes:
    - name: terramate
      emptyDir: {}
  extraVolumeMounts:
    - name: terramate
      mountPath: /usr/local/bin/terramate
      subPath: terramate
      readOnly: true

2. Use server-side config

Let's use server-side configuration for our repository. Therefore, add more values to the values.yaml:

environment:
  ATLANTIS_REPO_CONFIG: /etc/atlantis/server-side-config.yaml
extraVolumes:
  - name: atlantis-server-side-config
    configMap:
      name: atlantis-server-side-config
extraVolumeMounts:
  - name: atlantis-server-side-config
    mountPath: /etc/atlantis/server-side-config.yaml
    subPath: server-side-config.yaml
    readOnly: true

3. Deploy server-side config

Now, place the server-side configuration in a configMap and deploy it to the Atlantis namespace:

apiVersion: v1
kind: ConfigMap
metadata:
  name: atlantis-server-side-config
  labels:
    # I deploy it with the Atlantis chart, so here is helm templating
    app: {{ .Chart.Name }}
    chart: {{ .Chart.Name }}-{{ .Chart.AppVersion }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
data:
  server-side-config.yaml: |
    repos:
    # this steps run before every workflow. 
    # So we will generate tf-code here
    - pre_workflow_hooks:
        # to run 'safeguards' terramate wants origin master 
        #   and some previous commits
        - run: |
            git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master
            git fetch --depth=1 origin master
            git fetch --depth=2
          description: Fetch origin master branch

        # this step is optional
        # I use .tm-run-order later to made Atlantis 
        #   to run plan/apply in a specified order
        # Also, I generate an atlantis.yaml config 
        #   with terramate too, so I exclude it from run order: 
        #   it's just a yaml, not tf-code
        - run: terramate list --run-order -c --no-tags atlantis -B origin/master > .tm-run-order
          description: Get changed stacks

        # And the obvious final step =)
        - run: terramate generate
          description: Generating tf code
      # some other options, offtopic
      id: github.com/XXX/YYY
      apply_requirements: [mergeable, approved, undiverged]
      delete_source_branch_on_merge: true

4. (Optional) Generate atlantis.yaml with Terramate too

Add this 'stack' to your Terramate repository:

stack {
  name        = "atlantis"
  description = "atlantis"
  id          = "your-uniq-id"

  tags = [
    "atlantis"
  ]
}

generate_file "atlantis.yaml" {
  condition = (
    # I place it in the root repo dir so I want to avoid 
    #   file generation with child stacks, only with
    #   atlantis stack and with not empty .tm-run-order
    terramate.stack.name == "atlantis" && tm_length(let.run_order) > 0
  )

  lets {
    run_order = tm_try(
      tm_compact(tm_split("\n", tm_trim(tm_file(".tm-run-order"), "\n"))),
    [])
    # dirty magic to fill config options by my folder structure
    projects = [for x in let.run_order : {
      name = tm_replace(x, "/^terraform/(projectX/)?/", "")
      dir  = x
      autoplan = {
        # I want to trigger Atlantis for every project
        #   because I already know that every project 
        #   here is modified
        when_modified = (
          tm_substr(x, 0, 14) == "terraform/projectX/" ? ["../../../**/*"] : ["../**/*"]
        )
        enabled = false
      }
    }]
    config = {
      version        = 3
      automerge      = true
      parallel_plan  = false
      parallel_apply = false
      projects       = let.projects
    }
  }

  content = tm_yamlencode(let.config)
}

Done! Now, for every PR, Atlantis will:

1. Run pre-workflow hooks to generate Terraform code.

2. (Optional) Generate its own configuration to run projects in the desired order.

If you're looking for more tips and useful info, definitely swing by my blog post 10 Useful Aliases and Functions for Terramate and Zsh Users.

Solve annoying things

1. Too long to type

Obvious one: terraform and terramate commands are too long to type :)

Use aliases:

# ~/.zshrc
alias tf="terraform "
alias tm="terramate "
alias tmg="terramate generate "

You still have to type terraform after tm run because tm knows nothing about your aliases. But it will be solved below too.

2. Command chaining

Often, to execute a Terraform command, you need to run a preceding command. For example, before tf init, you should run tm generate. Similarly, before plan, you might need to run init, and so on.

You could use Makefiles or the terramate script run feature that comes with Terramate to handle command chaining. However, this only addresses the issue of chaining commands. Plus, you'd have to add these configurations to every repository you work with.

And it's still too much typing. You'll likely resort to using aliases anyway. So, why not start with shell configuration right from the start?

Zsh functions can help:

# ~/.zshrc
function tmi () {
  terramate generate && \
  terramate run terraform init $@
}
function tmp () {
  tmi $1 && \
  terramate run terraform plan -out=tfplan $@
}

3. Tags for stacks

Tags are a fantastic feature of Terramate, especially if you're managing many stacks. They allow all tm run commands to be executed in stack directories selected by tags. However, their usability could be better—you always need to type them somewhere in the middle of a tm command.

For example, if you want to check the plan for the dev environment and then for the stage environment:

terramate run --tags=mngt:dev terraform plan
terramate run --tags=mngt:stage terraform plan

To create the last command, you can use a sequence of keys like this before typing stage: up, option+left, option+left, left, esc, backspace.

Make functions smarter:

# ~/.zshrc
function tmi () {
  terramate generate && \
  terramate run --tags=$1 terraform init ${@:2}
}
function tmp () {
  tmi $1 && \
  terramate run --tags=$1 terraform plan -out=tfplan ${@:2}
}

Now, you have fewer steps before you start typing stage: up, esc, backspace.

Additionally, you can specify plan options as the second or later arguments thanks to ${@:2} in the function.

4. Not sure about what you've typed?

Print the command before executing it:

# ~/.zshrc
function tmr () {
  print -z "terramate run --tags=$@ "
}

So tmr dev:asd ls + enter will lead to terramate run --tags=dev:asd ls in the command line.

Let's put it all together

# ~/.zshrc
alias tf="terraform "
alias tm="terramate "
alias tmg="terramate generate "
function tmi () {
  terramate generate && \
  terramate run --tags=$1 terraform init ${@:2}
}
function tmv () {
  tmi $1 && \
  terramate run --tags=$1 terraform validate ${@:2}
}
function tmp () {
  tmi $1 && \
  terramate run --tags=$1 terraform plan -out=tfplan ${@:2}
}
function tma () {
  print -z "terramate run --tags=$1 terraform apply tfplan ${@:2}"
}
function tmpl () {
  tmi $1 && \
  terramate run --tags=$1 terraform providers lock -platform=linux_amd64 -platform=darwin_amd64 -platform=darwin_arm64 ${@:2}
}
function tmc () {
  tmi $1 && \
  terramate run --tags=$1 terraform console ${@:2}
}
function tmr () {
  print -z "terramate run --tags=$@ "
}

Usage

Examples

tmi admin        # will run `terramate run --tags=admin terraform init`
tmp dev:infra    # will run `terramate run --tags=dev:infra terraform plan -out=tfplan`
tmr admin        # will print `terramate run --tags=admin` in prompt so you can run any command in the admin stack dir
tmr admin ls -la # will print `terramate run --tags=admin ls -la` in prompt, press 'enter' to execute `ls -la` in the admin stack dir or append more args

Shortcuts description

Bonus

Use different environment variable values for different stacks.

For example, if stacks use different AWS accounts, place the account name in the stack's globals and add it to your repository's root tm-file.

# repo root terramate.tm
terramate {
  config {
    run {
      env {
        AWS_PROFILE = "${global.aws_profile}"
      }
    }
  }
}

tmr aws aws sso login FTW 🤟

To get a manifest from k8s in any supported API version you can use an extended kubectl get notation like kubectl get deployments.v1beta1.extensions mydeploy -o yaml

Briefly and with examples

Everybody knows how to get a resource manifest from Kubernetes. But do you know that you can put a manifest with one apiVersion set and get the same resource manifest with another apiVersion back?

Imagine

1. You have a deployment in a cluster, that uses api-version extensions/v1beta1 from a git repo.

2. You've updated the cluster (1.15 => 1.16). Old deployment works fine, but you can't deploy nothing new with the old manifest because in k8s 1.16 api-version extensions/v1beta1 is absent.

3. You have two options:

   1. Rewrite it manually or

   2. Get it from the cluster in updated spec format (apps/v1)

To get manifest from k8s you can:

below there is an example for v1.15, where extensions/v1beta1 is not removed yet

# from extensions
kubectl get deployments.extensions ext -o yaml
# from extensions, but for v1beta1
kubectl get deployments.v1beta1.extensions ext -o yaml
# from apps
kubectl get deployments.apps ext -o yaml
# from apps, but v1
kubectl get deployments.v1.apps ext -o yaml

Notice that if you do just kubectl get deployments ext -o yaml, you will get a manifest from extensions/v1beta1 nevertheless you've even applied apps/v1 before. Details here.

Prove it

1. Start 1.15

    minikube start --kubernetes-version=v1.15.12
   

2. Make 2 manifests for deployments with different versions. Note the absence of spec.selector in extensions/v1beta1

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      creationTimestamp: null
      labels:
        run: apps
      name: apps
    spec:
      replicas: 1
      selector:
        matchLabels:
          run: apps
      template:
        metadata:
          creationTimestamp: null
          labels:
            run: apps
        spec:
          containers:
          - image: nginx
            name: apps
    ---
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      labels:
        run: ext
      name: ext
    spec:
      replicas: 1
      template:
        metadata:
          creationTimestamp: null
          labels:
            run: ext
        spec:
          containers:
          - image: nginx
            name: ext
   

3. Apply

    ❯ minikube kubectl -- apply -f .
    deployment.apps/apps created
    deployment.extensions/ext created
   

4. Check that resources apps and ext are in extensions/v1beta1 and in apps/v1 too

   • The hard way - curl:

    ❯ minikube kubectl -- proxy
    Starting to serve on 127.0.0.1:8001
    ❯ curl -s 127.0.0.1:8001/apis/extensions/v1beta1/namespaces/default/deployments/apps | yq . --yaml-output
    kind: Deployment
    apiVersion: extensions/v1beta1
    # ...
    ❯ curl -s 127.0.0.1:8001/apis/extensions/v1beta1/namespaces/default/deployments/ext | yq . --yaml-output
    kind: Deployment
    apiVersion: extensions/v1beta1
    # ...
    ❯ curl -s 127.0.0.1:8001/apis/apps/v1/namespaces/default/deployments/apps | yq . --yaml-output
    kind: Deployment
    apiVersion: apps/v1
    # ...
    ❯ curl -s 127.0.0.1:8001/apis/apps/v1/namespaces/default/deployments/ext | yq . --yaml-output
    kind: Deployment
    apiVersion: apps/v1
    # ...

• Or kubectl:

    # from extensions
    kubectl get deployments.extensions ext -o yaml
    # from extensions, but v1beta1
    kubectl get deployments.v1beta1.extensions ext -o yaml
    # from apps
    kubectl get deployments.apps ext -o yaml
    # from apps, but v1
    kubectl get deployments.v1.apps ext -o yaml

Bonus: kubectl explain

If you do kubectl explain deployment than (surprise!) you'll get a description for extensions/v1beta1. Because kubectl explain works the same way, just like kubectl get:

If you want a specific version, use an --api-version flag :

minikube kubectl -- explain deployment.spec --api-version apps/v1
minikube kubectl -- explain deployment.spec --api-version extensions/v1beta1