If you've been running Argo CD on EKS, you know the drill — install the helm chart, manage the controllers, handle upgrades, configure SSO, worry about HA, and repeat for every cluster. It's a lot of operational overhead for a tool that's supposed to simplify your life.

AWS recently introduced EKS Capabilities, and one of the first capabilities available is a fully managed Argo CD. This means AWS takes care of running, scaling, and upgrading Argo CD controllers in their own service accounts — outside of your cluster. You don't install anything on your worker nodes.

The Argo CD software runs in the AWS control plane, not on your worker nodes.

Sounds interesting? Let's walk through the setup using Terraform.

What Changes with EKS Capability for Argo CD?

Before this, self-managing Argo CD meant:

• Installing and maintaining Argo CD controllers on your cluster

• Configuring SSO separately (Dex, OIDC, etc.)

• Managing HA, scaling, and upgrades yourself

• Setting up IRSA or cross-account IAM for multi-cluster deployments

With EKS Capabilities, all of that is handled by AWS. The key highlights:

• Argo CD runs in AWS-managed service accounts, not your worker nodes

• Native integration with AWS Identity Center (formerly SSO) for user management — no more Dex configs

• Simplified multi-cluster access using EKS Access Entries

• Native integrations with ECR, Secrets Manager, and CodeConnections

User management is completely taken out of your hands and integrated with AWS Identity Center. If you've been struggling with Argo CD RBAC + SSO, this is a relief.

The Catch with Terraform

Though we have the terraform-aws-modules/eks module supporting this capability, it does not work completely out of the box for a fresh EKS setup. There are a few manual steps and dependencies you need to wire together.

In this blog, I'll walk through the setup and how to get it running quickly.

Step 1: Create the Argo CD Capability

Using the EKS capability sub-module from terraform-aws-modules/eks:

module "argocd" {
  source  = "terraform-aws-modules/eks/aws//modules/capability"
  version = "~> 21.16"

  name         = "${local.cluster_name}-argocd"
  cluster_name = module.eks.cluster_name
  type         = "ARGOCD"

  configuration = {
    argo_cd = {
      aws_idc = {
        idc_instance_arn = "arn:aws:sso:::instance/<YOUR_IDC_INSTANCE_ID>"
      }
      namespace = "argocd"
      rbac_role_mapping = [{
        role = "ADMIN"
        identity = [{
          id   = data.aws_identitystore_group.argocd_admin.group_id
          type = "SSO_GROUP"
        }]
      }]
    }
  }

  iam_policy_statements = {
    ECRRead = {
      actions = [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
      ]
      resources = ["*"]
    }
  }

  tags = local.additional_tags
}

A few things to note here:

• idc_instance_arn — this is your AWS Identity Center instance ARN. Make sure Identity Center is configured in your account before proceeding.

• rbac_role_mapping — maps your Identity Center groups to Argo CD RBAC roles (ADMIN, VIEWER). This replaces the traditional Argo CD RBAC ConfigMap approach.

• The iam_policy_statements block grants ECR read access so Argo CD can pull images and Helm charts from your private registries.

Step 2: Configure EKS Access Entry for Argo CD

This is a critical step. The capability needs cluster-level access to manage deployments. We wire this up using EKS Access Entries:

resource "aws_eks_access_entry" "argocd" {
  cluster_name  = module.eks.cluster_name
  principal_arn = module.argocd.iam_role_arn
  type          = "STANDARD"
}

resource "aws_eks_access_policy_association" "argocd" {
  cluster_name  = module.eks.cluster_name
  principal_arn = module.argocd.iam_role_arn
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"

  access_scope {
    type = "cluster"
  }

  depends_on = [aws_eks_access_entry.argocd]
}

The AmazonEKSClusterAdminPolicy provides full cluster-admin access. This is fine for getting started, but for production consider scoping down with custom Kubernetes RBAC bindings.

If you've read my earlier blog on EKS Access Entries, this pattern should feel familiar.

Step 3: Register Your Cluster (Mandatory!)

Here's where most people get stuck. The Argo CD capability does not automatically register the local cluster. You must explicitly register it as a deployment target.

This is done by creating a Kubernetes Secret in the argocd namespace:

apiVersion: v1
kind: Secret
metadata:
  name: local-cluster
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
stringData:
  name: in-cluster
  server: arn:aws:eks:<REGION>:<ACCOUNT_ID>:cluster/<CLUSTER_NAME>
  project: default

Points to note:

• Use the EKS cluster ARN in the server field, not the Kubernetes API server URL. The managed capability requires ARNs to identify clusters.

• kubernetes.default.svc is not supported here.

• This step depends on the access policy association from Step 2 being completed first.

Apply it:

kubectl apply -f local-cluster.yaml

Once registered, the cluster will show in an Unknown connection state until you create your first application — that's expected behavior.

Step 4: Deploy Your Applications

Now you're ready to create Argo CD Applications and Projects. Here's a quick example:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/<YOUR_ORG>/<YOUR_REPO>.git
    targetRevision: HEAD
    path: k8s/
  destination:
    name: in-cluster
    namespace: my-app
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Use destination.name with the cluster name you registered (like in-cluster). The destination.server field also works with EKS cluster ARNs, but using names is cleaner.

Things to Keep in Mind

• AWS Identity Center is mandatory — local users are not supported. Make sure IDC is configured with the right groups before setting up the capability.

• Cluster registration is not automatic — don't skip Step 3, or your deployments won't have a target.

• Access Entry dependency — the cluster secret registration depends on the access policy being in place. Terraform depends_on is your friend here.

• Production RBAC — scope down from AmazonEKSClusterAdminPolicy to least-privilege custom roles for production workloads.

This is a solid step forward from AWS in reducing the operational burden of running GitOps tooling. No more managing Argo CD upgrades, HA configs, or SSO integrations manually. It just works as part of your EKS cluster lifecycle.

Happy GitOps-ing!

References:

https://docs.aws.amazon.com/eks/latest/userguide/argocd.html

https://docs.aws.amazon.com/eks/latest/userguide/argocd-register-clusters.html

https://aws.amazon.com/blogs/containers/deep-dive-streamlining-gitops-with-amazon-eks-capability-for-argo-cd/

https://github.com/terraform-aws-modules/terraform-aws-eks

If you've spent any time managing data pipelines in the real world, you know the pain of deploying changes manually — copy-pasting notebooks, praying nothing breaks in prod, and maintaining that one giant script nobody dares to touch. I've been there, and I'll be honest: it's not fun.

At some point, we decided enough was enough. We adopted Databricks Asset Bundles (DAB) to bring the same software engineering rigour we apply to application code — version control, code review, automated testing, and environment promotion — to our Databricks workloads.

This post walks through exactly how we built a fully automated CI/CD pipeline using DAB and Azure DevOps, and some of the hard-won lessons we picked up along the way.

What Are Databricks Asset Bundles?

Databricks Asset Bundles are a YAML-based configuration framework for defining and deploying Databricks resources — Delta Live Tables (DLT) pipelines, jobs/workflows, SQL warehouses, and permissions — as code. Think of it as Terraform, but purpose-built for Databricks.

Resources are declared in .yml files and deployed via the databricks CLI. This makes them version-controllable, reviewable, and CI/CD-friendly right out of the box. Once you start using it, going back to manual deployments feels unthinkable.

Our Bundle Structure

Here's how our project is laid out. The folder structure itself tells a story — environments, jobs, pipelines, and warehouses each have their own space, all orchestrated from a single root bundle file:

<your-project>/
├── databricks.yml              # Root bundle definition & targets (dev/test/prod)
├── azure-pipelines.yml         # Azure DevOps CI/CD pipeline
└── resources/
    ├── environments/
    │   ├── dev.yml             # Dev-specific variables (catalog, schema, storage)
    │   └── prod.yml            # Prod-specific variables
    ├── jobs/
    │   ├── wf_<domain_a>.yml   # Workflow: triggers a single DLT pipeline
    │   ├── wf_<domain_b>.yml   # Workflow: orchestrates multiple DLT pipelines
    │   └── wf_<domain_c>.yml   # Workflow: SQL task-based job
    ├── pipelines/
    │   ├── pl_<entity_a>.yml   # DLT: Bronze → Silver → Gold
    │   ├── pl_<entity_b>.yml   # DLT: another domain pipeline
    │   └── ...                 # (one YAML per domain/entity)
    └── sql_warehouses/
        └── cl_sql_xsm_stage.yml  # Serverless SQL warehouse (X-Small)

One pipeline, one YAML. Clean, predictable, and easy to find what you're looking for.

Multi-Target Configuration

This is where DAB really earns its keep. Instead of maintaining separate repos or complex branching strategies per environment, we use targets — environment-specific configurations all sitting inside a single databricks.yml:

Each target overrides variables like the Unity Catalog name, data lake schema, and environment label. The same pipeline YAML works across all environments — zero code changes required when promoting between them.

# databricks.yml (simplified)
targets:
  dev:
    mode: development
    workspace:
      host: https://<your-dev-workspace>.azuredatabricks.net/
    variables:
      pipeline_uc_catalog:
        default: dev_<your_catalog>

  prod:
    mode: production
    workspace:
      host: https://<your-prod-workspace>.azuredatabricks.net/
    variables:
      pipeline_uc_catalog:
        default: prod_<your_catalog>

Pipeline Resource Design

DLT Pipelines

Each DLT pipeline follows the Bronze → Silver → Gold medallion architecture. Common utilities are shared across all pipelines (think a shared loader module and init file). Pipeline-level configs — catalog names, schema names, storage paths — are injected via variables at deploy time, so there's nothing hardcoded:

# resources/pipelines/pl_<entity>.yml (simplified)
resources:
  pipelines:
    pipeline_pl_<entity>:
      name: pl_<entity>
      configuration:
        pipeline.uc_catalog: ${variables.pipeline_uc_catalog.default}
        pipeline.uc_schema_brz: ${variables.pipeline_uc_schema_brz.default}
      libraries:
        - file: { path: ../../src/code/bronze/<entity>_bronze_dlt.py }
        - file: { path: ../../src/code/silver/<entity>_silver_dlt.py }
        - file: { path: ../../src/code/gold/fct_<entity>_dlt.sql }
      photon: true
      serverless: true

Jobs / Workflows

Jobs wire DLT pipelines together. We use two patterns depending on the use case:

• Pipeline task jobs — trigger DLT pipelines by resource reference. Good for orchestrating multiple related pipelines in sequence.

• SQL task jobs — run SQL scripts against a SQL warehouse, with conditional full-refresh logic built in.

The Azure DevOps CI/CD Pipeline

Our azure-pipelines.yml defines three stages that execute on every push to any of our active branches:

Push to branch
      │
      ▼
┌─────────────┐     ┌───────────────────┐     ┌─────────────────────────┐
│  Stage 1    │────▶│     Stage 2       │────▶│       Stage 3           │
│  Test       │     │  Code Quality     │     │  Deploy to Environment  │
│  (pytest)   │     │  (SonarCloud)     │     │  (DAB deploy)           │
└─────────────┘     └───────────────────┘     └─────────────────────────┘

Branch → Environment Mapping

Simple and deliberate. Dev merges are instant; production deployments require a human to say yes.

Authentication

We use an Azure Service Principal with the AzureCLI@2 task to obtain a short-lived Databricks access token at deploy time. No secrets baked into YAML, no long-lived credentials lying around:

DATABRICKS_TOKEN=$(az account get-access-token \
  --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d \
  --query "accessToken" -o tsv)

Short-lived tokens via service principal is the way to go. It's one less thing to rotate manually.

Deploy Step

databricks bundle validate -t $(environment)
databricks bundle deploy -t $(environment) --auto-approve

--auto-approve is required in non-interactive CI/CD environments to bypass prompts for destructive actions like renaming or deleting resources. Skip it and your pipeline will hang indefinitely — ask me how I know.

Key Lessons Learned

These are the things I wish someone had told me when we started. Saving them here so you don't have to learn them the hard way:

1. Name resources without environment suffixes. Let DAB targets handle environment context — use pl_orders, not pl_dev_orders. The moment you bake the environment into the name, renames will break your CI/CD and your day.

2. Use variable references consistently. Never hardcode catalog names or storage accounts inside pipeline YAMLs. Always use ${variables.*} so the environment config files are the single source of truth. It pays off every time you add a new target.

3. --auto-approve is mandatory in CI/CD. The Databricks CLI will block on any destructive action when there's no interactive terminal. Always include this flag in your deploy command — you'll forget it at least once, and you'll remember it forever after.

4. Serverless + Photon = simplicity. Serverless pipelines remove the need to manage cluster sizing in each pipeline config. Less boilerplate, fewer misconfigurations, faster pipelines. It's an easy win.

5. Event log per pipeline. Store pipeline event logs in a dedicated schema. It enables centralised monitoring across all DLT pipelines via a single Unity Catalog query — much better than hunting through individual pipeline UIs.

In the next blog

I can give a technical deep dive of the pipeline code and how we are using the Unity Catalog and Delta Live Tables to build the data platform.

• Azure Devops pipeline

• Deploying to data bricks using azure Devops

Deprecation of ipvs mode

KEP-5495: Deprecate ipvs mode in kube-proxy

Since the iptables scalability issues are being solved using nftables, ipvs mode is essentially redundant. It's time to say goodbye to the complexity of IPVS. If you want the gritty details on why iptables needed a replacement, check out this deep dive.

Node Declared Features

KEP-4033: Discover cgroup driver from CRI

This solves the "does this node actually support what I need?" problem without manual labeling. Nodes can now self-report specific capabilities (like hardware or plugins) directly to the control plane via the API. This allows the scheduler to automatically place Pods on compatible nodes without you having to manually manage feature labels!

status:
  declaredFeatures:
    example.com/gpu: "true" # Node says: "I explicitly support this!"

In-Place Update of Pod Resources

KEP-1287: In-place Update of Pod Resources

This has been in the works for a while, but it's a game changer! Vertical Pod Autoscaling (VPA) can finally resize CPU and Memory limits without restarting the Pod. No more disruptions just to give a hungry container a bit more RAM.

Numeric Values for Taints

KEP-5471: Enable SLA-based Scheduling

Taints used to be boolean (it exists or it doesn't). Now, we get math!

Before: taint key=HighPriority:NoSchedule (You either match "HighPriority" exactly, or you don't).

After: taint reliability=999:NoSchedule Pod Toleration: operator: Gt, value: 950 (The Pod says: "I can only tolerate nodes with reliability > 950" — much more expressive!)

User Namespaces

KEP-127: User Namespaces

This is a massive security upgrade for running containers safely.

Before: Container root (UID 0) == Host root (UID 0). (If they break out of the container, they are root on your server. Scary!)

After: Container root (UID 0) == Host nobody (UID 65534). (They are root inside, but powerless outside. Much better.)

OCI Images as Volumes

KEP-4639: OCI Volume Source

Stop building massive images just to include data!

Before: InitContainer runs wget https://... -> saves to emptyDir -> MainContainer reads it. (Slow, complex startup scripts, wasted bandwidth.)

After:

volumes:
  - name: data
    image:
      reference: "my-data-layer:latest" # Mounts directly!

That's a wrap for the 1.35 highlights!

If you found this useful, feel free to reach out to me on LinkedIn. Let's discuss what feature you are most excited about!

For more detailed Reading:

1.35 Release notes: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.35.md

https://isovalent.com/blog/post/why-replace-iptables-with-ebpf/

Hello !

You ask your favorite AI assistant to whip up some Terraform code for a new module, and it confidently spits out syntax that was deprecated six months ago. 🤦‍♂️ While LLMs are fantastic for boosting productivity, they often rely on their training data, which can be a frozen snapshot in time. This means they miss out on the latest features and best practices.

I've been testing a new tool from HashiCorp that directly tackles this problem: the terraform-mcp-server. It essentially gives your LLM a direct, live line to the official Terraform documentation, ensuring the code it generates is accurate and up-to-date.

For my setup, I use Podman on my local machine, but you can use Docker or any container tool you prefer. And my go-to co-programmer for everything these days is Cline, which integrates with tools like this seamlessly.

Why is the Terraform MCP Server a Game-Changer?

The core issue is that LLMs, by default, refer to the data they were trained on. To make them more effective, we can give them "tools" they can use to fetch live information. The terraform-mcp-server acts as this tool, an independent agent with a specific set of capabilities.

Think of it as giving your AI assistant a toolkit specifically for Terraform. Instead of guessing, it can now ask questions and get real-time answers. The tools it gets access to include:

• get_latest_module_version

• get_latest_provider_version

• get_module_details

• get_policy_details

• get_provider_details

• search_modules

• search_policies

With these capabilities, the LLM can perform a thorough analysis before generating a single line of code. When I tested it on a GCP VPC module, it correctly used all the newest features without any hallucinated or outdated arguments. It was a beautiful thing to see!

The Setup is Super Simple

Getting this up and running is incredibly straightforward.

1. First, clone the repository from GitHub:

   Bash

    git clone https://github.com/hashicorp/terraform-mcp-server.git
   

2. Next, build the container image. I'm using podman, but docker build works just the same.

   Bash

    cd terraform-mcp-server
    podman build -t terraform-mcp-server:dev .
   

3. Finally, run the server!

   Bash

    podman run -p 8080:8080 --rm terraform-mcp-server:dev
   

   Note: The user's original notes included -e TRANSPORT_MODE=streamable-http -e TRANSPORT_HOST=0.0.0.0, which are the defaults and can be useful for specific network setups, but the basic command above works for most local use cases.

That's it! The server is now running and ready to accept requests from your AI tool.

Integrating with an AI Assistant like Cline

To get my assistant, Cline, to use this new tool, I just needed to add a simple configuration. This tells Cline how to run the terraform-mcp-server whenever it needs to access Terraform information.

Here is the JSON configuration I used:

JSON

{
  "mcpServers": {
    "terraform": {
      "command": "podman",
      "args": [
        "run",
        "-i",
        "--rm",
        "terraform-mcp-server:dev"
      ],
      "autoApprove": [
        "list_tools"
      ]
    }
  }
}

Here is the code generated and reference: https://github.com/jothimanikrish/terraform-ai-gcp-vpc/tree/main/gcp-vpc-module

This configuration defines a tool named terraform, specifies the podman command to execute it, and automatically approves the initial tool listing.

This terraform-mcp-server is a fantastic solution for anyone using LLMs in their Infrastructure as Code workflow. It bridges the gap between the static knowledge of the model and the dynamic, ever-evolving world of Terraform. No more copy-pasting docs or second-guessing AI-generated code.

Happy Terraforming!

GKE Dataplane V2 is built directly on top of Cilium, bringing the power of eBPF to Google's managed Kubernetes service.

Sounds cool, right? Let's dive into a few exciting things that Dataplane V2 brings to the table.

What's the Big Deal? If you've been using GKE for a while, you're probably familiar with the standard networking stack that used Calico for Network Policy and kube-proxy (with iptables) for service routing. Dataplane V2 changes the game completely.

Here are the highlights:

• Default on Autopilot: To make things easy, Dataplane V2 is enabled by default for all new GKE Autopilot clusters. You might already be using it without knowing!

• No More kube-proxy! That's right. GKE Dataplane V2 completely removes kube-proxy and its complex web of iptables rules. This means service routing is handled far more efficiently by eBPF, leading to better performance and scalability.

• Built-in Security: Security is now a first-class citizen. You don't need a third-party tool like Calico just to enforce Network Policies. You can enable policy enforcement with a single click in the GKE console or a simple flag in your cluster config.

• Network Policy Logging: Ever wondered if a connection was allowed or denied by a Network Policy? Dataplane V2 has built-in logging for this. You can configure a simple CRD on your cluster to get detailed logs, which is a massive help for debugging and security audits.

• Real-time Network Visibility: Thanks to its eBPF foundation, you get much deeper, real-time visibility into the network traffic flowing between your pods.

The Specs: By the Numbers Dataplane V2 isn't just about features; it's built for scale.

• Specification Limit on Dataplane V2 Number of nodes per cluster 7,500 Number of Pods per cluster 200,000 Number of Pods behind one Service 10,000 Number of Cluster IP Services 10,000 Number of LoadBalancer Services per cluster 750.

  Things to Keep in Mind (The Limitations)

  As with any powerful technology, there are a few things you should be aware of before jumping in:

• Creation Time Only: Dataplane V2 can only be enabled when you create a new GKE cluster. You can't migrate an existing cluster to it on the fly, so plan accordingly.

• eBPF Map Limits: GKE Dataplane V2 relies on eBPF maps, which are limited to 260,000 endpoints across all services. An "endpoint" here is a single Pod backing a Service.

• Missing kube-proxy Features: Since kube-proxy is gone, you might miss some specific metrics or behaviors you were used to. It's a new paradigm, so some old debugging habits might need to change.

• Update Your GKE Version: To get the most out of Dataplane V2 and all its features without limitations, you'll want to be on a recent GKE version (the docs often recommend 1.31+ for the latest enhancements).

Key Takeaways GKE Dataplane V2 is more than just an update; it's a fundamental shift in how Kubernetes networking is handled in GKE. By leveraging Cilium and eBPF, it offers:

• Increased performance and scalability by removing kube-proxy.

• Simplified and integrated security with built-in Network Policies.

• Better observability with features like policy logging.

It's a powerful and efficient foundation for your modern applications running on GKE.

Hello All,

Like many of you, I was incredibly excited about the VPA Beta release in Kubernetes 1.33 (which you can read about here). I've since taken the time to understand the current VPA release as of July 2025 in more detail.

Let's dive into VPA in this blog post.

The Critical Question Every DevOps Engineer Asks

I have critical workloads in production that cannot be scaled horizontally, and the replica count is always set to 1. Shall I directly use VPA now to scale vertically?

The answer is NO.

Why? Let's first understand VPA in detail, and then I'll share my experience on why you should be cautious.

What is Vertical Pod Autoscaler?

Vertical Pod Autoscaler (VPA) automatically adjusts the CPU and memory reservations for your pods to help "right-size" your applications. Unlike Horizontal Pod Autoscaler (HPA), which scales the number of replicas, VPA scales the resources allocated to existing pods.

Think of it this way: HPA says, "I need more workers," while VPA says, "I need stronger workers."

Installation

Installation is straightforward. Refer to the official guide here.

Bash

git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler/
./hack/vpa-up.sh

VPA Operating Modes

VPA can operate in four different modes:

Container Resize Policies

With Kubernetes 1.33, you can now define how containers should handle resource changes:

YAML

spec:
  containers:
    - name: my-app
      image: my-app:latest
      resizePolicy:
        - resourceName: cpu
          restartPolicy: NotRequired # apply directly to running container
        - resourceName: memory
          restartPolicy: RestartContainer # apply and restart to take effect

This is a game-changer! Finally, we can control whether a container needs to restart when resources are updated.

Real-World Example

Let me share a practical example. Here's how I set up VPA for a monitoring application:

YAML

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: monitoring-app-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: monitoring-app
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
      - containerName: monitoring-app
        minAllowed:
          cpu: 100m
          memory: 128Mi
        maxAllowed:
          cpu: 1
          memory: 1Gi
        controlledResources: ["cpu", "memory"]

Where Can VPA Be Used?

1. Analyzing Current Workload Patterns (Off Mode)

Start with Off mode to understand your application's resource consumption patterns:

YAML

spec:
  updatePolicy:
    updateMode: "Off"

This mode is perfect for:

• Understanding resource usage patterns

• Getting recommendations without any changes

• Planning capacity for new applications

• use prometheus compatible metrics

Limitations and Gotchas

Important: As of Kubernetes 1.33, VPA still has some limitations you need to be aware of:

1. Pod Disruption

In Recreate and Auto modes, VPA will terminate and recreate pods. This means downtime for single-replica applications. This is why I said NO to using it directly on critical production workloads!

2. Compatibility Issues

VPA and HPA cannot target the same metrics (CPU/Memory). You can use them together, but HPA should target custom metrics.

3. Resource Limits

VPA doesn't set resource limits, only requests. You need to set limits manually or use LimitRanges.

My Production Strategy

Here's how I approach VPA in production environments:

Observation (Off Mode)

YAML

updateMode: "Off"

Run for 2-4 weeks to gather data and understand patterns. And Action!

Best Practices

1. Always set resource bounds:

   YAML

    minAllowed:
      cpu: 100m
      memory: 128Mi
    maxAllowed:
      cpu: 2
      memory: 4Gi
   

2. Use PodDisruptionBudgets:

   YAML

    apiVersion: policy/v1
    kind: PodDisruptionBudget
    metadata:
      name: my-app-pdb
    spec:
      minAvailable: 1
      selector:
        matchLabels:
          app: my-app
   

3. Monitor VPA recommendations:

   Bash

    kubectl describe vpa my-app-vpa
   

The Future is Bright

With in-place resource updates coming to Kubernetes, VPA will become much more production-ready. The ability to update resources without pod restarts will be a game-changer for critical workloads.

Until then, use VPA wisely:

• Start with Off mode for analysis.

• Use Initial mode for new deployments.

• Be cautious with Auto mode on critical applications.

Conclusion

VPA is a powerful tool, but like any powerful tool, it needs to be used with care and understanding. Don't rush into production with Auto mode on critical workloads. Take time to understand your application's behavior, set proper bounds, and gradually roll it out.

Remember: With great power comes great responsibility!

Have you tried VPA in your environment? Share your experiences in the comments below.

Happy scaling! 🚀

Intro

When it comes to managing operations, wouldn't it be great to have a more intuitive, convenient way to execute tasks, delegate work, and empower teams for self-service? That's what ChatOps brings to the table.

It's not just a trend—it's a paradigm shift in how we interact with our systems. While bots themselves aren't new, we're about to revolutionize how we interact with them, making the experience smarter, more intuitive, and deeply conversational.

Today, I'm going to walk you through how I built a ChatOps bot using Slack, LangChain, and an LLM of my choice, all wrapped in Python.

Why ChatOps?

ChatOps is all about making operations more accessible and collaborative. By integrating operations directly into Slack (or any chat platform), it becomes:

1. Intuitive: Users interact in natural language rather than remembering complex commands.

2. Convenient: Operations happen where your team is already communicating.

3. Empowering: Enables delegation and self-service, minimizing reliance on admins for every minor operation.

4. Automated: Offloads repetitive tasks, leaving your team free to focus on higher-level decisions.

ChatOps with LLM

Traditionally, ChatOps bots like ErrBot work by triggering commands—users invoke specific commands directly. For example, typing !status would return the status of a service.

But what if we could make it smarter and more conversational? Instead of explicitly typing exact commands, we can leverage LLM to make our bot understand and execute intent, even if the phrasing isn't an exact match. This is where LangChain comes in.

LangChain

LangChain is a powerful framework that simplifies the integration of LLMs (large language models) into real-world applications. It manages components like querying the LLM, parsing responses, and chaining together operations to make it seamless for developers like us. LangChain enables us to easily adapt traditional bots like ErrBot, injecting them with the intelligence of an LLM—while still keeping operations under tight control.

Defining My ChatOps Framework

For any good operational framework, a few essentials are non-negotiable:

1. RBAC (Role-Based Access Control): Define who can trigger which commands and limit the actions they can perform.

2. Channel Control: Specify where the bot can operate—not all commands should work in every channel for safety reasons.

3. Command Rules: Decide what tasks the bot can execute.

In my case, I’m using ErrBot as the underlying framework for simplicity and layering LangChain on top of it to add conversational intelligence.

Features My Bot Supports

Before introducing LangChain to the mix, my bot already supported the following commands:

• help: Lists all the available commands.

• status: Returns the current status of a service or environment.

• approve_release: fetches the pending releases and prompts for approval

These commands work perfectly, but they lack the flexibility of natural language interactions. For example:

• Instead of typing !status, wouldn’t it be nice to type “What’s the system status?” and have the bot figure out the intent?

Architecture: The Big Idea

Here’s how I designed the bot after integrating LangChain:

1. Intent Classifier
   The heart of the system! It takes the user’s prompt, runs it through an LLM (via LangChain), and extracts the intent. For example:

   • Input: “Can you check on the system?”

   • Result: Intent is classified as status.

The intent classifier strips down the natural language input into actionable commands for ErrBot. This ensures that only valid intents trigger actions, reducing the risk of LLM hallucinations or misfires. It's contextual and scoped, making it both powerful and safe.

2. LLM Integration with JSON Responses
   The LangChain model responds with structured JSON, making it easy to decode:

    {
        "intent": "status",
        "parameters": {}
    }
   

   By relying on this structured response, we can map intents to specific bot commands confidently.

3. Mapping Intent to Commands
   Once the intent is identified, the bot triggers the appropriate command handler. It’s a simple and modular pipeline that feels scalable and robust: - User types natural language. - LLM determines intent. - The bot executes the mapped command.

Why This Approach Works

This architecture minimizes AI hallucination by restricting the context in which the LLM operates. The intent classifier ensures the bot doesn’t go rogue—it only triggers predefined, purposeful commands. This guards your ChatOps setup and makes AI-powered automation more predictable.

Code Example

Here’s a simplified Python example showing how this would work using LangChain and Slack as the frontend:

from typing import Optional, Dict, Any
from langchain_openai import OpenAI
from langchain.prompts import PromptTemplate
from langchain.schema.runnable import RunnablePassthrough
import json
import logging

logger = logging.getLogger(__name__)

class IntentClassifier:
    def __init__(self, api_key: str, api_base: str, config: dict):
        """Initialize the intent classifier with LangChain"""
        self.llm = OpenAI(
            model="claude-3-5-sonnet-latest",
            temperature=0.3,  # Lower temperature for more consistent results
            openai_api_key=api_key,
            openai_api_base=api_base
        )

        # Get commands from config
        self.commands = config.get("bot", {}).get("commands", {})

        # Build intent descriptions from commands
        intent_descriptions = []
        for cmd_name, cmd_config in self.commands.items():
            if cmd_config.get("enabled", True):
                description = cmd_config.get("description", "")
                aliases = cmd_config.get("aliases", [])
                alias_text = f" (aliases: {', '.join(aliases)})" if aliases else ""
                intent_descriptions.append(f"- {cmd_name}: {description}{alias_text}")

        logger.debug(f"Loaded {len(intent_descriptions)} commands for intent matching")

        # Define the classification prompt template
        template = """
        You are a Slack bot designed to classify user messages into predefined intents. 
        Your task is to analyze the user message and match it to the closest predefined intent or respond if no match is found.

        Predefined intents include:
        {commands}

        User message: {message}

        Return a JSON object with:
        - intent: The matched intent name or null if no match
        - confidence: Confidence score between 0 and 1
        - extracted_params: Any parameters extracted from the message (optional)        

        Important rules:
        1. Only return intents from the predefined list above
        2. Consider command aliases when matching intents
        3. Extract any relevant parameters mentioned in the message
        4. Return null if no intent matches with high confidence (below 0.6)
        5. Ensure the response is valid JSON with no additional text
        6. Be generous with confidence scores when the intent is clear
        """

        self.prompt = PromptTemplate(
            input_variables=["message", "commands"],
            template=template
        )

        self.commands_str = "\n".join(intent_descriptions)

        # Create a runnable chain
        self.chain = (
            {"message": RunnablePassthrough(), "commands": lambda _: self.commands_str}
            | self.prompt
            | self.llm
        )

        logger.info("Intent classifier initialized successfully")

    async def classify_message(self, message: str) -> Optional[Dict[str, Any]]:
        """Classify a message and return the intent details"""
        try:
            logger.debug(f"Classifying message: {message}")

            # Run the classification chain
            result = await self.chain.ainvoke(message)
            logger.debug(f"Raw classification result: {result}")

            # Parse the JSON response
            intent_data = json.loads(result)

            # Validate the response format
            if not isinstance(intent_data, dict):
                logger.error(f"Invalid classification response format: {result}")
                return None

            required_fields = ["intent", "confidence"]
            if not all(field in intent_data for field in required_fields):
                logger.error(f"Missing required fields in classification: {result}")
                return None

            # Log the classification result
            intent = intent_data.get("intent")
            confidence = intent_data.get("confidence", 0)
            params = intent_data.get("extracted_params", {})
            logger.info(
                f"Classification result - Intent: {intent}, "
                f"Confidence: {confidence}, Params: {params}"
            )

            return intent_data

        except Exception as e:
            logger.error(f"Error classifying message: {str(e)}")
            return None

    def _validate_confidence(self, confidence: float) -> bool:
        """Validate confidence score is between 0 and 1"""
        return isinstance(confidence, (int, float)) and 0 <= confidence <= 1

In this example:

1. The LLM determines intent based on my input.

2. Based on the intent (status, echo), the bot routes the input to the relevant handler.

3. Only predefined commands are triggered, keeping the system secure and deterministic.

Thank you !!!

Accessing the cluster involves three types:

Lets us understand how it was working before using aws-auth and CONFIGMAP.

The aws-auth ConfigMap (deprecated)

This process involves mapping AWS IAM identities, including users, groups, and roles, to Kubernetes role-based access control (RBAC) for authorization.

Challenges and Pain Points

Ideally, this configuration should be managed internally within the cluster. After provisioning the cluster, it is necessary to establish a configuration that facilitates the relationship between IAM and the Kubernetes system.

Although eksctl can be used for this setup, it's important to note that not all clusters are created with eksctl. Thus, alternative methods need to be available for those who do not use this tool.

Why I love EKS Access Entries? Simplicity!

You do not need to learn anything new; simply integrate existing IAM principles with Kubernetes permissions.

IAM principles can be mapped to any of the four EKS permissions. Below is the mapping between EKS access policies and Kubernetes' default RBAC:

• AmazonEKSClusterAdminPolicy: cluster-admin in kubernetes

• AmazonEKSAdminPolicy: admin

• AmazonEKSAdminViewPolicy: (get, list, watch across all API and resources)

• AmazonEKSEditPolicy: edit

• AmazonEKSViewPolicy: view

Excited?

How to enable the access entries API?

Eksctl

accessConfig: 
    authenticationMode: <>

Terraform:

authentication_mode = "API_AND_CONFIG_MAP"

Cluster administrators now have the capability to grant AWS IAM principals access to Amazon EKS clusters and Kubernetes objects across all supported versions (version 1.23 and later).

configmap is to be disabled soon?

Access Entries:

Image ref: https://aws.amazon.com/blogs/containers/a-deep-dive-into-simplified-amazon-eks-access-management-controls/

If you already using aws auth configmap, here is migration guide for the easy access management life

Here is an example of Terraform configuration for associating your IAM users and roles with the respective EKS access:

Step 1 - create access entry

resource "aws_eks_access_entry" "readonly" {
  cluster_name      = "eks-demo-cluster"
  principal_arn     = aws_iam_role.example.arn #user-iam-arn
  kubernetes_groups = []
  type              = "STANDARD"
}

Step 2 - associate policy to it.

resource "aws_eks_access_policy_association" "readonly" {
  cluster_name  = "eks-demo-cluster"
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
  principal_arn = aws_iam_user.example.arn #user-iam-arn

  access_scope {
    type       = "namespace"
    namespaces = ["example-namespace"]
  }
}

Thank you!

If you have any questions, feel free to reach out to me on LinkedIn, and let's discuss further.

Detailed References:

https://docs.aws.amazon.com/eks/latest/userguide/grant-k8s-access.html

https://aws.github.io/aws-eks-best-practices/security/docs/iam/#the-aws-auth-configmap-deprecated

Since its inception in 2006, S3 has undergone significant evolution. From offering a range of bucket classifications like Standard and Glacier to implementing zonal replications and versatile access policies, the advancements have been noteworthy.

One of the more recent additions is the concept of mount points for S3, which further enhances its functionality.

You can learn more about this development here: [Mountpoint for Amazon S3]

Another exciting feature recently introduced is "Directory Buckets." This addition to the S3 lineup offers even more flexibility and options for managing cloud storage efficiently.

What is a Directory Bucket?

S3 has now been categorized into two main types:

1. General Purpose Buckets

2. Directory Buckets

Let's explore what Directory Buckets are all about.

Key advantages:

This advanced storage class stands out with three key features:

- Single-digit millisecond first byte latency for compute-intensive and latency-sensitive applications

- Consistent performance eliminates tail latencies, driving down query times

- Data access speeds up to 10x faster, and requested costs up to 50% lower than S3 standard.

Points to consider:

• These buckets has naming standards

  • Base-name--azid--x-s3

More details here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-bucket-naming-rules.html

Points to Consider during design:

• This is one zone s3 storage so, in times of outage there will be dataloss or unavailability

• Directory buckets store data across multiple devices within a single Availability Zone

Thank you!!!

• We're all familiar with the syllabus and structure available on the official pages. In this post, I'll be sharing a fast-track method to attain the certification, assuming you have around two years of experience in Kubernetes.

Maximizing Efficiency in CKS Exam Preparation: Emphasis on Practice and Speed

When preparing for the Certified Kubernetes Security (CKS) or Certified Kubernetes Administrator (CKA) exam, two key elements play a crucial role in your success: practice and speed.

It's essential to familiarize yourself with imperative commands, which can significantly streamline your workflow. Keeping a handy list of these commands can be a game-changer. Here are a few critical commands, among others, to remember:

1. Creating Secrets Quickly: Instead of dealing with the complexities of YAML definitions and base64 encoding, use the create secret command for swift creation. For example:

    kubectl create secret generic --from-literal=username=admin --from-literal=password=admin
   

   This command allows you to swiftly set up secrets, a fundamental aspect of Kubernetes security.

2. Creating a Service Account: Simplify the creation of service accounts with this command:

    kubectl create serviceaccount readonly-sa --namespace=dev
   

   This is an efficient way to manage access within different namespaces.

3. Defining Roles: Establish roles easily with:

    kubectl create role dev-role --namespace=dev --verb=get --resource=pods
   

   This command helps in setting up roles that define what actions are permitted on specific resources.

4. Setting Up Role Bindings: To link roles to service accounts, use:

    kubectl create rolebinding dev-rolebinding --namespace=dev --role=dev-role --serviceaccount=dev:readonly-sa
   

Other keys to keep handy,

• Creation of Persistent Volumes (PV)

• Creation of Persistent Volume Claims (PVC)

• Creation and management of Pods

Effective Documentation Utilization:

For instance, consider using the official Kubernetes documentation at https://kubernetes.io/docs/home/.

It's crucial to develop proficiency in locating specific information, such as steps for upgrading a cluster.

When searching for "upgrade," ensure that the sources you refer to are from the official Kubernetes documentation (URLs containing 'kubernetes.io'), rather than relying on discussions or forum pages. This approach guarantees accurate and authoritative information.

Study Materials and Resources Utilized for Exam Preparation

CKA:

• Kodekloud

• practices test from http://killer.sh/

• https://github.com/kodekloudhub/certified-kubernetes-administrator-course/tree/master/docs

CKS:

• practices test from http://killer.sh/

• https://github.com/kodekloudhub/certified-kubernetes-security-specialist-cks-course/tree/main/docs

• https://kodekloud.com/courses/certified-kubernetes-security-specialist-cks/

• https://github.com/ramanagali/Interview_Guide/blob/main/CKS_Preparation_Guide.md

Good Luck!

This isn't just another typical blog post rehashing the HCTA exam details. We won't be covering the syllabus or exam structure that you can easily find on Hashicorp's official website.

In this post, I'll be detailing my journey to successfully passing the HCTA, focusing on the strategies and preparations I undertook.

Prerequisites:

The key is to gain as much practical experience as possible.

The more challenges you encounter and overcome in your hands-on practice, the smoother your exam experience will be.

Although the exam consists of multiple-choice questions, some of them are designed to test your practical knowledge to such an extent that you're likely to answer correctly only if you have real-world experience with the scenarios presented.

Key Preparation Strategies and Challenges

Understanding the nuances of Terraform commands and their differences is crucial.

• terraform state (all sub commands)

• terraform show

• terraform output

• Some of the tricky questions include - terraform push or terraform state push

• After the moving the state file to another backend, what should you do?

terraform init or terraform state push

• These questions cannot be answered unless you have worked; going on with the theory and white papers will not help.

Terraform count() vs foreach?

• Remote backends and available remote backends

How to Prepare Effectively?

1. Enroll in a Comprehensive HCTA Exam Preparation Course: This will provide a structured learning path.

2. Undertake a Small Project Covering All Terraform Use Cases: It's vital not just to know but to understand the use of all Terraform commands thoroughly.

3. Mock Exams: When you feel ready, test your knowledge with mock exams. There are numerous resources available for this.

Pls remember Theory + Hands-on + Mock exams = HCTA Certified

Good Luck!

References:

Course I used for Preparation: https://www.udemy.com/course/terraform-beginner-to-advanced/

Exam practice: https://www.udemy.com/course/terraform-associate-practice-exam

Kubernetes is widely recognized for its ability to manage containerized applications at scale. However, when it comes to managing stateful applications, certain considerations must be addressed. This article explores the challenges faced in scaling stateful applications and presents solutions for seamless scalability.

Let's consider a scenario where we have an application consisting of two microservices that communicate and require a shared volume to support specific processing operations.

Additionally, the data generated by these services need to be retained for further analysis.

Requirement:

To ensure scalability among worker nodes, it is necessary to implement a solution that meets the following criteria:

• The pods must have access to a shared persistent volume.

Create EFS

We are not going into the details of creating EFS. Assuming we have an existing EFS already.

my efs mount id: fs-582a0sdat

Make sure to create an access point. Access points are mount paths in EFS where the data should be accessed.

my efs access point /tmp/poc-efs

Deploy EFS driver in the cluster

Checkout the official document from AWS for EFS driver:

https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html

Create Storageclass

This is to define a storage class that tells the k8s to this resource to provision volumes. There are a lot of provisioners like EBS, GCE PD, EFS etc.

https://kubernetes.io/docs/concepts/storage/storage-classes/#provisioner

We are using EFS provisioner for our usecase.

storageClass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-582a0sdat
  directoryPerms: "700"
  gidRangeStart: "1000" # optional
  gidRangeEnd: "2000" # optional
  basePath: "/dynamic_provisioning" # optional

kubectl apply -f storageclass.yaml

Verify the installation.

kubectl get pods -n kube-system | grep efs-csi-controller

Create a persistent volume and persistent volume claim.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv1
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  mountOptions:
    - tls
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-582a0sdat:/tmp/poc-efs

-----------
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-claim
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 5Gi

Verify the status

kubectl get pv,pvc

Deploy Application

Let's use it in our deployment spec

deployment.yaml

apiVersion: v1
kind: Deployment
metadata:
name: efs-app
  labels:
    app: efs-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: efs-app
  containers:
    - name: myapp
      image: centos
      command: ["/bin/sh"]
      args: ["-c", "while true; do echo $(date -u) >> /data/out; sleep 5; done"]
      volumeMounts:
        - name: persistent-storage
          mountPath: /data
  volumes:
    - name: persistent-storage
      persistentVolumeClaim:
        claimName: efs-claim

kubectl apply -f deployment.yaml

Now bash into the pods and the mount path can be accessed across the pods.

Thank you, Happy Provisioning!

References:

https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html

Certainly! Let's consider a scenario where you have deployed a cluster (let's stick with EKS for simplicity and my preference) and have deployed node groups across 3 availability zones, consisting of 6 worker nodes. Now, when scheduling a deployment with 3 replicas, can you guarantee that they will be evenly spread across the nodes?

The answer is no!

It's challenging to expect such even distribution based on my exploration. However, Kubernetes provides an out-of-the-box solution to address this problem. This solution requires some preparation and planning during the setup stage.

We can delve into this detail in our blog post.

Strategies for Making Pods Highly Available in Kubernetes

Node affinity: Node affinity is similar to node selector with additional flexibility. There are two types of node affinity.

requiredDuringSchedulingIgnoredDuringExecution: The scheduler can't schedule the Pod unless the rule is met. This functions like nodeSelector, but with a more expressive/customizable/flexible syntax.**

preferredDuringSchedulingIgnoredDuringExecution: The scheduler tries to find a node that meets the rule. If a matching node is not available, the scheduler still schedules the Pod.**

The above will help us to attract pods to respective nodes based on the labels and constraints.

To deploy the pods across the worker nodes we use

Pod Topology Spread Constraints

TopologySpreadConstraints describes how a group of pods ought to spread across topology domains. The scheduler will schedule pods in a way that abides by the constraints. All topologySpreadConstraints are ANDed.**

Example definition file for the same

apiVersion: v1 kind: Pod metadata: name: example-pod spec:

Configure a topology spread constraint

---
apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  # Configure a topology spread constraint
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: ScheduleAnyway
      labelSelector: 
      matchLabelKeys:
        - app-name

  ### other Pod fields go here

maxSkew - The extent of uneven distribution in Kubernetes is defined by a parameter, which is set to a default value of 1. This means that at least one pod should be scheduled on each node.

For example, in a cluster with three zones, the initial deployment may have 2 pods in zone 1, 2 pods in zone 2, and 1 pod in zone 3 (2/2/1 distribution). If the deployment is scaled up to 6 pods, the distribution across zones would be adjusted to 2 pods in each zone (2/2/2 distribution).

topologyKey - in simplified terms, it defines a group of nodes using its labels kubernetes.io/hostname - is the node label and it recognizes all nodes with this label as topology and each server inside it is a domain.

labelSelector - used to select matching pods

The above definition will help us to deploy and spread the pods across the nodes making it highly available.

Other Useful Tools:

NodeLabeller - Auto labels node if it is GPU processor.

Descheduler - To effectively utilize resources in the nodes.

Thank you!

**definition from the k8s documentation

References:

https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/

https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector

https://github.com/RadeonOpenCompute/k8s-device-plugin/tree/master/cmd/k8s-node-labeller

Check out this blog post Go concurrency: https://opsinsights.dev/my-first-go-program

So, while getting started with Golang, I noticed similarities;

Otherwise, I tried to relate, and compare with python which helped me to grab the concepts, and syntax quicker. This is the motivation for this blog post, as understanding the similarities of what we already know helps our mind to easily adopt new learning.

Sharing my findings and understanding while learning GOlang. Part-1

A comprehensive guide to getting started with GO for python developers.

Highlights of Golang.

Go is expressive, concise, clean, and efficient\**

**as per documentation

• Go is a compiled language. Statically typed.

• Compiled executables are OS-specific, like a runtime. If compiled in windows will work only for windows.

• Case sensitive

• Go is designed for the next generation of C.

Sometimes it feels like python, but It's a fast, statically typed, compiled language that feels like a dynamically typed, interpreted language.

Advanced concepts in python like multiprocessing, and multithreading is day to day in Golang which is a core functionality.

Golang does not support.

• Type inheritance

• Method or Operator overloading

• Does not have structured exception handling. (try catch as in python)

• NO implicit numeric conversions

Importing packages.

.py

to import a package, below are some of the methods of importing modules, packages

import Game.Level.start #to import any specific module 
import Game.* #full import from Game.Level 
import start #without package prefix
import Game.Level as GL #alias imports

.go

import "fmt" import "os"

//to import multiple packages we can also enclose in simple brackets as shown below for.

import ( 
    "fmt" 
    "time" 
    "math" 
)

import mt "math" 
// syntax for alias import 

import . math 
// Dot import  allows to use modules without referencing package names.

Using variables

.py

variable declaration is a simple assignment using this operator. "="

variable_1 = 10 
variable_2 = "hello, team!"

*variables are case-sensitive.

.go

//var variableName dataType = initialValue
var x int = 10

there will be default initial values for variables if not assigned.

• int and float types: 0

• bool type: false

• string type: ""

• array and struct types: all of their elements are set to their respective zero values pointer, slice, map, channel, and function types: nil

For example, if you declare an integer variable x without an initial value, Go will assign it the zero value of int, which is 0:

Error handling

.py

try-catch block syntax in python.

try: 
    print(x) 
except: 
    print("An exception occurred")

.go

In Go, errors are represented as values of the built-in error, which can defined as.

type error interface { 
    Error() string 
}

If a function completes successfully the error value is null, else it will return the error.

The above example is with the interface, we will learn more in the next blog post.

All the above sections deserve an individual post, however*,* this will help you to kindle and kick-start Golang learning.

End of Part 1.

Happy learning. Thank you.

References:

https://go.dev/doc/code

https://blog.xojo.com/2017/12/06/compilers-101-overview-and-lexer/

A TF workspace isolates state information within a workspace. At the time of workspace creation, an isolated TF backend is created. This ensures existing configurations are not disturbed.

Sounds interesting? Let us find some interesting use in this blog post.

Using terraform workspaces is equivalent to using a container with different volumes.

NVM :p that's a metaphor.

By default, every time you do terraform init a workspace is created which is a default workspace. So by design or by choice, we are using terraform workspace every day.

you can quickly switch between workspaces using the terraform workspace cli. (Refer to TF workspace cheat sheet below)

Definition as in documentation:

You can create multiple working directories to maintain multiple instances of a configuration with completely separate state data.

As defined, state data if isolated between workspaces which helps to use the same code multiple provisioning.

To understand better let us try an example use case:

An identical infrastructure should be provisioned in two different regions, us-east-1(N.Virginia), and eu-west-1(Ireland)

Provided the use case, there are several ways to achieve this, however, let us see an example with terraform workspaces.

I have used a local provisioner for this demo to keep it simple. I did not create any workspace yet, let us check which workspace we are in.

Below terraform creates a new txt file with text inside "Workspace Blog post". Also if you notice I have used terraform workspace variable to dynamically name the txt file.

This will help us to identify from which workspace execution this file was created. Let us plan and apply our script. And a file will be created as shown below.

let us create a new workspace with the same blog-demo, and make sure we are in the correct workspace.

terraform workspace new blog-demo

Successfully created and terraform workspace list will display all the workspaces available. Trying to terraform plan and apply will results as shown below.

With the same terraform code without any modifications we had two different executions.

Now co-relate this with provisioning multi-region infrastructure with the common code base.

Key takeaways of using terraform workspace.

• in a similar approach, we can differentiate between regions based on the workspace names.

• terraform execution is faster since it will reuse the modules and packages downloaded stead of downloading again.

• increases code reusability and efficiency.

And any more use cases, it grows depending on our design pattern.

Workspace commands cheat sheet.

To list all the existing workspaces.

 terraform workspace list

To select/switch to a new workspace:

terraform workspace select <workspace name>

To create a new workspace:

terraform workspace new <workspace name>

To delete a workspace

terraform workspace delete  <workspace name>

ignore dependencies and track

terraform workspace delete -force <workspace name>

To display the current active workspace:

terraform workspace show

Thank you, Peace!

References:

Feel free to go through these articles if you prefer a more detailed understanding.

https://spacelift.io/blog/terraform-workspaces

https://developer.hashicorp.com/terraform/language/state/workspaces

https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file

This blog quickly shows the comparison between AWS Step functions and ARGO workflow.

AWS STEP Functions

AWS Step Functions is a serverless orchestration service that lets you integrate with AWS Lambda functions and other AWS services for any specific use cases.

Step functions help to execute stateful and stateless jobs in a sequential way to achieve a solution.

Assume 10-STEP operational workflow; in each step, it does a specific operation that is interrelated to all other jobs, like the success of STEP A triggers STEP B.

Each step can be in the different executor,

STEP-1: python, STEP-2: bash etc..,

And above is a single example and there can be n different use cases for such, like sequential Jobs, parallel jobs and at scale etc.

AWS Step functions are AWS agonistic. Yes, we can communicate with any of the other AWS services via API/Cloud SDK.

One of the practical AWS Step function use cases.

1. Restore a database from the backup.

2. Sanitize the sensitive data.

3. update the connection strings in the secrets manager.

ARGO Workflows.

ARGO workflow is an open-source container-native workflow engine for orchestrating jobs on Kubernetes.

container native - Yes, bring out your imagination of use cases here to leverage the power of ARGO-Workflow at scale.

Some of the most common use cases are,

• CICD pipelines

• analytics operation

• high volume data processing

• sequential, parallel and more

• directed-acyclic graph (DAG)

How it is different from AWS Step functions?

AWS Step functions are AWS agonistic whereas ARGO Workflows are cloud-agnostic.

• Can be provisioned using cloud formation templates and AWS CLI. (submitting step functions via CLI)

• job submission can be in JSON/YAML.

• support GUI, API, CLI and through supported AWS services. (Pls refer to the aws doc linked at the bottom.)

• support to create lambda functions at each stage.

ARGO Workflows can be used across cloud providers.

• Workflows can be templated/bootstrapped.

• ARGO jobs can be submitted using GUI and CLI.

• Workflows can be templated and YAML syntax

Pricing

AWS STEP Functions.

It is easy to pay as you go, However, if we decode below are the price points to be considered while using step fn.

• State transition charges + based on resource usage(CPU RAM utilized for each step)

• Lambda backend charges, EC2, ECS, ECR(if required)

ARGO

Cloud hosting charges in the Kubernetes environment. (can be a part of existing cluster) or Independent hosting in any vm.

Who knows maybe ARGO can come up with their native cloud-hosted environment in the future, and I am counting on it.

We will discuss ARGO workflow with an example in detail in our upcoming blog post

Thank you!

References:

https://docs.aws.amazon.com/step-functions/latest/dg/development-options.html

https://argoproj.github.io/workflows/#:~:text=What%20is%20Argo%20Workflows%3F,the%20workflow%20is%20a%20container.

https://aws.amazon.com/step-functions/pricing/

Merits of GO alongside Python.

GO solved two major problems,

• Ease of programming as an interpreted language like Python.
• With efficiency and safety as a static language like C++.
• Competitive multi-core computing as built-in support

Concurrency. This blog will focus only concurrency speciality of GO.

• Concurrency in other languages comes as extended functionality unlike in GO we have built-in support also known as goroutines. Goroutines are deeply integrated with Go's runtime, this runtime engine takes care of managing the threads (blocking & unblocking)

• The runtime and the logic of a goroutine work together. Goroutines can communicate with one another and synchronize their execution.

• In Go, one of the synchronization elements is called a channel. Channel helps to share data between goroutines

To learn more about concurrency in python: https://dev.to/kcdchennai/optimising-python-workloads-for-kubernetes-1d6c

To run a function as a goroutine, call that function prefixed with the go statement.

sum()     // A normal function call that executes sum synchronously and waits for completing it
go sum()  // A goroutine that executes sum asynchronously and doesn't wait for completing it

Lets the learn this concurrency in GO using an example.

Using Opensource API endpoint here http://worldtimeapi.org/pages/examples, this API helps us with timezone using various params, we are using this to find timezone based on IP.

package main

import (
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
)

func request(url string) {
    res, err := http.Get(url)
    if err != nil {
        panic(err)
    }

    defer res.Body.Close()
    b, err := io.ReadAll(res.Body)
    fmt.Println(string(b))
}

func main() {
    if len(os.Args) < 2 {
        log.Fatalln("Usage: go run main.go <url1> <url2> ... <urln>")
    }
    for _, url := range os.Args[1:] {
        request("http://" + url)
    }
}

Below is the response to my above excerpt. I have passed 4 args to the above function.

What is happening in the above program is Sequential execution

We sent a request to the first argument and when it completes, a response comes in. Then the program returns to the for loop and sends another request to the next argument, and the process continues.

Let's do concurrency using goroutines now

GOroutines

GOroutines are managed by GO runtime scheduler which takes care managing threads accessing CPU.

asynchronous, powerful

As we already know to make fn into goroutine, add a prefix go behind the fn invocation. Updating the code below

package main

import (
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
)

func request(url string) {
    res, err := http.Get(url)
    if err != nil {
        panic(err)
    }

    defer res.Body.Close()
    b, err := io.ReadAll(res.Body)
    fmt.Println(string(b))
}

func main() {
    if len(os.Args) < 2 {
        log.Fatalln("Usage: go run main.go <url1> <url2> ... <urln>")
    }
    for _, url := range os.Args[1:] {
        go request("http://" + url)
    }
}

The above program will result in below, which in 0.843 seconds.

:| why? Yes, we have started the concurrency and it triggered the fn 4 times but did not wait for the output. In order to visualize this let us add a waitgroup

WaitGroup

WaitGroup is included in the Golang sync package. It includes features that allow it to block and wait for any number of goroutines to complete their execution.

Code with added wait:

package main

import (
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "sync"
)

var wg sync.WaitGroup

func request(url string) {
    defer wg.Done()
    res, err := http.Get(url)
    if err != nil {
        panic(err)
    }

    defer res.Body.Close()
    b, err := io.ReadAll(res.Body)
    fmt.Println(string(b))
}

func main() {
    if len(os.Args) < 2 {
        log.Fatalln("Usage: go run main.go <url1> <url2> ... <urln>")
    }
    for _, url := range os.Args[1:] {
        go request("http://" + url)
        wg.Add(1)
    }
    wg.Wait()
}

While executing the same fn with 8 args, the response was within 0.853s which is 4x faster than the normal way.

Hope you enjoyed this new exploration. We will discuss more about Go in my upcoming blog posts.

Peace!

Author: Jothimani Radhakrishnan (Lifion by ADP). A Software Product Engineer, Cloud enthusiast | Blogger | DevOps | SRE | Python Developer. I usually automate my day-to-day stuff and Blog my experience on challenging items.

Intro

Hey Docker lovers <3, this is not going to be a happy story for you guys. Yes, for me as well. Like everyone, I loved using Docker very much. When I get to know about this project (Dockershim), it was a heartbreaking moment for me.

Before knowing about Dockershim let us discuss the following.

Kubelet takes care of managing worker nodes in relation to the master node. It ensures that the specified containers for the pod are up and running.

To know more about kubelet: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

Container runtime:

Container runtime is a software that is responsible for running containers. To create a pod, kubelet needs a container runtime environment. For a long time, Kubernetes used Docker as its default container runtime.

This creates a problem/dependency that whenever docker release updates/upgrades it breaks the Kubernetes.

There are also several container runtime tools.

• containerd
• CRI-O
• Docker
• Rocket
• LXD
• OpenVZ
• Windows Server Containers

Okay! Coming back to the context of this blog. Docker is going to be deprecated from Kubernetes as default and containerd is going to replace the place.

What is Dockershim?

Docker existed as a default engine in k8s, after introducing additional CRI access (Container runtime interface) in k8s, Kubernetes created an adaptor component called dockershim.

The dockershim adapter allows the kubelet to interact with Docker as if Docker were a CRI compatible runtime.

Img src: Kubernetes documentation

Switching to Containerd as a container runtime eliminates the middleman

Points to Check to make sure your environment is not affected by this change. You can continue to use Docker to build images, using Docker is not considered a dependency.

All these below pointers should be considered and updated as per your native CRI.

Make sure to update your docker commands operations that are running inside the pod. For example, listing running containers in the worker node using docker ps which might not work after the deprecation.

Check for any private registries or image mirror settings in the Docker configuration file (like /etc/docker/daemon.json)

1. Any scripts that ssh in worker nodes and does any docker CRUD operations.
2. Any third-party tools using docker needs to be updated. Migrating Telemetry
3. Any alerts that are configured based on Docker specific errors should be updated.
4. Any automation or bootstrap scripts based on Docker commands should be updated.
5. The list is not limited as mentioned above and might vary based on your adoption of usage.

To know more about the deprecation FAQ: https://kubernetes.io/blog/2020/12/02/dockershim-faq/

Thank you,

Happy containerd! :p

Reference: https://developer.ibm.com/blogs/kube-cri-overview/ https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-deprecation-affects-you/ https://kubernetes.io/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/

Before getting into the discussion of Karpenter, let's discuss k8s native cluster auto-scaler.

According to the documentation below is the definition:

Cluster Autoscaler is a tool that automatically adjusts the size of the Kubernetes cluster when one of the following conditions is true:

• there are pods that failed to run in the cluster due to insufficient resources.
• there are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes.

And if we prefer to adjust the CA for various use cases we have to create each group for them, as explained below.

Traditional way:

Creating autoscaling groups different types of node groups based on our needs.

Example:

ASG node group for GPU use ASG node group for general use and based on instance family etc..

This creates more overhead in maintenance and operation costs. :(

Why do we need a cloud-native cluster auto-scaler?

Cloud-native CA (cluster-autoscaler) explores the full capability of the native tools and technologies which helps to effectively and efficiently use the resources in all aspects.

This creates a debate about Kubernetes native vs AWS native.

Karpenter:

Karpenter solves the following problem making effective decisions and scheduling.

What does the pod need? Where is the pod best fit-in? What can I do to best fit that pod in a node?

it provides complete control over ec2 instances

cat <<EOF |kubectl apply -f -apiVersion: karpenter.sh/v1alpha5
 kind: Provisioner
 metadata:
 name: default
 spec:
 #Requirements that constrain the parameters of provisioned nodes. 
 #Operators { In, NotIn } are supported to enable including or excluding values
   requirements:
     - key: node.k8s.aws/instance-type #If not included, all instance types are considered
       operator: In
       values: ["m5.large", "m5.2xlarge"]
     - key: "topology.kubernetes.io/zone" #If not included, all zones are considered
       operator: In
       values: ["us-east-1a", "us-east-1b"]
     - key: "kubernetes.io/arch" #If not included, all architectures are considered
       values: ["arm64", "amd64"]
     - key: " karpenter.sh/capacity-type" #If not included, the webhook for the AWS cloud provider will default to on-demand
       operator: In
       values: ["spot", "on-demand"]
   provider:
     instanceProfile: KarpenterNodeInstanceProfile-eks-karpenter-demo
   ttlSecondsAfterEmpty: 30  
 EOF

From the above provisioner config file, we can see we can control instance type, Availability zone, architecture and capacity type (on-demand vs Spot)

Highlights:

• Faster Scheduling - Karpenter directly manages the nodes.

• No Node group provisioning - Karpenter takes care of instance provisioning with node groups, it can schedule pods in instances based on the configuration.

• Cost-effective than k8s native cluster auto-scaler.

Karpenter: Right now it only supports AWS native, anyone can contribute to other cloud tools.

Wonderfull self-explained video from JustinGarrison about Karpenter https://www.youtube.com/watch?v=3QsVRHVdOnM&ab_channel=JustinGarrison

--- End-of-Blog ---

Reference: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md

Elasticsearch(ES) is a distributed, full-text document store, search engine which is based on Apache Lucene as a core library. ES was originally designed for rich text-based search with advanced features to support complex queries, filters, analyzers with multiple languages. ES indexes and stores data so that it can be retrieved/searched in near real-time.

Distributed: ES saves the data in multiple nodes, data can be retrieved from any node at any time

Highlights of Lucene core:

• It can return search responses quickly.
• Based on the inverted index data structure. (stores mapping from content such as words or numbers to its location in a document(s))

Possibilities on using ES as primary database:

It depends on a lot of factors, such as,

(One ongoing WIP from the elastic team is that they are constantly working towards improving the resiliency)

    ○ Size of document
    ○ Number of concurrent requests of read per second
    ○ Writes per second

ES always does best in read requests, as the name suggests it is indexed data store.

If we prefer to increase the write (aka indexing rate) it can be achieved in several ways. refresh_interval, flush_threshold_size are some of the key parameters to be considered while improving writes.

Not impossible, however, is it advisable to go? My take;

Below are the pointers out of my experience in using ES as a database in production.

• It is not preferred to use ES as the primary database when you have (WPS) writes per second dominant reads (RPS).

• ES is very helpful when one of your primary use cases is to read, visualize the data using the complex combination.

• Best works for structured data set with less number of nested items.

To mitigate the concerns of write in elastic search, add another layer over ES, that can be REDIS or KAFKA buffer to queue the incoming data and to avoid data loss.

The main question is it worth adding more complexity to architecture? Which adds maintenance and other ad-hoc items related to the new services.

The use-cases which use this architecture are;

• Realtime logs storing systems
• Data capturing systems, where continuous ingestion of data from frontend applications.

And if your primary goal is to save JSON data and with the best performance in writes and reads, (considering all the points above in this section )please go ahead with mainstream NoSQL databases like MongoDB.

Invention and Discovery always being the part of human nature,

I always love the idea of extending the capabilities of any such machine with a pinch of salt and pepper to adapt to our complex use-cases in our daily. However, this should happen without killing the nature, purpose for which the tool is built since we might have straight forward alternative for any such use case.

EOB (End-of-Blog)

Reference:

https://medium.com/@merrinkurian/elasticsearch-as-the-primary-database-5e41b2a0189d https://cloud.netapp.com/blog/cvo-blg-elasticsearch-vs-mongodb-6-key-differences https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-indexing-performance/