7 things no one will ever tell you about Kubernetes

Key points

• The Day-1 illusion: Cloud providers manage the control plane, but upgrading add-ons, rotating certificates, and patching nodes remain your responsibility.
• Cost overruns are guaranteed: Without automated governance to kill idle staging environments, container density drops and cloud bills multiply.
• Portability is a myth: Moving clusters between AWS and GCP requires rewriting storage classes, network policies, and identity integrations unless abstracted by an agentic orchestration layer.

Cloud providers sell Kubernetes as a magic abstraction layer. It is not. It is a highly complex distributed system that demands rigorous operational hygiene. Provisioning a cluster on AWS takes minutes, leading many engineering teams into a false sense of security. The reality of Day-2 operations is harsh.

If you treat Kubernetes like a massive, single Linux server, it will eagerly consume every dollar and IP address you provide.

I have watched countless organizations deploy Kubernetes to production, only to spend the next year fighting configuration drift, debugging networking failures, and explaining massive cloud bills to their finance department. To survive enterprise scale, you must understand these seven hard truths.

The 1,000-cluster reality: why manual operations fail

When you run a single cluster, manually updating Helm charts or writing AWS VPC CNI configurations is an acceptable chore. As your infrastructure footprint expands to thousands of clusters, manual configuration guarantees drift and security incidents.

Managing a global fleet requires you to stop interacting with the Kubernetes API directly. You must deploy an Agentic Kubernetes Management Platform to enforce strict, intent-based policies across all environments, ensuring consistency without human intervention.

Day 2 Operations & Scaling Checklist

Is Kubernetes a bottleneck? Audit your Day 2 readiness and get a direct roadmap to transition to a mature, scalable Platform Engineering model.

Download the checklist!

7 hard truths about Kubernetes at scale

1. Managed Kubernetes does not mean zero operations

AWS manages the etcd database and the API server. Everything else is your problem. You are responsible for upgrading the Amazon VPC CNI, managing CoreDNS, and rotating node certificates.

When AWS deprecates a Kubernetes API version, you must hunt down every outdated Deployment manifest in your repositories before the forced upgrade bricks your applications. The operational tax of keeping a cluster healthy requires dedicated engineering capacity.

2. Multi-cloud portability is an architectural lie

The primary marketing pitch for Kubernetes is that you can move your workloads anywhere. In practice, your manifests are heavily coupled to the underlying cloud provider. An Ingress resource on AWS requires the AWS Load Balancer Controller and specific annotations that are entirely useless on Google Cloud.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: external-ingress
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: backend-service
                port:
                  number: 80

Try applying that manifest to a GKE cluster and watch the scheduler reject it. To achieve true portability, you need an intent-based abstraction layer that translates your deployment requirements into provider-specific configurations automatically.

3. Default RBAC policies are a security vulnerability

Out of the box, Kubernetes architecture favors operational convenience over security. Teams frequently grant cluster-admin privileges to their CI/CD pipelines to ensure deployments do not fail.

If an attacker compromises your deployment pipeline, they gain total control over your production infrastructure. You must restrict deployment pipelines to specific namespaces using targeted RoleBindings.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: github-actions-deployer
  namespace: payment-service
subjects:
  - kind: ServiceAccount
    name: ci-pipeline
    namespace: default
roleRef:
  kind: Role
  name: deployment-manager
  apiGroup: rbac.authorization.k8s.io

4. You will hit AWS quotas before Kubernetes limits

Kubernetes supports 5,000 nodes per cluster, but your AWS account does not. As you scale, you will exhaust Elastic Network Interfaces, VPC IP addresses, and AWS API rate limits long before you hit Kubernetes scaling ceilings.

If you do not enable Prefix Delegation for the Amazon VPC CNI, your nodes will provision, but your pods will remain stuck in a Pending state due to IP starvation.

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

5. Cost overruns are guaranteed without FinOps automation

Kubernetes obscures hardware costs. Developers request 4GB of RAM for a microservice that needs 512MB. Over hundreds of deployments, your worker nodes scale out needlessly.

Implementing Karpenter to dynamically provision spot instances helps, but it is not enough. You must automate the teardown of non-production environments during nights and weekends to enforce strict FinOps policies.

6. Developers actively hate writing YAML

Software engineers want to write business logic, not debug CrashLoopBackOff errors or decipher readiness probe syntax. Forcing developers to manage their own Kubernetes configurations destroys feature velocity.

They require a self-service interface that abstracts the infrastructure into simple, declarative intents.

application:
  backend-api:
    build_mode: docker
    dockerfile_path: ./Dockerfile
    auto_scaling:
      min_instances: 3
      max_instances: 50
      cpu_threshold: 75

7. Building a custom internal platform guarantees technical debt

To hide Kubernetes complexity, Platform Architects often build custom internal developer portals using open-source tools. This DIY approach creates a permanent maintenance burden.

Your senior engineers spend their weeks patching internal Jenkins pipelines and Terraform modules instead of improving application reliability.

🚀 Real-world proof

Hyperline wanted to scale their fintech application globally without building a dedicated infrastructure team to manage raw Kubernetes components.

By utilizing an Agentic Kubernetes Management Platform like Qovery, platform teams eliminate toil, standardize multi-cluster deployments, and enforce global compliance automatically. Stop fighting raw Kubernetes primitives and start delivering reliable software.

FAQs

Is Kubernetes completely cloud agnostic?

No. While the core Kubernetes API is identical across providers, the configurations required for storage, networking, and load balancing are heavily coupled to the specific cloud provider. An Ingress manifest written for AWS Application Load Balancers will not work on Google Cloud without modifications.

What is the biggest hidden cost of running Kubernetes?

The biggest hidden cost is the operational toil required for Day-2 maintenance. Managing control plane upgrades, rotating node certificates, and patching network interfaces consumes massive amounts of senior engineering time that should be spent on product development.

Why do pods get stuck in a Pending state on AWS?

Pods typically get stuck in a Pending state on AWS due to IP address exhaustion within the Amazon VPC Container Network Interface. Even if the cluster has enough CPU and memory, the underlying subnet may lack available IPv4 addresses to assign to the new pods.