Documenting a system you inherited

To properly document the existing Kubernetes environment setup and replicate it to a new server, you need a comprehensive approach. This includes gathering details about the infrastructure, Kubernetes cluster, services, configurations, and deployment pipelines. Here’s a step-by-step plan to help you document the current setup thoroughly and accurately.

Step 1: Gather Cluster Information

Check Cluster Details

Run the following commands to understand the existing cluster configuration:

• Cluster Info:

kubectl cluster-info

• Node Information:

kubectl get nodes -o wide

Document the following:

• Kubernetes version

• Node count and types (e.g., worker, master)

• Node OS and specifications (CPU, memory)

• Namespace Overview:

kubectl get namespaces

Document the namespaces in use (e.g., default, dev, prod).

Check Storage and Networking

• Storage Classes:

kubectl get storageclass

Document the available storage classes (e.g., standard, fast, gp2).

• Network Configuration:

Check the CNI plugin in use (e.g., Calico, Flannel):

kubectl get pods -n kube-system

Look for networking components like calico-node or flannel.

Step 2: Document Namespace-Specific Details

For each namespace (e.g., dev, prod), gather the following details:

Deployment Configurations

• List Deployments:

kubectl get deployments -n <namespace>

• Describe a Deployment:

kubectl describe deployment <deployment-name> -n <namespace>

• Export Deployment YAML:

kubectl get deployment <deployment-name> -n <namespace> -o yaml > deployment.yaml

Document the following:

• Deployment name and replicas

• Image versions (tagged versions)

• Resource requests and limits (CPU, memory)

• Environment variables

Service and Ingress Configurations

• List Services:

kubectl get svc -n <namespace>

• Describe a Service:

kubectl describe svc <service-name> -n <namespace>

• List Ingress Resources:

kubectl get ingress -n <namespace>

• Describe an Ingress:

kubectl describe ingress <ingress-name> -n <namespace>

Document the following:

• Service types (ClusterIP, NodePort, LoadBalancer)

• Ports and target ports

• Ingress rules (hostnames, paths, TLS settings)

Step 3: Document ConfigMaps and Secrets

ConfigMaps and Secrets store configuration data for your services.

• List ConfigMaps:

kubectl get configmap -n <namespace>

• Export ConfigMap YAML:

kubectl get configmap <configmap-name> -n <namespace> -o yaml > configmap.yaml

• List Secrets:

kubectl get secret -n <namespace>

• Export Secret YAML (base64 encoded):

kubectl get secret <secret-name> -n <namespace> -o yaml > secret.yaml

Document:

• ConfigMap keys and values (non-sensitive information)

• Secrets usage (e.g., database passwords, API keys)

Note: Avoid storing Secrets directly in version control; use tools like Helm or Kubernetes Secret management solutions (e.g., Sealed Secrets, HashiCorp Vault).

Step 4: Document Persistent Volumes (PV) and Persistent Volume Claims (PVC)

• List Persistent Volumes:

kubectl get pv

• List Persistent Volume Claims:

kubectl get pvc -n <namespace>

Document:

• Volume types (e.g., nfs, hostPath, gp2)

• Bound claims and storage capacity

• Access modes (ReadWriteOnce, ReadOnlyMany)

Step 5: Document CI/CD Pipeline

If the deployment uses a CI/CD pipeline (e.g., GitHub Actions, Jenkins, GitLab CI):

• Check the pipeline configuration files (.yml files) in the repository.

• Document the stages (build, test, deploy).

• Identify triggers (manual, automatic based on branch/tag push).

• Note any environment variables or secrets used in the pipeline.

Example GitHub Actions configuration:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
      - name: Build Docker image
        run: docker build -t myapp:${{ github.sha }} .
      - name: Push to registry
        run: docker push myapp:${{ github.sha }}
      - name: Deploy to Kubernetes
        run: kubectl apply -f k8s/deployment.yaml

Step 6: Document Monitoring and Logging Setup

Check the monitoring and logging setup in the current environment:

• Monitoring: Identify tools like Prometheus, Grafana, or Datadog.

  • Check for existing Prometheus configuration:

      kubectl get pods -n monitoring

• Logging: Identify tools like ELK stack (Elasticsearch, Logstash, Kibana), Loki, or Fluentd.

  • Check Fluentd or logging DaemonSet:

      kubectl get daemonset -n logging

Document:

• Monitoring and logging stack components

• Alerts and dashboards configuration

• Data retention policies

Step 7: Document Rollback and Disaster Recovery Plan

• Rollback Strategy: Note how to revert to the previous version (using Helm rollback or previous deployment YAML files).

• Backup Strategy: Document how to back up persistent data (e.g., volume snapshots, database backups).

• Disaster Recovery: Document the steps for recovering the cluster or data in case of failure.

Example Rollback Command:

kubectl rollout undo deployment <deployment-name> -n <namespace>

Step 8: Export the Current Cluster State

For a complete backup of the current environment, export the entire namespace configuration:

kubectl get all -n <namespace> -o yaml > namespace-backup.yaml

You can also use Helm if the applications are deployed using Helm charts:

helm list -n <namespace>
helm get all <release-name> -n <namespace>

Step 9: Create Infrastructure as Code (IaC) Documentation

If possible, use tools like Helm, Kustomize, or Terraform to define the infrastructure setup as code. This makes the process repeatable and consistent across environments.

Example Helm Chart:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: myapp
          image: myapp:latest

Conclusion

By following these steps, you will have a comprehensive understanding of the current environment and can easily replicate it on a new server. Documenting your environment setup helps ensure consistency, reduces the risk of misconfigurations, and simplifies future migrations or scaling efforts.