Shutdown & Startup Scripts

Bash scripts for ordered shutdown and startup of the k3s cluster

Motivation

Shutting down a Kubernetes cluster in the right order without losing data is more complex than it seems. Longhorn needs volumes to be properly unmounted, some services must stop before others, and the control plane must be the last to go.

Script Structure

cluster-scripts/
├── shutdown.sh       # Full ordered shutdown
├── startup.sh        # Startup and health checks
├── drain-node.sh     # Drain a specific node
└── lib/
    ├── k8s.sh        # kubectl helpers
    └── logging.sh    # Colored logging with levels

Shutdown Logic

Ordered cluster shutdown sequence

  1. Scale down stateful deployments (Immich, Gitea)
  2. Wait for Longhorn PVC flush to complete
  3. Sequential drain of workers (worker3 → worker2 → worker1)
  4. Shutdown the control plane
  5. SSH to each Pi and run sudo shutdown now

Notable Features

  • Configurable timeout with retry for each step
  • Automatic detection if Longhorn is in a degraded state before starting
  • Timestamped logs with levels (INFO/WARN/ERROR)
  • --dry-run mode to simulate without executing

Usage

# Full shutdown
./shutdown.sh --timeout 120

# Restart a node
./drain-node.sh worker2 && ssh pi@worker2 sudo reboot