infraops

Gopher DigitalOcean Referral Badge

CircleCI codecov Go Report Card Go Reference Stability: Active

🧭 InfraOps

A Go-based infrastructure “monorepo” powered by Pulumi, with configuration stored as HJSON and decoded into a protobuf schema.

[!NOTE] This repository is the operational source for several personal infrastructure areas. Prefer previews and narrow area changes before applying updates.

🗺️ Overview

This repository manages multiple infrastructure “areas”:

Each Pulumi area has:

Shared implementation lives under internal/ (e.g. internal/app, internal/cf, internal/do, internal/gh).

🧰 Tooling

🧱 Development tools

Development and CI-style commands use:

🛠️ Operator tools

Infrastructure operation also uses:

Operator-only helper targets use these tools as needed:

🧾 Configuration (HJSON + Protobuf Schema)

All area configuration files are HJSON (*.hjson). They are decoded into protobuf messages defined in:

Those protobuf messages are then converted into internal Go types and used to provision resources.

The protobuf types provide the expected configuration shape. Semantic checks that are not modeled in the schema may still fail later during Pulumi or provider operations.

The top-level version field in each HJSON file is the configuration schema version used by this repo’s tooling. It is not a Kubernetes, provider, Pulumi, or application runtime version.

🧹 Format and Normalize Config

This repo includes a small CLI to normalize/format config files by:

  1. decoding the HJSON into the appropriate protobuf message
  2. writing it back out in a canonical form

Build:

make build-format

Format a config kind (apps|cf|do|gh), using the default location area/<kind>/<kind>.hjson:

./format -k cf

Override the path:

./format -k cf -p area/cf/cf.hjson

The selected kind must match the file’s area-specific fields. A mismatch is rejected without changing the file.

📝 Config Schema Notes

A few conventions are implemented by the Go code and are worth knowing when editing HJSON:

🧭 Application.kind Deployment Model (apps)

Supported values:

Other values are unsupported and are not pre-validated by the helper code; malformed values may fail later during Pulumi/Kubernetes application.

All apps receive CircleCI release-agent labels/annotations and must serve HTTP liveness/readiness probes at /<name>/livez and /<name>/readyz on port 8080; startup uses a TCP probe on 8080.

All app containers use native lifecycle.preStop.sleep for five seconds before Kubernetes sends the termination signal, allowing endpoint and routing updates to propagate. Their 35-second pod termination grace period leaves 30 seconds for graceful process shutdown after the delay.

🪪 Application.id CircleCI Project ID (apps)

Application.id is written to the deployment annotation circleci.com/project-id. Keep it aligned with the CircleCI project identifier when adding or renaming apps.

🔐 EnvVar.value Secret References (apps)

Environment variables support literal values, and a secret reference format:

At deploy time, this becomes a Kubernetes SecretKeyRef:

The format is intentionally not pre-validated by the helper code; malformed references are expected to fail during Pulumi/Kubernetes application.

Example:

env_vars: [
  { name: "DATABASE_URL", value: "secret:db/url" }
]

🧩 Application.secrets vs Secret Env Vars (apps)

📏 Application.resource Sizing (apps)

Application.resource selects a resource profile. Current mapping:

Unknown values fall back to "small".

🔢 Application.replicas Pod Count (apps)

Always set replicas explicitly; there is no fallback to three. Because this is a plain proto3 scalar, omission decodes to zero. Use replicas: 0 explicitly when the application should run no pods. Canonical rewrites by the format or bump tool may omit an explicit zero; that does not change the deployed replica count.

replicas: 3

[!WARNING] replicas: 1 is valid, but the current rolling update and PodDisruptionBudget both allow one unavailable pod, so it does not guarantee continuous availability during disruption or rollout.

🛡️ App NetworkPolicy Baseline

The apps Pulumi program creates a NetworkPolicy that selects each app’s pods. Ingress is limited to debug on 6060, HTTP on 8080, and gRPC on 9090 for every app, regardless of kind. Egress currently remains open because outbound traffic flows are not modeled per app yet. Future egress restrictions should be introduced per namespace/app after the required flows are known.

🔁 Common Workflows

🚚 Checkout / Bootstrap

The root Makefile includes shared build tooling from the bin submodule. Initialize it before running Make targets from a fresh checkout:

git submodule sync
git submodule update --init

✅ Dependencies, Linting, Tests, and Security

[!IMPORTANT] Run make dep before local validation after checkout, dependency changes, or generated/vendor-state changes.

From the repository root:

make dep          # download/tidy/vendor deps
make lint         # lint Go sources (including field alignment)
make scripts-lint # lint repository shell workflow helpers
make sec          # govulncheck + Trivy
make specs        # gotestsum + go test (junit/coverage under test/reports)
make coverage     # HTML + function coverage under test/reports

🧬 Protobuf / API

[!IMPORTANT] Do not edit generated Go code under api/infraops/v2/*.pb.go directly.

Instead:

make api-lint
make api-breaking
make api-generate

(Or: make -C api lint|breaking|generate.)

make api-breaking compares the API module against the GitHub master branch, so it needs network access to run outside CI.

🚀 Pulumi: Preview/Update Per Area

Pulumi is typically run via Makefile targets from the repo root.

Login:

make pulumi-login

🔑 Provider Credentials

Pulumi also needs the provider credentials for the area being previewed or updated:

Area Required local access
apps Kubernetes access through the current kubeconfig context
cf CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID
do DIGITALOCEAN_TOKEN
gh GITHUB_TOKEN

Preview/update:

make area=cf pulumi-preview
make area=cf pulumi-update

Supported areas for these targets:

The Makefile runs Pulumi with:

That working directory matters because the programs read <area>.hjson via a relative path.

Other stack operations:

make area=cf pulumi-refresh
make area=cf pulumi-cancel
make area=cf pulumi-delete

[!WARNING] pulumi-update, pulumi-refresh, and pulumi-cancel affect remote infrastructure or stack state. The Make targets pass --yes, so treat the preview as the confirmation step unless you are recovering from a known failed operation.

[!CAUTION] pulumi-delete runs pulumi stack rm --force; it removes Pulumi stack state and can orphan managed resources if they still exist.

🧭 Areas

📦 Applications (area/apps)

Deploys Kubernetes applications described in area/apps/apps.hjson.

⚙️ Configure

See:

This file uses the Kubernetes message in api/infraops/v2/service.proto.

Apps also need an application config file at:

For example, the bezeichner app in namespace lean reads area/apps/lean/bezeichner.yaml. The apps Pulumi program turns that file into a Kubernetes ConfigMap entry named <app>.yaml, then mounts it into the container at /etc/<app>/<app>.yaml.

Only applications listed in area/apps/apps.hjson are deployed. The tracked files under area/apps/fiction/ are example app configs and are not used unless a matching app entry references that namespace/name.

Adding an application to apps.hjson only wires the Pulumi resources. To include it in release rollout, verify, load, and kubescape workflows, also update area/apps/release with the app name and its per-app helper commands, including any request payload fixture needed by load testing.

The abgleich app is a worker, so it has no Service, Ingress, or public DNS record. Its verify helper waits up to 120 seconds for Kubernetes rollout readiness, and its load helper skips HTTP load testing. Kubernetes still probes the worker’s internal HTTP liveness and readiness endpoints.

🏗️ Install / Setup

Save the cluster kubeconfig if your local context is not already configured:

make -C area/apps save-config

Prepare the local app namespace/helper resources:

make -C area/apps setup

Before deploying Abgleich, create abgleich-secret in the existing lean namespace from a local GitHub token file. github_token_file must be a nonempty file path:

make -C area/apps abgleich-secret github_token_file=/absolute/path/to/github-token

This explicit creation step is separate from setup and requires kubectl with the intended cluster context. It creates the github-token key using --from-file without printing its contents; creation fails if the Secret already exists. Abgleich’s secrets list mounts it read-only at /etc/secrets/abgleich/github-token. area/apps/lean/abgleich.yaml reads that file through github.token and configures the worker to reconcile auto-merge pull requests in alexfalkowski/infraops every five minutes. The token needs pull request write access and permission to update repository contents. Credentials are read once at startup; restart the worker after replacing the token.

Apply the Pulumi resources described by apps.hjson:

make area=apps pulumi-update

🧪 App Helpers

The default rollout, verify, load, and kubescape helper targets are release-aware. They inspect the app release diff and target only apps with version-only changes in area/apps/apps.hjson; when the diff is not a release-only app change, they fall back to all supported apps.

Supported apps are the apps explicitly wired in area/apps/release. The API verify helpers use fail-fast curl requests with bounded connection and total request time, and the API load helpers run Vegeta and fail unless every request succeeds. Load request payloads live with the app fixtures under area/apps/<namespace>/.

The release helper compares APPS_RELEASE_BASE to APPS_RELEASE_HEAD when those environment variables are set. Otherwise it uses the current commit’s parent, falling back to origin/master when the parent is unavailable. In CI, area/apps/release halt-ci halts the general build for a release-only apps version bump so the apps update workflow can run the targeted Pulumi update, verification, load, and live-resource lint path. Set APPS_RELEASE_DRY_RUN=true to print the release helper action instead of running it.

make -C area/apps rollout
make -C area/apps verify
make -C area/apps load

Use the *-all targets when you want to cover every supported app explicitly:

make -C area/apps rollout-all
make -C area/apps verify-all
make -C area/apps load-all

The load* targets write ignored vegeta result files under area/apps/lean/*.bin.

Per-app commands are exposed by the release helper:

cd area/apps
./release verify-web
./release load-bezeichner
./release rollout-standort
./release kubescape-web

The lint target requires a working cluster context. kube-score always scans live resources from the lean namespace, while kubescape follows the release-aware workflow: it scans each changed app Deployment and its resolved container image for release-only version changes and otherwise scans the whole namespace. The kube-score helper intentionally ignores the host pod anti-affinity check and fails when Kubernetes discovery or manifest collection returns no live resources.

🗑️ Delete

[!CAUTION] This deletes the lean namespace through kubectl.

make -C area/apps delete

⬆️ Update an Application Version (bump tool)

Build:

make build-bump

Update a single app version in config. This is an internal helper used by automation, and the -v value is expected to be a semantic version:

./bump -n bezeichner -v <version>

By default it edits area/apps/apps.hjson. Override path:

./bump -n bezeichner -v <version> -p area/apps/apps.hjson

[!TIP] Run ./format -k apps after edits to keep config normalized.

☁️ Cloudflare (area/cf)

Manages Cloudflare resources using Pulumi’s Cloudflare provider:

Config:

🌐 Zone Models

The shared Cloudflare zone baseline enables HTTPS, the code-defined minimum TLS setting, aggressive caching, HTTP/3, and HTTP/2 prioritization, disables email obfuscation, and applies the SSL mode for the zone model.

balancer_zones create a Cloudflare zone plus proxied A records for each configured record_names entry under the zone domain. When ipv6 is configured, they also create proxied AAAA records. Balancer zones use the shared baseline with SSL mode full.

balancer_zones: [
  {
    name: example
    domain: example.com
    ipv4: 203.0.113.10
    ipv6: 2001:db8::10
    record_names: [
      api
      web
    ]
  }
]

page_zones create a Cloudflare zone plus a proxied www.<domain> CNAME to host. Page zones use the shared baseline with SSL mode strict.

page_zones: [
  {
    name: docs
    domain: example.com
    host: example.github.io
  }
]

🪣 R2 Buckets

R2 buckets are optional. Omit zone to create only the bucket. Include zone to create an enabled R2 custom domain using the code-defined TLS minimum:

buckets: [
  {
    name: assets
    region: EEUR
    zone: {
      id: "<cloudflare-zone-id>"
      domain: assets.example.com
    }
  }
]

🔑 Required Environment Variables

Cloudflare Pulumi runs require:

🌊 DigitalOcean (area/do)

Manages DigitalOcean resources using Pulumi’s DigitalOcean provider:

Config:

📐 Cluster Defaults

The DigitalOcean program creates the cluster VPC and then provisions the cluster with fixed operational defaults:

Cluster resource values map to node capacity:

Unknown or empty values fall back to "small".

🧱 Manual Prerequisites (DigitalOcean UI)

Some items may be created manually depending on account setup:

Name Description
lean-thoughts All experiments for lean-thoughts.

The Pulumi program creates the VPC used by the cluster and attaches the cluster to it.

⬆️ Kubernetes Cluster Upgrades

Cluster version is pinned in code:

Guidance:

🐙 GitHub (area/gh)

Manages GitHub resources using Pulumi’s GitHub provider:

Config:

This area was inspired by:

🧱 Repository Baseline

Every managed repository receives a fixed baseline in addition to the HJSON fields: merge commits and rebase merges are disabled, squash merge, auto-merge, update branch, auto-init, branch deletion on merge, issues, projects, wiki, secret scanning, push protection, and vulnerability alerts are enabled, and web commit signoff is disabled. These settings are intentional and are not configurable in HJSON.

Every non-archived managed repository also receives an automerge label with color 0E8A16 and description “Eligible for automated merging.” Archived repositories are read-only and are skipped for label creation.

🧬 Repository Templates

template creates a repository from another GitHub template repository and requires both owner and repository. is_template is separate: it marks the resulting repository as a reusable template.

is_template: true
template: {
  owner: alexfalkowski
  repository: go-service-template
}

🛡️ Branch Protection Baseline

Every repository must list at least one checks entry. Those values become the required status check contexts for strict branch protection on master; an empty list fails during provisioning.

Branch protection also enforces linear history and intentionally requires zero approving PR reviews because required status checks are the primary merge gate for this solo-maintainer workflow.

🪜 Repository Creation Caveat (2-step enablement)

Some repository features may require a two-step approach: create the repository first, then enable features in a follow-up change. This avoids timing issues around initial default branch creation.

📄 GitHub Pages

First change: disable Pages (or omit pages config):

pages: { enabled: false }

Second change: enable Pages:

pages: { enabled: true }

Optional CNAME:

pages: {
  enabled: true
  cname: www.yoursite.com
}

Pages uses legacy GitHub Pages sourced from master; branch, path, and build type are not configurable in HJSON.

👥 Collaborators

When enabled, collaborator management grants push permission to lean-thoughts-ci on alexfalkowski/<repository>.

First change:

collaborators: { enabled: false }

Second change:

collaborators: { enabled: true }

If the pipeline fails due to timing, a rerun often succeeds.

⚓ Kubernetes Add-ons (area/k8s)

This is not a Pulumi area. It contains cluster add-ons installed via helm/kubectl. These targets are intended to be run manually from an operator workstation, not from CI.

[!CAUTION] Run this only after you have a Kubernetes cluster, a kubeconfig context for that cluster, and the required local secrets.

Save the cluster kubeconfig if your local context is not already configured:

make -C area/k8s save-config

Setup:

make -C area/k8s setup

The nginx ingress add-on uses the Helm values in area/k8s/nginx/values.yaml. Those values restrict the DigitalOcean load balancer allow rules to Cloudflare CIDR ranges, so the ingress is intended for Cloudflare-proxied origin traffic. Direct clients or non-Cloudflare DNS paths may be blocked at the load balancer.

Delete:

[!CAUTION] This deletes the nginx-ingress, circleci, and metrics-server namespaces through kubectl.

make -C area/k8s delete

Useful debugging:

make -C area/k8s pods

🔑 Required Environment Variables

The default make -C area/k8s setup path installs the CircleCI release agent along with nginx ingress and metrics-server.

[!IMPORTANT] CIRCLECI_K8S_TOKEN is required when running the default k8s setup target.

Because the Makefile is a local-operator workflow, the CircleCI release-agent token is passed directly to Helm rather than through a CI secret-handling path.

🗂️ Repository Structure

This site is open source. Improve this page.