migrieren

Gopher CircleCI codecov Go Report Card Go Reference Stability: Active

🧭 Migrieren

Migrieren is a Go service that runs database schema migrations through a gRPC API with an HTTP RPC façade.

The service wraps golang-migrate/migrate so callers can ask it to migrate a named database to a target version without embedding migration logic into every application.

🧰 What the service does

🚚 Supported drivers

The currently wired drivers are defined in code:

Database URLs use the pgx5:// scheme in config and secrets. Internally the service rewrites that to the driver URL format expected by the Postgres migrate driver.

[!NOTE] Driver support comes from the drivers imported by this service. Adding another source or database kind requires wiring the driver in code, not only changing YAML.

✅ Prerequisites

Initialize the submodule before relying on make:

git submodule sync
git submodule update --init

If you hit vendoring errors such as “inconsistent vendoring”, refresh dependencies with:

make dep

🚀 Quick start

Install dependencies, build the binary, and start the server with the checked-in development/test config:

make dep
make build
cd test
../migrieren server -config file:.config/server.yaml

This builds ./migrieren in the repository root. The checked-in test config is run from test/ because its file:... references point at test/secrets/ and its file migration sources are relative to that working directory.

[!WARNING] The checked-in test config points Postgres at localhost:5433 and OTLP tracing at http://localhost:4318/v1/traces. Start the local feature-test services or provide your own config before expecting migrations, health checks, and tracing to succeed. In the feature harness, localhost:5433 is a nonnative fault-injection proxy to a backing Postgres instance on localhost:5432; both ports must line up with either the local harness or your replacement config.

If you use this repository’s shared local Docker environment, start and stop it with:

make start
make stop

Those targets manage the shared sibling Docker checkout and may require SSH access to that repository. If you do not use them, provide an equivalent local Postgres database named test with user/password test:test on localhost:5432; the feature harness exposes it to the service through the proxy on localhost:5433.

For live reload during local development:

make dev

make dev uses air and starts the same server command with test/.config/server.yaml.

[!TIP] Use make help to list the Make targets exposed by this checkout before reaching for direct tool commands.

🔎 How Migrieren resolves a migration

The API accepts a logical database name such as postgres. The service then:

  1. Looks that name up in migrate.databases.
  2. Reads the configured source value through its filesystem abstraction.
  3. Reads the configured url value through the same abstraction.
  4. Passes the resolved source URL and database URL to the core migrator.

That means source and url in the YAML config are usually references to files or other resolvable inputs, not the final literal migration/source strings themselves.

The resolver accepts the shared go-service source-string forms:

⚙️ Configuration

Migrieren is configured through the server command input file. The repository includes a representative config at test/.config/server.yaml.

🧩 Minimal configuration shape

At minimum, you need:

Example:

environment: development
health:
  duration: 1s
  timeout: 1s
migrate:
  databases:
    - name: postgres
      source: file:secrets/source
      url: file:secrets/pg
transport:
  http:
    address: tcp://:11000
    timeout: 5s
  grpc:
    address: tcp://:12000
    timeout: 5s

health.duration sets how often the service evaluates its noop, online, and per-database health registrations. health.timeout bounds the online check and each database target’s source validation and database ping. Migration request timeouts are configured separately by the transport and database URL options.

🗄️ Migration database entries

migrate.databases must contain at least one entry. Each entry must have a unique name and all of these fields:

migrate.logs is optional. migrate.logs.max bounds how many migration log lines each Migrate/ApplyMigrations response returns. Omitting it or setting it to 0 selects the default of 100; only positive values set a custom cap, and there is no value that disables returned logs. Negative values are invalid. When the limit is exceeded, the oldest lines are dropped and the first returned entry is a migration logs truncated (showing last N of M) marker.

In the test setup, the referenced secret files contain the values consumed by the migrator. The GitHub source file is generated locally and is not checked in:

# test/secrets/source
file://migrations

# test/secrets/github (generated by make github-source)

# test/secrets/pg
pgx5://test:test@localhost:5433/test?sslmode=disable&x-migrations-table=migrieren_schema_migrations&x-statement-timeout=5000&x-multi-statement=true&x-multi-statement-max-size=2097152

So a config entry like this:

migrate:
  databases:
    - name: postgres
      source: file:secrets/source
      url: file:secrets/pg

ultimately migrates Postgres using the file://migrations source and the pgx5://... database URL resolved from those files.

🐙 GitHub migration sources

GitHub sources use this URL shape:

github://[user:personal-access-token@]owner/repository[/path][#ref]

The migration directory is listed when the source opens. Keep the token in a local or CI secret; make features generates the ignored source file before running the feature suite.

🐘 Postgres URL options

Postgres targets use pgx5://... URLs. In addition to normal Postgres connection parameters, Migrieren recognizes these migration-driver query parameters:

Malformed boolean or integer values reject the configured database URL.

The upstream golang-migrate pgx driver can acquire its migration advisory lock or inspect migration metadata while constructing or using the database driver. Those upstream paths do not consistently expose Migrieren’s request context, so request cancellation and deadlines should not be treated as a strict bound for every advisory-lock wait or status inspection. Migrieren relies on the upstream driver behavior here rather than maintaining a local pgx driver fork, and can tighten this once context-aware migrate v5 driver APIs are available.

🔐 Authentication and access control

Migrieren can require an authenticated, authorized caller on both the gRPC API and the HTTP façade. This reuses the shared go-service transport wiring:

The verification audience is bound to the request so a token cannot be replayed against another endpoint:

The transport-scoped access objects are prefixed by transport, for example http:POST /migrieren.v1.Service/Status and grpc:/migrieren.v1.Service/Status.

The checked-in test config wires the jwt kind on both transports. A JWT’s kid selects its trusted public key and its sub is the verified subject that the policy is evaluated against. The test config trusts only the migrieren signing key; the feature harness issues tokens with either the migrieren or guest subject:

Each configured public key represents a trusted issuer that can assert any subject, so only configure issuer-controlled keys there; authorization belongs in the access policy.

transport:
  access:
    model: file:secrets/access_model
    policy: file:secrets/access_policy
  http:
    address: tcp://:11000
    token:
      kind: jwt
      jwt:
        iss: migrieren
        key: migrieren
        exp: 1h
        keys:
          migrieren:
            public: file:secrets/jwt_migrieren.pub
  grpc:
    address: tcp://:12000
    token:
      kind: jwt
      jwt:
        iss: migrieren
        key: migrieren
        exp: 1h
        keys:
          migrieren:
            public: file:secrets/jwt_migrieren.pub

The secrets/access_policy file grants the migrieren subject invoke on every method for both transports. A JWT issued by the trusted signer with a guest subject authenticates yet is denied because that subject is absent from the policy. The harness also verifies that a token from its untrusted guest key cannot impersonate the privileged subject.

The Ruby feature harness mints matching tokens with nonnative (Migrieren.auth_token) and attaches them automatically: an HTTP post override adds a route-scoped Bearer header, and a gRPC client interceptor adds route-scoped authorization metadata. See test/features/v1/transport/*/auth.feature.

🧪 About the checked-in test config

test/.config/server.yaml intentionally contains both valid and invalid database definitions:

[!IMPORTANT] The checked-in config is a test fixture. Use it for local development and feature coverage, but do not treat it as a healthy production config.

That is why the checked-in config is useful for development and testing, but it is not a “healthy production example” as-is.

🔌 API

The protobuf contract lives at api/migrieren/v1/service.proto.

🔷 gRPC

The service exposes:

Request:

Response:

Conceptual request:

database: "postgres"
version: 1

⏫ Apply all pending migrations

migrieren.v1.Service/ApplyMigrations applies all pending up migrations for a configured database and reports the resulting migration version. If the database is already current, the request succeeds as a no-op and reports the current version.

Request:

Response:

Conceptual request:

database: "postgres"

[!NOTE] ApplyMigrations uses the upstream golang-migrate v4 all-pending up path. Request cancellation and deadlines are honored around the migration call, but strict cancellation still depends on upstream context-aware driver support in some database paths.

Migrieren intentionally does not expose step-based up/down, rollback-by-step, all-down, or version-zero migration APIs. Use Migrate when the caller knows the target version, or ApplyMigrations to converge to the latest available up migration.

🧭 Migration plan

migrieren.v1.Service/PlanMigrations reports current status and the migration versions that would run for a configured database without applying migration files. It opens the configured source to discover available migration versions, but it does not read migration bodies, execute migrations, or expose configured source strings, database URL strings, resolved URLs, or secret values.

Request:

Response:

Conceptual request:

database: "postgres"
target_version: 1

[!NOTE] PlanMigrations is informational and does not reserve the migration source or database for a later request. Migrieren intentionally does not expose step-based up/down, rollback-by-step, all-down, or version-zero migration APIs. Use Migrate for an explicit target version and ApplyMigrations for latest. An explicit plan reports the same safe migration failure as Migrate when the current database state is dirty, its requested target is absent from the source, or a recorded current version is absent from the source.

🔎 Migration status

migrieren.v1.Service/Status reports the current migration state for a configured database without applying migration files.

Request:

Response:

Conceptual request:

database: "postgres"

[!NOTE] Status is non-migrating, but strict request cancellation depends on upstream golang-migrate v4 context support. Some database-driver inspection paths do not accept Migrieren’s request context until the upstream project provides context-aware driver APIs.

Migrieren intentionally does not expose a public force-version or dirty-state repair RPC. Dirty status means an operator must investigate the failed migration and repair the database state outside this service before any migration metadata is changed. A remote repair endpoint would be destructive metadata surgery that Migrieren cannot validate generically, because the service cannot prove the real schema or data was repaired correctly for every application.

📋 Database discovery

migrieren.v1.Service/ListDatabases reports configured logical database names from migrate.databases in config order.

Request:

Response:

The response does not include configured source strings, configured url strings, resolved database URLs, resolved migration source URLs, or secret values.

🌐 HTTP façade

The HTTP RPC façade exposes the same operation at:

The checked-in test config requires a route-scoped Bearer value for application RPCs. After make dep, run the examples below from test/ and define this helper once; it uses the throwaway test signing key to mint a header for the exact route passed to it without placing the token in curl’s process arguments:

http_authorization() {
  bundle exec ruby -Ilib -rnonnative -rmigrieren \
    -e 'puts "Authorization: #{Migrieren.http_authorization(ARGV.fetch(0))}"' "$1"
}

If your own config omits token verification and access control, the Authorization header is unnecessary.

Example:

POST http://localhost:11000/migrieren.v1.Service/Migrate
Content-Type: application/json
Authorization: Bearer <route-scoped-token>

{
  "database": "postgres",
  "version": 1
}

Copy-paste request against the local HTTP façade:

http_authorization '/migrieren.v1.Service/Migrate' |
  curl -sS -X POST http://localhost:11000/migrieren.v1.Service/Migrate \
  -H 'Content-Type: application/json' \
  --header @- \
  -d '{"database":"postgres","version":1}'

Copy-paste apply-all request against the local HTTP façade:

http_authorization '/migrieren.v1.Service/ApplyMigrations' |
  curl -sS -X POST http://localhost:11000/migrieren.v1.Service/ApplyMigrations \
  -H 'Content-Type: application/json' \
  --header @- \
  -d '{"database":"postgres"}'

Copy-paste plan request against the local HTTP façade:

http_authorization '/migrieren.v1.Service/PlanMigrations' |
  curl -sS -X POST http://localhost:11000/migrieren.v1.Service/PlanMigrations \
  -H 'Content-Type: application/json' \
  --header @- \
  -d '{"database":"postgres","target_version":1}'

Copy-paste status request against the local HTTP façade:

http_authorization '/migrieren.v1.Service/Status' |
  curl -sS -X POST http://localhost:11000/migrieren.v1.Service/Status \
  -H 'Content-Type: application/json' \
  --header @- \
  -d '{"database":"postgres"}'

Copy-paste database discovery request against the local HTTP façade:

http_authorization '/migrieren.v1.Service/ListDatabases' |
  curl -sS -X POST http://localhost:11000/migrieren.v1.Service/ListDatabases \
  -H 'Content-Type: application/json' \
  --header @- \
  -d '{}'

🚦 Error mapping

Transport behavior is intentionally simple:

The core migrator also treats migrate.ErrNoChange as a successful no-op and still returns any accumulated logs.

For Migrate, ApplyMigrations, PlanMigrations, and Status requests that pass request validation and then fail, the server adds safe failure diagnostics when they are available. gRPC exposes them as trailers and HTTP exposes them as response headers without changing the HTTP status code or error body shape. This includes unknown database names, source/database setup and reference-resolution failures, and core migration failures; invalid version values return InvalidArgument/400 before this diagnostic path runs.

HTTP uses the same names as response headers:

💓 Health and observability

When running with the shared service runtime, Migrieren exposes:

The probe contracts are deliberately different:

Surface Checks observed Operational meaning
HTTP /migrieren/livez noop Process liveness only.
HTTP /migrieren/readyz noop Process readiness only; it does not gate on migration dependencies.
HTTP /migrieren/healthz online and every configured database target Public connectivity plus source validation and database reachability.
gRPC health for migrieren.v1.Service noop Service-process health independent of migration dependencies.

The shared online check uses its default public connectivity endpoints. Use healthz for dependency health, not as a liveness probe that should remain stable during an egress or database outage.

There is an important detail in the checked-in test config:

Health probes use internal registration names such as noop and online. Treat those names as reserved for health wiring rather than as migration database names.

The sample test config also enables:

GitHub migration sources have one health-check exception: /migrieren/healthz parses a configured github:// source but does not open the remote repository during the health check. The remote source is opened during migration execution, so GitHub reachability failures can still appear on a migration request after health has reported on the configured target.

🛠️ Development

📋 Common commands

Use make help to list available targets. Common ones are:

make dep
make build
make build-test
make specs
make features
make benchmarks
make lint
make format
make sec
make coverage
make proto-generate
make proto-lint
make proto-breaking
make proto-stale

What those do in this repository:

🧪 Local test harness expectations

The Ruby harness under test/ assumes:

The feature harness process wiring lives in test/nonnative.yml.

🗂️ Repository layout

Key locations:

🧬 Protobuf workflow

The API contract is managed with buf.

Generate code:

make proto-generate

Lint and breaking-change checks:

make proto-lint
make proto-breaking
make proto-stale

make proto-stale verifies generated protobuf outputs are current; CI runs it after the breaking-change check.

Generation is configured in api/buf.gen.yaml and currently writes:

[!CAUTION] Do not hand-edit generated protobuf stubs. Update api/migrieren/v1/service.proto and regenerate instead.

🤝 Notes for contributors

This site is open source. Improve this page.