At Fenerum, we used Spinnaker for Kubernetes deployments for many years. It served us well, and one of the things I particularly liked was how sophisticated its pipelines could be. We could model dependencies between stages and build fairly elaborate deployment flows.
Over time, though, operating Spinnaker became increasingly difficult to justify.
We managed it with Halyard, and upgrades were painful. Roughly every month, someone had to work through a manual upgrade process, read the changelogs and deal with incompatibilities. When Halyard was deprecated and the direction moved toward Kustomize, we were facing a migration in which much of our setup would need to change anyway.
That prompted us to step back and ask a more useful question: if we were going to put significant work into our deployment setup, what did we actually need today?
What we really used the pipeline for
Spinnaker could do a lot, but the essential requirement behind most of our pipeline complexity was much smaller: before a new application version went live, we needed to run database migrations and allow the deployment to proceed only if they succeeded.
Kubernetes already provides the primitives needed to model that behavior. A Job can run the migration and report success or failure, and the remaining resources can depend on that result. What Kubernetes does not provide is a generic preDeploy field. That part is our own abstraction, built with KRO.
Once we separated the behavior we needed from the particular way Spinnaker implemented it, the replacement became much simpler. We did not need to reproduce a sophisticated pipeline engine. We needed Kubernetes resources, a small amount of orchestration and a clean way for engineers to describe an application.
Making Kubernetes manifests manageable with KRO
Plain Kubernetes manifests are powerful, but they can also be repetitive and difficult to read. A fairly ordinary application can involve a Deployment, Service, migration Job, health checks, environment sources, observability configuration and several sets of labels and annotations. Much of that YAML is boilerplate, and the parts that matter to an application developer can be hard to find among it.
KRO lets us define that complexity once and expose a much smaller application-specific API on top of it. We can provide sensible defaults while retaining the option to configure the things that genuinely vary between applications.
This is an anonymized example of one of our application definitions. The names, domain, registry, commands and values have all been changed:
apiVersion: example.com/v1alpha1
kind: App
metadata:
name: example-app-staging
namespace: example-app-staging
annotations:
example.com/deploy: auto # auto | pr
spec:
app: example-app
env: staging # staging | production; also the Datadog environment
image:
repository: ghcr.io/example-org/example-app
tag: sha-0123456789abcdef0123456789abcdef01234567
envFrom:
configMaps: [example-app-staging-env]
secrets: []
datadog:
apm: false
logSource: python
preDeploy:
command: [python, manage.py, migrate, --noinput]
resources:
requests: {cpu: 100m, memory: 256Mi}
limits: {memory: 512Mi}
workloads:
- name: web
port: 8000
healthPath: /health/
healthHost: staging.example.com
replicas: 2
command: [gunicorn, example.wsgi:application]
resources:
requests: {cpu: 100m, memory: 512Mi}
limits: {memory: 1Gi}
- name: scheduler
recreate: true
command: [python, manage.py, run_scheduler]
That is essentially the contract an application has with our platform.
Looking at this file, I can immediately see the image we are running, whether deployment is automatic, the migration that must succeed first, the web process, the scheduler, replicas, health checks and resource requirements. I do not have to read through several generated Kubernetes resources to understand how the application runs.
I think of it as something like a more capable Heroku Procfile. It is still Kubernetes underneath, and engineers can inspect the resulting resources with normal Kubernetes tooling. KRO has not replaced Kubernetes with a separate platform. It has given us a concise internal API and removed the repetitive parts.
Adding an application to the cluster is now often just a small YAML file in a pull request.
Fleet as the GitOps layer
We keep these definitions in a Fleet repository that manages both staging and production. That repository is the source of truth for what runs in our clusters, so changing an application version or rolling out a shared Helm chart is an ordinary Git change.
Our application repositories use GitHub Actions to build container images and push them to GitHub Container Registry. After a successful build, the workflow notifies the Fleet configuration repository. A workflow there updates the application's image tag to the new commit-based tag.
What happens next is controlled by the annotation in the application manifest:
metadata:
annotations:
example.com/deploy: auto # auto | pr
For an application configured with auto, the tag update is committed and Fleet deploys it. For an application configured with pr, the workflow opens a pull request instead, leaving the promotion to be reviewed and merged.
This gives us the two modes we need without constructing separate deployment pipelines for each application. The desired version remains visible in Git, and Fleet handles reconciling it to the cluster.
Why not Argo CD?
The obvious question is why we did not use Argo CD. It is widely used, and I am sure it works well for many teams. We looked at it briefly, and there was no fundamental reason it could not work for us.
My candid first impression, though, was that it appeared unnecessarily complex for what we were trying to achieve. I also found the interface dated. Coming from Spinnaker, we were not looking to replace one deployment platform with another system carrying its own large set of concepts, a separate GUI and another login.
We wanted fewer abstractions and less ceremony. Fleet gave us GitOps while keeping us close to Kubernetes. Engineers use the Kubernetes access and tooling they already have, and they can see the actual Jobs, Deployments, Pods and Services involved. There is no separate deployment UI that becomes the only place where the system makes sense.
That closeness is valuable for another reason: it encourages everyone working on deployments to understand Kubernetes, which is the underlying technology in any case. We have an abstraction for the repetitive configuration, but it does not try to hide the platform completely.
This is a personal preference rather than a general verdict on Argo CD. For our team and our requirements, Fleet plus KRO was the smaller and clearer solution.
Where we ended up
The move has removed the cost of running our Spinnaker cluster and, more importantly, the maintenance surrounding it. We no longer have a monthly Spinnaker upgrade task involving Halyard, changelogs and compatibility problems.
Our entire deployment configuration for staging and production lives in one Fleet repository. Application manifests describe the details we care about, KRO expands them into the necessary Kubernetes resources, and Fleet reconciles those resources to the clusters. GitHub Actions connects image builds to version updates, with a small annotation deciding whether the update is automatic or reviewed through a pull request.
There is still complexity in the system. Database migrations must still be modeled correctly, generated resources still need to be understood, and Kubernetes still needs to be operated. The difference is that this complexity now lives in places that fit it: reusable platform definitions, standard Kubernetes resources and Git history, rather than a separate deployment platform we have to maintain.
So far, the setup has been working smoothly. We kept the capability that mattered to us, removed quite a lot that did not, and ended up with a deployment process that is easier to understand and cheaper to run.