Why Harness Was a No-Go for My Home Lab
I have used Jenkins at my day job for a long time, and more recently self-hosted Harness, which has given me no problems there. So I tried Harness in my home lab Kubernetes cluster for a bit. My home lab has an RKE2 cluster managed with Rancher, with Fleet for GitOps and Harbor as the image registry. I tried two versions of Harness: the SaaS free plan, and Harness Open Source. Neither one could run my builds, so I went back to Jenkins.
Harness SaaS free plan#
On the SaaS plan, Harness hosts the UI and the pipeline engine, and a Harness Delegate runs in my cluster. The delegate polls Harness for work and starts each build stage as a pod in the cluster. Builds that ran on my own nodes through the delegate used no Harness build credits, so the credit limits never came into play, but these other limits did:
Run time caps#
The free plan stops any stage at 1 hour and any pipeline at 2 hours. These caps override whatever timeouts the pipeline, stage, or step sets. I found this out the hard way.
My largest pipeline runs the test suite and a coverage build for a Rust project. With every check in one stage, it was killed at the 1 hour mark partway through coverage. I split each check into its own stage and ran the three in parallel, which got each stage under an hour. Any growth in the test suite would have put it back over, and the Harness free plan has no setting that raises the cap.
Default resource limits on Run steps#
A Harness Run step with no explicit limits gets about 500 MiB of memory and less than one CPU. My Rust builds were killed under those limits. Every step needed its CPU and memory limits set by hand. I gave mine 8 GiB and 6 or 8 CPUs.
The clone step needs AVX2#
Harness's Git Clone step image is built for x86-64-v3, so it requires AVX2 in the CPUs. On my older CPUs it failed with:
Fatal glibc error: CPU does not support x86-64-v3
Only two of my nodes have AVX2. I labeled them and pinned every Harness stage to them with a node selector, which left the other nodes unused for CI.
It depends on the internet uplink#
The delegate gets its work from app.harness.io. If my internet connection were to go down, no pipeline could run, even though the code, the cluster, and the registry are all on my local network.
The delegate runs as cluster-admin#
The delegate Helm chart binds its service account to cluster-admin. A pipeline run from Harness's servers could change anything in my cluster.
Unpublished limits#
The free plan's limit on CD services is not in Harness's public docs. The number is visible only on the account's subscription page, so I could not plan around it before signing up. Again, I found out the hard way.
Harness Open Source#
Harness Open Source, which used to be called Gitness, is one self-hosted container under the Apache 2.0 license. It has Git hosting, YAML pipelines, and an artifact registry, with no account and no usage limits. It would run on my own hardware, so the run time caps and the internet dependency would have gone away. I had planned to switch to it but dropped the plan for several reasons.
Steps run through a Docker daemon#
Harness Open Source runs every pipeline step as a container through a Docker daemon. It cannot start Kubernetes pods. My RKE2 nodes run containerd and have no Docker daemon.
To run it in the cluster, its pod would have needed a Docker daemon of its own, which means a privileged docker:dind (Docker-in-Docker) sidecar. Every build step would then have been a container inside that one Docker daemon, inside one pod, on one node. The Kubernetes scheduler would have seen one pod but none of the steps.
No node selectors or per-step resource limits#
Since the scheduler never sees the steps, Harness Open Source ignores node_selector and per-step resource limits. My builds need both. Each stage has to land on a node with AVX2 and with its warm build cache, and each stage needs its CPU and memory reserved so that three parallel stages do not all land on one node. The cache is a hostPath volume on each build node, which also has to be mounted into the step's pod.
No GitHub triggers#
Harness Open Source triggers builds only for repositories it hosts. A lot of my public repositories are on GitHub, and it could not start a build from a GitHub push.
Back to Jenkins#
Jenkins runs in the cluster with no run time caps. Its Kubernetes plugin runs each build as a pod, with a node selector, CPU and memory limits, and hostPath volumes. Multibranch jobs scan GitHub and build any branch with a Jenkinsfile. Harness works fine for me at work, but in my home lab cluster it was a failed experiment that wasted a lot of my time.