Skip to content

About

Spin up an autoscaling stack of Buildkite Agents on Kubernetes

Topics

Resources

Code of conduct

Stars

112 stars

Watchers

19 watching

Forks

Latest commit

 

History

2,023 Commits

Folders and files

Repository files navigation

Buildkite Agent Stack for Kubernetes

Build status

Overview

The Buildkite Agent Stack for Kubernetes (also known as agent-stack-k8s) is a Kubernetes controller that uses the Buildkite Agent API to watch for scheduled jobs assigned to the controller's queue.

Note

Starting with v0.28.0, the Buildkite GraphQL API is no longer used. If you are upgrading from an older version, your GraphQL-enabled token can be safely removed from your configuration or Kubernetes secret. Only the Agent token is required.

Requirements

Usage

The simplest way to launch a stack on the default queue:

helm install agent-stack-k8s oci://ghcr.io/buildkite/helm/agent-stack-k8s \
    --set agentToken=<buildkite-agent-token>

To specify a non-default queue:

helm install agent-stack-k8s oci://ghcr.io/buildkite/helm/agent-stack-k8s \
    --set agentToken=<buildkite-agent-token> \
    --set config.queue=arm64

Multiple independent controllers serving the same queue

Most installations need no controller ID configuration. If multiple independent controller installations service the same queue, give each installation a distinct controller ID (its Agent API stack key), stable across restarts and upgrades.

With Helm, use distinct release full names or set BUILDKITE_K8S_STACK_CONTROLLER_ID through controllerEnv to leave Kubernetes resource names unchanged. Non-Helm deployments can set the same environment variable or pass --id.

Full instructions can be found in the documentation.

Dispatch and shutdown

The controller reserves jobs only after local dispatch capacity is available, combining admitted jobs into reservation batches. With job acquisition tokens enabled, it also batches token requests and paces all attempts, including retries, at up to 10 requests per second per controller. This is not a distributed limit across controller replicas. Transient token failures retain the job for retry until it succeeds, its reservation expires, or dispatch is canceled; permanent rejections are not retried.

Fetching job information retries HTTP 404s within the existing five-attempt budget, respecting Retry-After and the reservation deadline. Persistent fetch failures can still leave a reservation held until it expires.

On shutdown, the controller stops accepting work and allows up to 20 seconds to finish creating Kubernetes Jobs and their required Secrets, followed by up to 5 seconds for stack deregistration. It does not wait for workloads to finish. Allow at least Kubernetes' default 30-second termination grace period. A forced kill or a dispatch that exceeds the drain deadline can still leave reservations held until they expire; deregistering the stack does not release them.

Documentation

Comprehensive documentation for the Buildkite Agent Stack for Kubernetes controller can be found in the Agent Stack for Kubernetes section of the Buildkite Docs.

Development

For guidelines and requirements regarding contributing to the Buildkite Agent Stack for Kubernetes controller, please see the Development guide.

About

Spin up an autoscaling stack of Buildkite Agents on Kubernetes

Topics

Resources

Code of conduct

Stars

112 stars

Watchers

19 watching

Forks

Releases

Packages

Used by

Contributors

Languages