TL;DR: Streamlining your CI/CD pipelines with Dagger.io can future proof your application delivery process and speed up the time to deliver.
CI/CD (Continuous Integration/Continuous Deployment) pipelines have become a standard approach for streamlining software development. Yet, it can be both difficult and time-consuming to manage and optimise these pipelines. Furthermore, switching between different CI/CD engines is costly, thus it rarely happens. This is where Dagger.io, a programmable CI/CD engine that runs your pipelines anywhere, comes in. By utilising Dagger.io and writing your pipelines the “Dagger way”, you will be able to build portable pipelines that can run in any CI/CD engine.
Dagger.io is a FOSS (Free and open-source software) created and maintained by key people from the Docker team, Solomon Hykes, Sam Alba and Andrea Luzzardi.
Advantages of Dagger.io
By using Dagger.io we can streamline any pipeline, and make it able to run wherever we want. This is the next generation of CI tooling and will be able to replace old pipelines written solely for fx. Jenkins. Dagger.io will help not get vendor-locked in the future of CI tooling. Shown below is a small example of a pipeline written in GO utilising the Dagger SDK. The example starts by starting a new container. It then uses the container to test the code, build it and lastly publish it to a container registry. The blog continues under the code block
func main() {
ctx := context.Background()
// initialize Dagger client
client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
if err != nil {
panic(err)
}
defer client.Close()
// use a node:20-slim container
// mount the source code in the container at /src in the container
source := client.Container().
From("node:20-slim").
WithDirectory("/src", client.Host().Directory(".", dagger.HostDirectoryOpts{
Exclude: []string{"node_modules/", "ci/", "build/"},
}))
// set the working directory in the new container
// install necessary application dependencies
runner := source.WithWorkdir("/src").
WithExec([]string{"npm", "install"})
// run application tests
test := runner.WithExec([]string{"npm", "test", "--", "--watchAll=false"})
// build the application and output into build/
buildDir := test.WithExec([]string{"npm", "run", "build"}).
Directory("./build")
// use an nginx:alpine container
// copy the build/ directory from the earlier
// publish the resulting container to a registry
ref, err := client.Container().
From("nginx:1.23-alpine").
WithDirectory("/usr/share/nginx/html", buildDir).
Publish(ctx, fmt.Sprintf("ttl.sh/hello-dagger-%.0f", math.Floor(rand.Float64()*10000000))) //#nosec
if err != nil {
panic(err)
}
fmt.Printf("Published image to: %s\n", ref)
}
- Example from Dagger.io Docs
This pipeline is simple and able to be run using the Dagger CLI in any CI/CD engine. GitHub Actions is just one engine to use, and a small example of this can be seen below. Other engines include, but are not limited to; Circle-CI, AWS Codebuild, Gitlab, Jenkins and even locally by just utilising the Docker Engine.
name: 'go ci/cd'
on:
push:
branches:
- main
jobs:
dagger:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Setup Go
uses: actions/setup-go@v4
with:
go-version: '>=1.20'
- name: Install
run: go get dagger.io/dagger@latest
- name: Install Dagger CLI
run: cd /usr/local && { curl -L https://dl.dagger.io/dagger/install.sh | sh; cd -; }
- name: Release and deploy with Dagger
run: dagger run go run ci/main.go
Seeing it in action
Running a pipeline locally can improve consistency and speed up testing significantly. To show how this can be done in practice we at Tech Chapter have created a simple example located in a public GitHub repository:
https://github.com/techchapter/dagger-pipeline-example
When running the pipeline from the repository locally we get a long output and at the bottom it shows us how we can run the exported image it has created:
Output from local Dagger run
$ dagger run go run ci/ci.go
Creating new Engine session... OK!
Establishing connection to Engine... 1: connect
1: > in init
1: starting engine
1: starting engine [0.07s]
1: starting session
1: [0.89s] OK!
1: starting session [0.20s]
1: connect DONE
OK!
7: upload . from 3201439c6656 (client id: l0z9uuhafnpu7dadyiks2zjdj) (exclude: ci/, .devcontainer/) DONE
7: > in host.directory .
8: blob://sha256:610f147de71b9045f9103cace3d061632115f56edf4f18947b2975c230bcc16e DONE
8: > in host.directory .
12: resolve image config for docker.io/library/golang:1.21
12: > in from golang:1.21
12: resolve image config for docker.io/library/golang:1.21 DONE
26: pull docker.io/library/golang:1.21
26: > in from golang:1.21
26: pull docker.io/library/golang:1.21 DONE
25: copy / /app
25: copy / /app DONE
26: pull docker.io/library/golang:1.21
26: > in from golang:1.21
26: pull docker.io/library/golang:1.21 DONE
22: exec go mod tidy
22: [0.11s] go: downloading github.com/go-playground/assert/v2 v2.2.0
22: [0.11s] go: downloading github.com/gin-gonic/gin v1.9.1
22: [0.30s] go: downloading google.golang.org/protobuf v1.32.0
22: [0.30s] go: downloading github.com/mattn/go-isatty v0.0.20
22: [0.30s] go: downloading github.com/gin-contrib/sse v0.1.0
22: [0.30s] go: downloading golang.org/x/net v0.20.0
22: [0.30s] go: downloading github.com/go-playground/validator/v10 v10.16.0
22: [0.30s] go: downloading github.com/pelletier/go-toml/v2 v2.1.1
22: [0.30s] go: downloading github.com/bytedance/sonic v1.10.2
22: [0.30s] go: downloading gopkg.in/yaml.v3 v3.0.1
22: [0.30s] go: downloading github.com/ugorji/go/codec v1.2.12
22: [0.30s] go: downloading github.com/goccy/go-json v0.10.2
22: [0.30s] go: downloading github.com/json-iterator/go v1.1.12
22: [0.41s] go: downloading github.com/go-playground/universal-translator v0.18.1
22: [0.41s] go: downloading github.com/gabriel-vasile/mimetype v1.4.3
22: [0.45s] go: downloading github.com/leodido/go-urn v1.2.4
22: [0.45s] go: downloading golang.org/x/crypto v0.18.0
22: [0.55s] go: downloading golang.org/x/text v0.14.0
22: [0.57s] go: downloading github.com/go-playground/locales v0.14.1
22: [0.61s] go: downloading golang.org/x/sys v0.16.0
22: [0.64s] go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
22: [0.64s] go: downloading github.com/modern-go/reflect2 v1.0.2
22: [0.80s] go: downloading github.com/chenzhuoyu/base64x v0.0.0-20230717121745-296ad89f973d
22: [0.83s] go: downloading golang.org/x/arch v0.7.0
22: [1.06s] go: downloading github.com/twitchyliquid64/golang-asm v0.15.1
22: [1.07s] go: downloading github.com/stretchr/testify v1.8.4
22: [1.07s] go: downloading gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c
22: [1.07s] go: downloading github.com/google/go-cmp v0.5.8
22: [1.07s] go: downloading github.com/davecgh/go-spew v1.1.1
22: [1.17s] go: downloading github.com/klauspost/cpuid/v2 v2.2.6
22: [1.28s] go: downloading github.com/pmezard/go-difflib v1.0.0
22: [1.28s] go: downloading github.com/chenzhuoyu/iasm v0.9.1
22: [1.30s] go: downloading github.com/kr/pretty v0.3.1
22: [1.36s] go: downloading github.com/rogpeppe/go-internal v1.11.0
22: [1.36s] go: downloading github.com/kr/text v0.1.0
22: [3.55s] go: downloading github.com/kr/text v0.2.0
22: exec go mod tidy DONE
21: exec go test ./src/
21: [6.86s] ok github.com/techchapter/dagger-pipeline-example/src 0.004s
21: exec go test ./src/ DONE
20: exec go build -o /build/ping-socket ./src/ DONE
29: resolve image config for docker.io/library/alpine:3.19
29: > in from alpine:3.19
29: resolve image config for docker.io/library/alpine:3.19 DONE
35: pull docker.io/library/alpine:3.19
35: > in from alpine:3.19
35: pull docker.io/library/alpine:3.19 DONE
34: copy /build /
35: pull docker.io/library/alpine:3.19
35: > in from alpine:3.19
35: pull docker.io/library/alpine:3.19 DONE
34: copy /build / DONE
DEV environment: exported file to ./ping-socket.tar
To run use docker load -i ./ping-socket.tar
DEV environment: exported file to ./ping-socket.tar
To run use docker load -i ./ping-socket.tar
If we where to run the same pipeline in GitHub actions we would get a similar output, but instead of a local image output, the pipeline publishes the image to a registry:
Full GitHub Actions log from Dagger run
Run dagger run go run ci/ci.go --env prod
1: connect
1: > in init
1: pulling registry.dagger.io/engine:v0.9.5
1: pulling registry.dagger.io/engine:v0.9.5 [3.85s]
1: starting engine
1: starting engine [3.07s]
Connected to engine 89207bc99e41 (version v0.9.5)
1: starting session
1: [8.93s] OK!
1: starting session [0.19s]
1: connect DONE
3: go run ci/ci.go --env prod
3: ...
8: upload . from fv-az1432-522 (client id: yp5rf7fzk3kku0auquqkxe31i) (exclude: ci/, .devcontainer/) DONE
8: > in host.directory .
8: upload . from fv-az1432-522 (client id: yp5rf7fzk3kku0auquqkxe31i) (exclude: ci/, .devcontainer/) DONE
3: go run ci/ci.go --env prod
3: ...
13: resolve image config for docker.io/library/golang:1.21
13: > in from golang:1.21
13: resolve image config for docker.io/library/golang:1.21 DONE
27: pull docker.io/library/golang:1.21
27: > in from golang:1.21
27: resolve docker.io/library/golang:1.21@sha256:7026fb72cfa9cc112e4d1bf4b35a15cac61a413d0252d06615808e7c987b33a7
27: resolve docker.io/library/golang:1.21@sha256:7026fb72cfa9cc112e4d1bf4b35a15cac61a413d0252d06615808e7c987b33a7 [0.01s]
27: pull docker.io/library/golang:1.21 DONE
26: copy / /app
26: copy / /app DONE
27: pull docker.io/library/golang:1.21
27: > in from golang:1.21
27: pull docker.io/library/golang:1.21 DONE
23: exec go mod tidy
23: [0.08s] go: downloading github.com/gin-gonic/gin v1.9.1
23: [0.08s] go: downloading github.com/go-playground/assert/v2 v2.2.0
23: [0.23s] go: downloading github.com/gin-contrib/sse v0.1.0
23: [0.23s] go: downloading github.com/mattn/go-isatty v0.0.20
23: [0.23s] go: downloading golang.org/x/net v0.20.0
23: [0.26s] go: downloading google.golang.org/protobuf v1.32.0
23: [0.48s] go: downloading github.com/stretchr/testify v1.8.4
23: [0.61s] go: downloading github.com/bytedance/sonic v1.10.2
23: [0.65s] go: downloading github.com/goccy/go-json v0.10.2
23: [0.67s] go: downloading github.com/json-iterator/go v1.1.12
23: [0.67s] go: downloading github.com/go-playground/validator/v10 v10.16.0
23: [0.78s] go: downloading github.com/pelletier/go-toml/v2 v2.1.1
23: [0.78s] go: downloading github.com/ugorji/go/codec v1.2.12
23: [0.78s] go: downloading gopkg.in/yaml.v3 v3.0.1
23: [0.81s] go: downloading golang.org/x/sys v0.16.0
23: [0.83s] go: downloading github.com/davecgh/go-spew v1.1.1
23: [0.83s] go: downloading github.com/pmezard/go-difflib v1.0.0
23: [0.84s] go: downloading github.com/google/go-cmp v0.5.8
23: [0.85s] go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
23: [0.86s] go: downloading github.com/modern-go/reflect2 v1.0.2
23: [0.87s] go: downloading github.com/gabriel-vasile/mimetype v1.4.3
23: [0.88s] go: downloading github.com/go-playground/universal-translator v0.18.1
23: [0.88s] go: downloading github.com/leodido/go-urn v1.2.4
23: [0.89s] go: downloading golang.org/x/crypto v0.18.0
23: [0.94s] go: downloading golang.org/x/text v0.14.0
23: [1.06s] go: downloading github.com/go-playground/locales v0.14.1
23: [1.84s] go: downloading gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c
23: [1.88s] go: downloading github.com/kr/pretty v0.3.1
23: [1.88s] go: downloading github.com/chenzhuoyu/base64x v0.0.0-20230717121745-296ad89f973d
23: [1.88s] go: downloading golang.org/x/arch v0.7.0
23: [1.88s] go: downloading github.com/twitchyliquid64/golang-asm v0.15.1
23: [1.90s] go: downloading github.com/klauspost/cpuid/v2 v2.2.6
23: [1.94s] go: downloading github.com/kr/text v0.1.0
23: [1.94s] go: downloading github.com/rogpeppe/go-internal v1.11.0
23: [1.94s] go: downloading github.com/chenzhuoyu/iasm v0.9.1
23: [2.17s] go: downloading github.com/kr/text v0.2.0
23: exec go mod tidy DONE
22: exec go test ./src/
22: [16.1s] ok github.com/techchapter/dagger-pipeline-example/src 0.005s
22: exec go test ./src/ DONE
21: exec go build -o /build/ping-socket ./src/ DONE
30: resolve image config for docker.io/library/alpine:3.19
30: > in from alpine:3.19
30: resolve image config for docker.io/library/alpine:3.19 DONE
35: pull docker.io/library/alpine:3.19
35: > in from alpine:3.19
35: pull docker.io/library/alpine:3.19 DONE
34: copy /build /
34: copy /build / DONE
Published image to: ttl.sh/ping-socket-5507822@sha256:e335659c6f89315574c4f5245da2a666981e7f0fb902c14d28d2ac1608082748
3: go run ci/ci.go --env prod DONE
Published image to: ttl.sh/ping-socket-5507822@sha256:e335659c6f89315574c4f5245da2a666981e7f0fb902c14d28d2ac1608082748
Conclusion: Taking your CI/CD pipeline to the next level with Dagger.io
To sum up, Dagger.iois an effective solution that can advance your continuous integration and delivery pipeline. By addressing common challenges such as maintaining consistency across environments, managing testing, building and publishing, and allowing for long-term use without the hassle of rewriting from scratch. Dagger.io aids in simplifying the software delivery process by being the new tool for portable CI/CD pipelines. Our best recommendations would be to implement Dagger as your new CI tool and implement it together with a GitOps tool like Argo-CD to reach the highest portability and best streamlined setup.
Read more about Dagger.io and how we can help you implement it at our skills page.
