drone-ci-testing

testing cloud.drone.io

Contents

Contents of this page

• Setups

  • Drone CLI Setup
  • Drone CLI Secrets

• Pipeline Examples

  • Shell Commands in a Step
  • Secrets in Environment Variables
  • Publish to Dockerhub
  • Publish to S3
  • Slack Notifications
  • Telegram Notifications
  • Email Notifications
  • SSH Example
  • SCP Example
  • Rsync Example
  • Volumes: Persistence
  • Volumes: Temporary and Bind Mounts
  • Parallel Builds
  • Docker Remote Tunnel Example
  • Use Services with Pipelines
  • Multipipeline Depends on Steps
  • 3 Times Multiple Pipeline Dependencies with Volumes
  • Full Pipeline Examples

Notes

• Drone clones github repo to /drone/src
• Context of the data gets transferred to each step
• Having each step do hostname >> data.txt, will output this result on the 3rd step:

+ hostname >> data.txt
312	+ cat data.txt
313	2cf48dff07ac
314	8f9b2662253a
315	90c191ee99cc

• To mount volumes, you need to have yourself as admin and enable the repo as trusted in the settings

Drone CLI

Drone CLI Setup

Docs

curl -L https://github.com/drone/drone-cli/releases/download/v1.0.7/drone_darwin_amd64.tar.gz | tar zx
sudo cp drone /usr/local/bin

Get your account details at https://drone.your-domain.com/account

export DRONE_SERVER=https://yourdomain
export DRONE_TOKEN=xx
drone info

Drone CLI Secrets

Creating Secrets:

$ drone secret add --repository ruanbekker/drone-ci-testing -name SSH_PASSWORD -value secret-password
$ drone secret add --repository ruanbekker/drone-ci-testing -name SSH_KEY -value @/home/myuser/.ssh/id_rsa

Pipeline with Secrets:

• Ref

pipeline:
  deploy:
    image:    appleboy/drone-scp
    host:     HOST
    port:     22
    secrets: [ SSH_USERNAME, SSH_PASSWORD ]
    rm: true
    script:
      - echo "Hello World" > /var/www/html/data.txt
    source:
      - src/
    target:
      - /var/www/html
    strip_components: 1

Usage

Run shell commands in a step

kind: pipeline
name: default

steps:
  - name: test
    image: alpine
    environment:
      MYUSERNAME:
        from_secret: myusername
    commands:
      - echo "echo secret"
      - echo $${MYUSERNAME}
      - echo "run env command"
      - env

or:

pipeline:
  docker:
    image: alpine
    secrets: [ myusername ]
    commands:
      - echo "echo secret"
      - echo $${MYUSERNAME}
      - echo "run env command"
      - env

This will output:

+ echo "echo secret"
echo secret
+ echo ${MYUSERNAME}
********
+ echo "run env command"
run env command
+ env
DRONE_SYSTEM_HOST=cloud.drone.io
DRONE_COMMIT_AUTHOR_AVATAR=https://avatars0.githubusercontent.com/u/567298?v=4
DRONE_BRANCH=master
[email protected]:ruanbekker/drone-ci-testing.git
DRONE_JOB_FINISHED=1555573281
CI=true
HOSTNAME=7a20b000f25c
CI_BUILD_STARTED=1555573280
DRONE_REPO_LINK=https://github.com/ruanbekker/drone-ci-testing
DRONE_TARGET_BRANCH=master
DRONE_REPO_NAMESPACE=ruanbekker
DRONE_STAGE_OS=linux
DRONE_COMMIT_AUTHOR=ruanbekker
CI_WORKSPACE=/drone/src
DRONE_GIT_HTTP_URL=https://github.com/ruanbekker/drone-ci-testing.git
DRONE_RUNNER_HOSTNAME=15e89c0f84f1
SHLVL=1
DRONE_COMMIT_BRANCH=master
HOME=/root
DRONE_REPO_SCM=
DRONE_REPO_PRIVATE=false
DRONE_SYSTEM_PROTO=https
DRONE_STEP_NUMBER=2
DRONE_REPO_VISIBILITY=public
DRONE_BUILD_STATUS=success
DRONE_BUILD_ACTION=
DRONE_RUNNER_PLATFORM=linux/amd64
DRONE_WORKSPACE_BASE=/drone/src
DRONE_COMMIT_BEFORE=138bc48338f44be2ef4ad8a57d63d4bf2403da3c
CI_JOB_STARTED=1555573280
DRONE_STAGE_ARCH=amd64
CI_WORKSPACE_PATH=
DRONE_STAGE_STATUS=success
DRONE_SOURCE_BRANCH=master
DRONE_STAGE_NAME=default
DRONE_REPO_BRANCH=master
DRONE_COMMIT_MESSAGE=add envvar
DRONE_DEPLOY_TO=
CI_BUILD_FINISHED=1555573281
DRONE_SYSTEM_HOSTNAME=cloud.drone.io
DRONE_REMOTE_URL=https://github.com/ruanbekker/drone-ci-testing.git
DRONE_JOB_STATUS=success
MYUSERNAME=********
DRONE_REPO_OWNER=ruanbekker
DRONE_STAGE_KIND=pipeline
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DRONE=true
DRONE_BUILD_NUMBER=15
DRONE_BUILD_STARTED=1555573280
DRONE_BUILD_LINK=https://cloud.drone.io/ruanbekker/drone-ci-testing/15
CI_JOB_FINISHED=1555573281
DRONE_WORKSPACE=/drone/src
DRONE_STAGE_VARIANT=
DRONE_STAGE_DEPENDS_ON=
DRONE_COMMIT_AFTER=05818a52403e07b2cab4d980522e7264be4aa18c
DRONE_STAGE_NUMBER=1
DRONE_STAGE_STARTED=1555573280
DRONE_COMMIT=05818a52403e07b2cab4d980522e7264be4aa18c
DRONE_RUNNER_HOST=15e89c0f84f1
DRONE_JOB_STARTED=1555573280
DRONE_WORKSPACE_PATH=
DRONE_BUILD_EVENT=push
DRONE_COMMIT_SHA=05818a52403e07b2cab4d980522e7264be4aa18c
DRONE_BUILD_CREATED=1555573279
[email protected]
DRONE_REPO_NAME=drone-ci-testing
DRONE_STAGE_MACHINE=15e89c0f84f1
DRONE_COMMIT_REF=refs/heads/master
CI_BUILD_STATUS=success
DRONE_MACHINE=15e89c0f84f1
CI_WORKSPACE_BASE=/drone/src
PWD=/drone/src
DRONE_BUILD_FINISHED=1555573281
DRONE_COMMIT_LINK=https://github.com/ruanbekker/drone-ci-testing/compare/138bc48338f4...05818a52403e
DRONE_REPO=ruanbekker/drone-ci-testing
DRONE_STEP_NAME=test
DRONE_SYSTEM_VERSION=1.0.0
DRONE_COMMIT_AUTHOR_NAME=Ruan Bekker
CI_JOB_STATUS=success
DRONE_STAGE_FINISHED=1555573281

Secrets in Environment Variables

Secrets are exposed to your pipeline steps and can be referenced with uppercase names: reference

pipeline:
  docker:
    image: docker
    secrets: [ swarm_key ]
    commands:
    - echo $${SWARM_KEY}
    when:
      branch: [master, develop, release/*]

will result in:

echo ${SWARM_KEY}
-----rsa key ----- 

Publish to Dockerhub:

• http://plugins.drone.io/drone-plugins/drone-docker/

• Clone

• Build Docker Image

• Publish to Dockerhub with latest and git_commit_id and use credentials from stored secrets

The .drone.yml

kind: pipeline
name: default

steps:
  - name: test
    image: alpine
    environment:
      MYUSERNAME:
        from_secret: myusername
    commands:
      - echo "echo secret"
      - echo $${MYUSERNAME}
      - echo "run env command"
      - env
      - echo $${MYUSERNAME} > /tmp/file
      - mkdir -p /tmp/foo/dir
      - touch /tmp/foo/dir/test.txt
      
  - name: publish
    image: plugins/docker
    settings:
      repo: ruanbekker/dronetest
      auto_tag: false
      auto_tag_suffix: alpine
      tags:
        - ${DRONE_COMMIT}
        - latest
      username: 
        from_secret: dockerhub_username
      password:
        from_secret: dockerhub_password

Publish to S3 Object Storage

Publish data under public/* to minio object stroage:

• http://plugins.drone.io/drone-plugins/drone-s3/

kind: pipeline
name: default

steps:
  - name: test
    ..
      
  - name: publish
    ..
        
  - name: upload
    image: plugins/s3
    settings:
      bucket: dump
      source: public/*
      target: /path/to
      path_style: true
      access_key:
        from_secret: minio_access_key
      secret_key:
        from_secret: minio_secret_key
      endpoint: 
        from_secret: minio_host

Verifyin the output:

❯ aws --profile objects s3 --endpoint-url=https://objects.domain.com ls --recursive s3://dump/
2019-04-18 10:47:30          4 path/to/public/data.txt

Slack Notifications:

• http://plugins.drone.io/drone-plugins/drone-slack/

  - name: slack
    image: plugins/slack
    settings:
      webhook:
        from_secret: slack_webhook
      channel: system_events
      image_url: https://unsplash.it/256/256/?random
      icon_url: https://unsplash.it/256/256/?random
      template: >
        {{#success build.status}}
          build {{build.number}} succeeded. Good job. link: {{commit.link}} .
        {{else}}
          build {{build.number}} failed. Fix me please.
        {{/success}}

More verbose info:

kind: pipeline
name: default

steps:
..

  - name: slack
    image: plugins/slack
    settings:
      webhook:
        from_secret: slack_webhook
      channel: system_events
      icon_url: https://www.pngkit.com/png/full/92-920674_drone-logo-png-transparent-drone-ci-logo.png
      template: >
        {{#success build.status}}
           build {{build.number}} status: *{{build.status}}*
          build link: {{build.link}}
          build author: {{build.author}}
          build ref: {{build.ref}}
          build event: {{build.event}}
          build commit: {{build.commit}}
          build branch: {{build.branch}}
          build deploy: {{build.deployTo}}
        {{else}}
          build {{build.number}} failed. Fix me please.
        {{/success}}

Telegram Notifications:

..
- name: telegram
  image: appleboy/drone-telegram
  settings:
    token:
      from_secret: telegram_token
    to:
      from_secret: telegram_to
  when:
    status: [ success, failure ]
  photo:
    - https://cdn.shopify.com/s/files/1/1061/1924/products/Virus_Emoji_large.png?v=1480481048
  format: markdown

Email Notifications

  notify:
    image: drillster/drone-email
    from: [email protected]
    host: smtp.example.com
    skip_verify: true
    secrets: [ email_username, email_password ]
    subject: >
      [DRONE CI]: {{ build.status }}: {{ repo.owner }}/{{ repo.name }}
      ({{ commit.branch }} - {{ truncate commit.sha 8 }})
    recipients:
      - [email protected]

SSH Example

• http://plugins.drone.io/appleboy/drone-ssh/

...
- name: ssh
  image: appleboy/drone-ssh
  pull: true
  settings:
    username: root
    port: 2222
    host:
      from_secret: ssh_host
    key:
        from_secret: ssh_key
    script:
      - ls -lah
      - find / -name Dockerfile
      - docker node ls

SCP Example

• https://github.com/appleboy/drone-scp

 myscp-step:
   image: appleboy/drone-scp
   host: 
     from_secret: swarm_host
   username:
     from_secret: swarm_user
   key:
     from_secret: swarm_key
   target: /root/context
   source: /root
   when:
     event: [push]

Mount Docker Socket Example

This will allow your container to run docker commands as you are logged onto the host.

Ensure you set your user as admin:

--env=DRONE_USER_CREATE=username:youruser,admin:true

Mentioned here:

• https://discourse.drone.io/t/linter-untrusted-repositories-cannot-mount-host-volumes/3438

And that your repository is marked as trusted.

Note that this will run on the host where the drone agent is located.

  deploy:
    image: docker
    secrets: [ swarm_key, swarm_host ]
    commands:
    - docker node ls
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    when:
      event: [push]
      branch: [master, develop, release/*]

Volumes: Persistence

• https://docs.drone.io/user-guide/pipeline/volumes/

kind: pipeline
name: default

steps:
- name: host-persistence
  image: busybox
  commands:
  - find /data
  volumes:
  - name: hostvol
    path: /data1

- name: build-persistence
  image: busybox
  commands:
  - touch /data2/file.txt
  volumes:
  - name: buildvol
    path: /data2/file.txt

depends_on:
- after

volumes:
- name: hostvol
  host:
    path: /tmp/artifacts
- name: buildvol
  temp: {}

Volumes: Temporary and Bind Mounts

• https://docs.drone.io/user-guide/pipeline/volumes/

kind: pipeline
name: default

steps:
  - name: test
    image: golang:alpine
    volumes:
      - name: cache
        path: /tmp/cache
    commands:
      - go test -cover -v
      - echo $$HOSTNAME >> /tmp/cache/hostname.txt
      - env

  - name: build
    image: golang:alpine
    volumes:
      - name: cache
        path: /tmp/cache
    commands:
      - go build -o app .
      - echo $$HOSTNAME >> /tmp/cache/hostname.txt
      
  - name: run
    image: golang:alpine
    volumes:
      - name: cache
        path: /tmp/cache
    commands:
      - ./app
      - echo $$HOSTNAME >> /tmp/cache/hostname.txt
      
  - name: docker-build
    image: docker:dind
    volumes:
      - name: docker-socket
        path: /var/run/docker.sock
    commands:
      - docker build -t ruanbekker/golang:$$DRONE_COMMIT_SHA .
      
  - name: docker-deploy
    image: docker:dind
    volumes:
      - name: docker-socket
        path: /var/run/docker.sock
    commands:
      - docker run ruanbekker/golang:$$DRONE_COMMIT_SHA
      - docker ps
      
volumes:
  - name: docker-socket
    host:
      path: /var/run/docker.sock
  - name: cache
    temp: {}

Rsync Example

  push:
    image: drillster/drone-rsync
    hosts: 
      from_secret: swarm_host
    key:
      from_secret: swarm_key
    source: _site/*
    target: ~/example.com
    recursive: true
    user:
      from_secret: swarm_user
    delete: true
    script:
      - chmod -R 755 ~/example.com
    when:
      event: [push]

Parallel Builds

Running steps in parallel:

- name: step-one
  image: alpine
  commands:
  - date +%s

- name: step-two
  image: alpine
  commands:
  - date +%s
  
- name: parallel-step
  image: alpine
  depends_on: [ parallel-one, parallel-two ]  

Docker Remote Tunnel Example

Using docker-remote-tunnel to run docker commands remotely via docker socket

  deploy:
    image: ruanbekker/docker-remote-tunnel
    secrets: [ swarm_key_base64, swarm_host ]
    commands:
    - echo $${SWARM_KEY_BASE64} | base64 -d > /tmp/key
    - chmod 600 /tmp/key
    - docker-tunnel --connect root@$${SWARM_HOST}
    - sleep 5
    - source /root/.docker-tunnel.sh
    - docker ps
    - docker-tunnel --terminate
    when:
      event: [push]
      branch: [master, develop, release/*]

Use Services with Pipelines

You can use services like postgres databases in your steps:

pipeline:
  ping:
    image: postgres
    commands:
      - sleep 10
      - psql -U postgres -d test -h database -p 5432 -c "CREATE TABLE person( NAME TEXT );"
      - psql -U postgres -d test -h database -p 5432 -c "INSERT INTO person VALUES('john smith');"
      - psql -U postgres -d test -h database -p 5432 -c "INSERT INTO person VALUES('jane doe');"
      - psql -U postgres -d test -h database -p 5432 -c "SELECT * FROM person;"

services:
  database:
    image: postgres
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_DB=test

services:
- name: mysql
  image: mysql:5.7
  environment:
    MYSQL_DATABASE: rest_api_example
    MYSQL_ROOT_PASSWORD: password
    MYSQL_USER: demo
    MYSQL_PASSWORD: demo

Multipipeline Depends on Steps

Resource:

• https://docs.drone.io/user-guide/pipeline/multi-machine/

Dependency graph, spans across multiple machines. Last step only executes when the frontend, backend pipelines has finished

kind: pipeline
name: frontend

steps:
- name: build
  image: alpine
  group: one
  commands:
  - sleep 10
  - echo done
  - date

- name: build2
  image: alpine
  group: one
  commands:
  - sleep 10
  - echo done
  - date

---
kind: pipeline
name: backend

steps:
- name: build
  image: alpine
  commands:
  - sleep 10
  - echo done
  - date

services:
- name: redis
  image: redis

---
kind: pipeline
name: after

steps:
- name: notify
  image: alpine
  commands:
  - sleep 10
  - echo done
  - date

depends_on:
- frontend
- backend

3 Times Multiple Pipeline Dependencies with Volumes

kind: pipeline
type: docker
name: one

steps:
- name: one-a
  image: busybox
  commands:
  - touch /data/$$RANDOM.txt
  - ls
  volumes:
  - name: artifacts
    path: /data

volumes:
- name: artifacts
  host:
    path: /tmp/artifacts
---
kind: pipeline
type: docker
name: two

steps:
- name: two-a
  image: busybox
  commands:
  - touch /data/$$RANDOM.txt
  - ls
  volumes:
  - name: artifacts
    path: /data

volumes:
- name: artifacts
  host:
    path: /tmp/artifacts
---
kind: pipeline
type: docker
name: after

steps:
- name: after-a
  image: busybox
  commands:
  - touch /data/after.txt
  - ls
  volumes:
  - name: artifacts
    path: /data

depends_on:
- one
- two

volumes:
- name: artifacts
  host:
    path: /tmp/artifacts
---
kind: pipeline
type: docker
name: after-after

steps:
- name: after-a
  image: busybox
  commands:
  - find /data
  volumes:
  - name: artifacts
    path: /data

depends_on:
- after

volumes:
- name: artifacts
  host:
    path: /tmp/artifacts

Example Pipelines

• myth/overflow drone pipeline
• drone/docs pipeline
• cncd/pipelines samples
• xueshanf/docker-awscli example
• owncloud-docker/server pipeline
• justinbarrick pipeline

Docker Swarm Stacks

• https://github.com/codestation/drone-stack
• https://github.com/drone/drone/issues/2277

Resources

• https://www.slideshare.net/appleboy/drone-cicd-platform
• https://laszlo.cloud/the-ultimate-droneci-caching-guide