Browse Source

ci: workflows: added multiple workflows for running integration tests (#3166)

* Added workflow to close stale PR(s)
* Added workflow to remove stale docker image(s)
* Added workflow to run integration tests on master
* Added workflow to run integration tests on PR(s)
* Added myself to the maintainers list.
* Added readme for workflows

Signed-off-by: Jorge Niedbalski <j@calyptia.com>
Co-authored-by: Jorge Niedbalski <j@calyptia.com>
pull/3153/head
Jorge Niedbalski 7 months ago
committed by GitHub
parent
commit
2f898b3b05
No known key found for this signature in database GPG Key ID: 4AEE18F83AFDEB23
  1. 12
      .github/workflows/README.workflows.md
  2. 20
      .github/workflows/pr-closed-docker.yaml
  3. 20
      .github/workflows/pr-stale.yaml
  4. 109
      .github/workflows/test-integration-master-microk8s.yaml
  5. 120
      .github/workflows/test-integration-pr-microk8s.yaml
  6. 1
      MAINTAINERS.md

12
.github/workflows/README.workflows.md

@ -0,0 +1,12 @@
### Available workflows
| Workflow file | Description | Run event |
| :---------------------------------------------------- | ------------------------ | ------------------------------------------------- |
| [build-release](./build-release.yaml) | Builds the distro packages and docker images from a tagged release| on new release/tag|
| [publish-release](./publish-release.yaml) | Publishes the docker images/manifest on hub.docker.io/fluent/ and the distro packages | on new release/tag on build-release completes|
| [pr-closed-docker](./pr-closed-docker.yaml) | Removes docker images for PR on hub.docker.io/fluentbitdev/| on pr closed|
| [pr-stale](./pr-stale.yaml) | Closes stale PR(s) with no activity in 30 days | scheduled daily 01:30 AM UTC|
| [test-integration-master-microk8s](./test-integration-master-microk8s.yaml) | Runs [fluent-bit-ci](https://github.com/calyptia/fluent-bit-ci/) integration tests on microk8s/x86| on new commit/push on master|
| [test-integration-pr-microk8s](./test-integration-master-microk8s.yaml) | Runs [fluent-bit-ci](https://github.com/calyptia/fluent-bit-ci/) integration tests on microk8s/x86| on new pr with the 'ok-to-test' label against master|

20
.github/workflows/pr-closed-docker.yaml

@ -0,0 +1,20 @@
name: Remove docker images for stale/closed PR(s).
on:
pull_request:
branches:
- master
types: [closed]
jobs:
docker_cleanup:
runs-on: ubuntu-latest
steps:
- uses: addnab/docker-run-action@v2
with:
image: lumir/remove-dockerhub-tag
run: python3 remove-dockerhub-tag.py --user ${{ env.dockerhub_username }} --password ${{ env.dockerhub_password }} ${{ env.dockerhub_organization }}/fluent-bit:${{ env.arch }}-master-pr-${{ env.pr }}
env:
pr: ${{ github.event.number }}
arch: x86_64
dockerhub_username: ${{ secrets.DOCKERHUB_USERNAME }}
dockerhub_password: ${{ secrets.DOCKERHUB_PASSWORD }}
dockerhub_organization: ${{ secrets.DOCKERHUB_DEV_ORGANIZATION }}

20
.github/workflows/pr-stale.yaml

@ -0,0 +1,20 @@
name: 'Close stale issues and PR(s)'
on:
schedule:
- cron: '30 1 * * *'
jobs:
stale:
runs-on: ubuntu-latest
steps:
- uses: actions/stale@v3
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
stale-issue-message: 'This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.'
stale-pr-message: 'This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.'
close-issue-message: 'This issue was closed because it has been stalled for 5 days with no activity.'
days-before-stale: 30
days-before-close: 5
days-before-pr-close: -1
exempt-all-pr-assignees: true
exempt-all-pr-milestones: true

109
.github/workflows/test-integration-master-microk8s.yaml

@ -0,0 +1,109 @@
name: Integration tests for master on microk8s
on:
push:
branches:
- master
jobs:
docker_build:
runs-on: ubuntu-latest
steps:
- name: Setup environment
run: |
sudo apt-get --yes update
sudo apt-get install --yes docker.io containerd runc
sudo systemctl unmask docker && sudo systemctl start docker
- uses: actions/checkout@v2
- name: Login to Docker Hub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build docker images for master branch
run: docker build --no-cache -f ./dockerfiles/Dockerfile.${{ env.arch }}-master -t ${{ env.dockerhub_organization }}/fluent-bit:${{ env.arch }}-master .
env:
arch: x86_64
dockerhub_organization: ${{ secrets.DOCKERHUB_DEV_ORGANIZATION }}
- name: Push master image to Docker Hub
run: docker push ${{ env.dockerhub_organization }}/fluent-bit:${{ env.arch }}-master
env:
arch: x86_64
dockerhub_organization: ${{ secrets.DOCKERHUB_DEV_ORGANIZATION }}
run-integration:
needs: docker_build
name: run-integration on ${{ matrix.k8s-release }}
strategy:
max-parallel: 48
fail-fast: true
matrix:
k8s-release: [ 1.20/stable ] #, 1.19/stable, 1.18/stable ]
runs-on: ubuntu-latest
steps:
- name: Configure LXD
run: |
sudo snap install lxd
sudo lxd.migrate -yes || true
sudo lxd waitready
sudo lxd init --auto
sudo usermod -a -G lxd ${USER}
sudo groupadd --force --system lxd
sudo usermod --append --groups lxd ${USER}
newgrp lxd
sudo newgrp lxd
sudo systemctl start snap.lxd.daemon.service
sudo systemctl status snap.lxd.daemon.service
sudo systemctl start snap.lxd.daemon.unix.socket
sudo systemctl status snap.lxd.daemon.unix.socket
sudo chmod 0777 /var/snap/lxd/common/lxd/unix.socket
- uses: actions/checkout@v2
with:
repository: calyptia/fluent-bit-ci
path: ci
- name: Terraform fmt
id: fmt
run: terraform fmt -check
continue-on-error: true
working-directory: ci/terraform/microk8s/
- name: Terraform Init
id: init
run: terraform init
working-directory: ci/terraform/microk8s/
- name: Terraform Validate
id: validate
run: terraform validate -no-color
working-directory: ci/terraform/microk8s/
- name: Terraform Apply
id: apply
run: |
newgrp lxd
terraform apply -input=false -auto-approve -var k8s-version=${{ env.k8s_release }}
working-directory: ci/terraform/microk8s/
env:
k8s_release: ${{ matrix.k8s-release }}
- run: lxc exec mk8s-node-integration-0 -- microk8s enable dns
- run: lxc exec mk8s-node-integration-0 -- microk8s enable storage
- uses: actions/setup-go@v2
with:
go-version: '1.15' # The Go version to download (if necessary) and use.
- uses: azure/setup-helm@v1
id: install
- run: go mod download
working-directory: ci/integration/
- run: make integration
env:
IMAGE_REPOSITORY: ${{ secrets.DOCKERHUB_DEV_ORGANIZATION }}/fluent-bit
IMAGE_TAG: x86_64-master
working-directory: ci/

120
.github/workflows/test-integration-pr-microk8s.yaml

@ -0,0 +1,120 @@
name: Integration tests for PR(s) on microk8s
on:
pull_request:
branches:
- master
types: [ labeled ]
jobs:
docker_build:
runs-on: ubuntu-latest
if: contains(github.event.pull_request.labels.*.name, 'ok-to-test')
steps:
- name: Setup environment
run: |
sudo apt-get --yes update
sudo apt-get install --yes docker.io containerd runc
sudo systemctl unmask docker && sudo systemctl start docker
- uses: actions/checkout@v2
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Login to Docker Hub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build the docker images for master branch
run: docker build --no-cache -f ./dockerfiles/Dockerfile.${{ env.arch }}-master -t ${{ env.dockerhub_organization }}/fluent-bit:${{ env.arch }}-master-pr-${{ env.pr }} .
env:
pr: ${{ github.event.number }}
arch: x86_64
dockerhub_organization: ${{ secrets.DOCKERHUB_DEV_ORGANIZATION }}
- name: Push master image to Docker Hub
run: docker push ${{ env.dockerhub_organization }}/fluent-bit:${{ env.arch }}-master-pr-${{ env.pr }}
env:
pr: ${{ github.event.number }}
arch: x86_64
dockerhub_organization: ${{ secrets.DOCKERHUB_DEV_ORGANIZATION }}
run-integration:
name: run-integration on ${{ matrix.k8s-release }}
needs: docker_build
strategy:
max-parallel: 48
fail-fast: true
matrix:
k8s-release: [ 1.20/stable ] #, 1.19/stable, 1.18/stable ]
runs-on: ubuntu-latest
steps:
- name: Configure LXD
run: |
sudo snap install lxd
sudo lxd.migrate -yes || true
sudo lxd waitready
sudo lxd init --auto
sudo usermod -a -G lxd ${USER}
sudo groupadd --force --system lxd
sudo usermod --append --groups lxd ${USER}
newgrp lxd
sudo newgrp lxd
sudo systemctl start snap.lxd.daemon.service
sudo systemctl status snap.lxd.daemon.service
sudo systemctl start snap.lxd.daemon.unix.socket
sudo systemctl status snap.lxd.daemon.unix.socket
sudo chmod 0777 /var/snap/lxd/common/lxd/unix.socket
- uses: actions/checkout@v2
with:
repository: calyptia/fluent-bit-ci
path: ci
- name: Terraform fmt
id: fmt
run: terraform fmt -check
continue-on-error: true
working-directory: ci/terraform/microk8s/
- name: Terraform Init
id: init
run: terraform init
working-directory: ci/terraform/microk8s/
- name: Terraform Validate
id: validate
run: terraform validate -no-color
working-directory: ci/terraform/microk8s/
- name: Terraform Apply
id: apply
run: |
newgrp lxd
terraform apply -input=false -auto-approve -var k8s-version=${{ env.k8s_release }}
working-directory: ci/terraform/microk8s/
env:
k8s_release: ${{ matrix.k8s-release }}
- run: lxc exec mk8s-node-integration-0 -- microk8s enable dns
- run: lxc exec mk8s-node-integration-0 -- microk8s enable storage
- uses: actions/setup-go@v2
with:
go-version: '1.15' # The Go version to download (if necessary) and use.
- uses: azure/setup-helm@v1
id: install
- run: go mod download
working-directory: ci/integration/
- run: make integration
env:
IMAGE_REPOSITORY: ${{ secrets.DOCKERHUB_DEV_ORGANIZATION }}/fluent-bit
IMAGE_TAG: x86_64-master-pr-${{ github.event.number }}
working-directory: ci/
- uses: actions-ecosystem/action-add-labels@v1
with:
labels: integration-tests-ok

1
MAINTAINERS.md

@ -10,4 +10,5 @@ Fluent Bit is developed and supported by many individuals and companies. The fo
| [Wesley Pettit](https://github.com/PettitWesley) | Amazon Plugins (AWS) | [Amazon Web Services](https://aws.amazon.com/) |
| [Cedric Lamoriniere](https://github.com/clamoriniere) | Datadog Output Plugin | [Datadog](https://www.datadoghq.com/) |
| [Jonathan Gonzalez V.](https://github.com/sxd) | PostgreSQL Output Plugin | [2ndQuadrant](https://www.2ndquadrant.com/en/) |
| [Jorge Niedbalski](https://github.com/niedbalski) | CI && Containers | [Calyptia](https://www.calyptia.com/) |

Loading…
Cancel
Save