Pipelines

Comprehensive, enterprise-grade SDLC pipeline templates for GitHub Actions, GitLab CI, and Azure DevOps with security scanning (SAST), dependency analysis (SCA), supply chain security (SSCA), testing, and deployment automation for multiple programming languages.

Supported Platforms & Languages

Platforms

Programming Languages

Project Structure

pipelines/
├── .github/workflows/          # GitHub Actions reusable workflows
│   ├── go-docker.yaml         # Go with Docker delivery
│   ├── go-binary.yaml         # Go binary compilation
│   ├── pdm-docker.yaml        # Python/PDM with Docker
│   ├── gradle-docker.yaml     # Java/Gradle with Docker delivery
│   ├── maven-docker.yaml      # Java/Maven with Docker delivery
│   ├── yarn-docker.yaml       # JavaScript/Yarn with Docker delivery
│   ├── npm-docker.yaml        # JavaScript/npm with Docker delivery
│   ├── composer-docker.yaml   # PHP/Composer with Docker delivery
│   ├── bundler-docker.yaml    # Ruby/Bundler with Docker delivery
│   ├── dotnet-docker.yaml     # .NET with Docker delivery
│   └── ...
├── gitlab/                     # GitLab CI pipeline templates
│   ├── golang/                # Go language pipelines
│   ├── java/                  # Java language pipelines
│   ├── python/                # Python language pipelines
│   ├── javascript/            # JavaScript/Node.js pipelines
│   ├── dotnet/                # .NET language pipelines
│   ├── terraform/             # Terraform pipelines (raw terraform/terragrunt)
│   ├── terra/                 # Terra CLI pipelines (terraform/terragrunt wrapper)
│   └── global/                # Shared GitLab configurations
├── azure-devops/              # Azure DevOps pipeline templates
│   ├── golang/                # Go language pipelines
│   ├── java/                  # Java language pipelines
│   ├── python/                # Python language pipelines
│   ├── javascript/            # JavaScript/Node.js pipelines
│   ├── dotnet/                # .NET language pipelines
│   ├── terraform/             # Terraform pipelines (raw terraform/terragrunt)
│   ├── terra/                 # Terra CLI pipelines (terraform/terragrunt wrapper)
│   └── global/                # Shared Azure DevOps templates
├── global/                     # Shared resources across platforms
│   ├── scripts/               # Automation scripts
│   │   ├── tools/             # Language-agnostic tools
│   │   │   ├── codeql/        # SAST security scanning (CodeQL)
│   │   │   ├── gitleaks/      # Secret scanning
│   │   │   ├── hadolint/      # Dockerfile linting
│   │   │   ├── semgrep/       # Static analysis
│   │   │   ├── sonarqube/     # Code quality
│   │   │   ├── trivy/         # IaC misconfiguration scanning
│   │   │   └── dependency-track/ # SCA analysis
│   │   ├── languages/         # Language-specific scripts
│   │   │   ├── golang/        # Go scripts (test, cyclonedx, golangci-lint, init)
│   │   │   └── python/        # Python scripts (cyclonedx)
│   │   └── shared/            # Common utilities
│   ├── containers/            # Custom Docker images
│   │   ├── golang.*/          # Go development images
│   │   ├── python.*/          # Python development images
│   │   ├── awscli.latest/     # AWS CLI tools
│   │   └── tor-proxy.latest/  # Network proxy tools
│   └── configs/               # Configuration files
├── makefiles/                  # Includable Makefile fragments for local usage
│   ├── common.mk              # Security tools (sast) and setup
│   ├── golang.mk              # Go targets (lint, test)
│   ├── python.mk              # Python/PDM targets (lint, test)
│   ├── java.mk                # Java/Gradle targets (lint, test)
│   ├── javascript.mk          # JavaScript/Yarn targets (lint, test)
│   ├── dotnet.mk              # .NET/C# targets (lint, test)
│   ├── terraform.mk           # Terraform targets (lint, test)
│   └── terra.mk               # Terra CLI targets (lint, test)
├── .docs/                      # Documentation and examples
│   └── examples/              # Per-provider usage examples
└── .github/tests/              # Validation scripts for this repository

Pipeline Architecture

Each platform follows a consistent 5-stage pipeline architecture:

1. Code Check (Style/Quality) - Linting, formatting, code quality, basic checks (rebase verification, changelog validation)
2. Security (SCA/SAST) - Vulnerability scanning, secret detection
3. Tests - Unit tests, integration tests, coverage reporting
4. Management - Dependency tracking, SBOM generation
5. Delivery - Build artifacts, container images, deployments

Installation

Quick Installation

curl -sSL https://raw.githubusercontent.com/rios0rios0/pipelines/main/clone.sh | bash

You can override the installation location with the PIPELINES_HOME environment variable:

PIPELINES_HOME=/opt/pipelines curl -sSL https://raw.githubusercontent.com/rios0rios0/pipelines/main/clone.sh | bash

Manual Installation

mkdir -p $HOME/Development/github.com/rios0rios0
cd $HOME/Development/github.com/rios0rios0
git clone https://github.com/rios0rios0/pipelines.git

Platform Usage

GitHub Actions

GitHub Actions workflows are located in .github/workflows/ and can be used as reusable workflows.

Available Workflows

Usage Example (Go with Docker)

name: 'CI/CD Pipeline'

on:
  push:
    branches: [ main ]
    tags: [ '*' ]
  pull_request:
    branches: [ main ]

permissions:
  checks: write      # Required for test results
  contents: write    # Required for releases
  packages: write    # Required for container registry

jobs:
  pipeline:
    uses: 'rios0rios0/pipelines/.github/workflows/go-docker.yaml@main'

Usage Example (Python/PDM with Docker)

name: 'CI/CD Pipeline'

on:
  push:
    branches: [ main ]
    tags: [ '*' ]
  pull_request:
    branches: [ main ]

permissions:
  security-events: 'write'
  contents: 'write'
  packages: 'write'

jobs:
  default:
    uses: 'rios0rios0/pipelines/.github/workflows/pdm-docker.yaml@main'

Usage Example (Java with Docker)

name: 'CI/CD Pipeline'

on:
  push:
    branches: [ main ]
    tags: [ '*' ]
  pull_request:
    branches: [ main ]

permissions:
  security-events: write
  contents: write
  packages: write

jobs:
  pipeline:
    uses: 'rios0rios0/pipelines/.github/workflows/gradle-docker.yaml@main'

Usage Example (JavaScript/Yarn with Docker)

name: 'CI/CD Pipeline'

on:
  push:
    branches: [ main ]
    tags: [ '*' ]
  pull_request:
    branches: [ main ]

permissions:
  security-events: write
  contents: write
  packages: write
  pull-requests: write
  checks: write

jobs:
  pipeline:
    uses: 'rios0rios0/pipelines/.github/workflows/yarn-docker.yaml@main'

Usage Example (.NET with Docker)

name: 'CI/CD Pipeline'

on:
  push:
    branches: [ main ]
    tags: [ '*' ]
  pull_request:
    branches: [ main ]

permissions:
  security-events: write
  contents: write
  packages: write

jobs:
  pipeline:
    uses: 'rios0rios0/pipelines/.github/workflows/dotnet-docker.yaml@main'

Usage Example (JavaScript/npm with Docker)

name: 'CI/CD Pipeline'

on:
  push:
    branches: [ main ]
    tags: [ '*' ]
  pull_request:
    branches: [ main ]

permissions:
  security-events: write
  contents: write
  packages: write
  pull-requests: write
  checks: write

jobs:
  pipeline:
    uses: 'rios0rios0/pipelines/.github/workflows/npm-docker.yaml@main'

Usage Example (Java/Maven with Docker)

name: 'CI/CD Pipeline'

on:
  push:
    branches: [ main ]
    tags: [ '*' ]
  pull_request:
    branches: [ main ]

permissions:
  security-events: write
  contents: write
  packages: write

jobs:
  pipeline:
    uses: 'rios0rios0/pipelines/.github/workflows/maven-docker.yaml@main'

Usage Example (PHP with Docker)

name: 'CI/CD Pipeline'

on:
  push:
    branches: [ main ]
    tags: [ '*' ]
  pull_request:
    branches: [ main ]

permissions:
  contents: write
  packages: write

jobs:
  pipeline:
    uses: 'rios0rios0/pipelines/.github/workflows/composer-docker.yaml@main'

Usage Example (Ruby with Docker)

name: 'CI/CD Pipeline'

on:
  push:
    branches: [ main ]
    tags: [ '*' ]
  pull_request:
    branches: [ main ]

permissions:
  security-events: write
  contents: write
  packages: write

jobs:
  pipeline:
    uses: 'rios0rios0/pipelines/.github/workflows/bundler-docker.yaml@main'

GitLab CI

GitLab CI templates use remote includes and are organized by language in the gitlab/ directory.

Available Templates

Usage Example (Go with Docker)

include:
  - remote: 'https://raw.githubusercontent.com/rios0rios0/pipelines/main/gitlab/golang/go-docker.yaml'

# Optional: Override delivery stage for custom Docker build
.delivery:
  script:
    - docker build -t "$REGISTRY_PATH$IMAGE_SUFFIX:$TAG" -f .ci/stages/40-delivery/Dockerfile .
  cache:
    key: 'test:all'
    paths: !reference [ .go, cache, paths ]
    policy: 'pull'

Usage Example (Python PDM)

include:
  - remote: 'https://raw.githubusercontent.com/rios0rios0/pipelines/main/gitlab/python/pdm-docker.yaml'

variables:
  PYTHON_VERSION: "3.11"  # Optional: specify a Python version

Usage Example (Terraform -- raw terraform/terragrunt)

include:
  - remote: 'https://raw.githubusercontent.com/rios0rios0/pipelines/main/gitlab/terraform/terra.yaml'

Usage Example (Terra CLI)

The terra CLI wraps Terraform and Terragrunt with a simplified interface, auto-answering prompts, and parallel execution. The terra pipeline provides code check, security, tests, and management stages. Delivery is intentionally excluded because it is project-specific (plan/apply targets, environments, stack ordering). See examples for all providers in the Azure DevOps section below.

Terra Test Stage

Every Terra pipeline (Azure DevOps, GitLab CI, GitHub Actions) exposes a single unified test:all job that delegates to global/scripts/languages/terraform/test-all/run.sh. The runner orchestrates two tiers:

The runner auto-detects which tiers the consumer actually has, runs only those, merges both JUnit files into junit-terra-all.xml for the single-artifact upload contract used by GitLab CI and GitHub Actions, and propagates a non-zero exit from either tier so CI correctly fails. When neither tier has tests (e.g., a stack-only repo without modules/ tests or tests/terratest/), the runner emits an empty-but-valid JUnit and exits 0 so the job passes without a bespoke opt-out.

Required GitLab Variables

Configure these in your GitLab project settings:

Azure DevOps

Azure DevOps templates are located in the azure-devops/ directory and use template references.

Available Templates

Usage Example (Go with Docker)

trigger:
  branches:
    include: [ main ]
  tags:
    include: [ '*' ]

pool:
  vmImage: 'ubuntu-latest'

variables:
  - ${{ if startsWith(variables['Build.SourceBranch'], 'refs/tags/') }}:
      - group: 'production-variables'
  - ${{ else }}:
      - group: 'development-variables'

resources:
  repositories:
    - repository: 'pipelines'
      type: 'github'
      name: 'rios0rios0/pipelines'
      endpoint: 'YOUR_GITHUB_SERVICE_CONNECTION'  # Configure this

stages:
  - template: 'azure-devops/golang/go-docker.yaml@pipelines'

Usage Example (Go with ARM Deployment)

resources:
  repositories:
    - repository: 'pipelines'
      type: 'github'
      name: 'rios0rios0/pipelines'
      endpoint: 'YOUR_GITHUB_SERVICE_CONNECTION'

stages:
  - template: 'azure-devops/golang/go-arm.yaml@pipelines'
    parameters:
      DOCKER_BUILD_ARGS: '--build-arg VERSION=$(Build.BuildNumber)'
      RUN_BEFORE_BUILD: 'echo "Preparing build environment"'

Usage Example (Go with AWS Lambda)

trigger:
  branches:
    include: [ main ]
  tags:
    include: [ '*' ]

pool:
  vmImage: 'ubuntu-latest'

variables:
  - ${{ if startsWith(variables['Build.SourceBranch'], 'refs/tags/') }}:
      - group: 'production-variables'
  - ${{ else }}:
      - group: 'development-variables'

resources:
  repositories:
    - repository: 'pipelines'
      type: 'github'
      name: 'rios0rios0/pipelines'
      endpoint: 'YOUR_GITHUB_SERVICE_CONNECTION'

stages:
  - template: 'azure-devops/golang/go-lambda.yaml@pipelines'
    parameters:
      LAMBDA_FUNCTION_NAME: 'my-go-lambda-function'
      AWS_REGION: 'us-east-1'
      AWS_SERVICE_CONNECTION: 'AWS-Service-Connection'  # Configure in Azure DevOps
      DEPLOY_STRATEGY: 'zip'  # or 'sam'
      GOARCH: 'amd64'  # or 'arm64'
      LAMBDA_TIMEOUT: '30'
      LAMBDA_MEMORY_SIZE: '128'

For SAM-based deployments:

stages:
  - template: 'azure-devops/golang/go-lambda-sam.yaml@pipelines'
    parameters:
      S3_BUCKET: 'my-deployment-bucket'
      AWS_REGION: 'us-east-1'
      AWS_SERVICE_CONNECTION: 'AWS-Service-Connection'
      SAM_CONFIG_ENV: 'default'  # References samconfig.toml environment

Required Variable Groups

Create these variable groups in Azure DevOps Library:

Shared Variables (All Projects):

Project-Specific Variables (.NET Example):

AWS Lambda Deployment Variables (Optional):

Note: For AWS deployments, it is recommended to use Azure DevOps AWS Service Connection instead of storing credentials in variable groups. Configure the service connection in Azure DevOps Project Settings > Service Connections.

Available Tools & Scripts

Security & Analysis Tools

SAST (Static Application Security Testing)

SCA (Software Composition Analysis)

Quality & Management

Basic Checks

Every pipeline includes basic checks that run in parallel with linting during the Code Check stage. These checks verify:

1. Rebase verification — the PR/MR branch is rebased on top of the target branch (usually main). If the branch is behind, the pipeline fails with clear instructions to rebase. This enforces a linear commit history and prevents merge conflicts from reaching the test and delivery stages.
2. Changelog validation — the CHANGELOG.md file was modified and new entries are placed under the [Unreleased] section. If entries appear below an existing version section (e.g., due to an erroneous rebase), the pipeline fails with instructions to fix the placement.

Language-Specific Tools

Go Tools

Terraform / Terra Tools

File-Ordering Standard (order-check)

Dense Terragrunt monorepos keep their *.hcl / *.tf files in a consistent order. The order-check job runs in the Code Check stage of the terra and terraform pipelines (all three platforms) and enforces:

• environments/**/root.hcl — dependency blocks and the inputs block ordered ascending by dependency number (the environments/NN_ prefix).
• stacks/*/variables.tf — a // SET ON .HCL section before a // SET ON .ENV section; dependency-derived variables ordered by dependency number inside .HCL.
• **/providers.tf (stacks + modules) — required_providers and provider blocks ordered heaviest → lightest (cloud → data → orchestration → network → utility → trivial like random/null/local).
• stacks/*/outputs.tf — outputs ordered to follow the declaration order of the modules/resources they reference in main*.tf.

# check (CI gate; writes build/reports/junit-order-check.xml)
"$SCRIPTS_DIR/global/scripts/languages/terraform/order-check/run.sh"

# auto-sort, then re-align with your formatter
"$SCRIPTS_DIR/global/scripts/languages/terraform/order-check/run.sh" --fix
terra format   # or: terraform fmt -recursive && terragrunt hcl format

The provider ranking and path exclusions can be overridden per-repo with an optional .terraform-order.json in the repo root:

{
  "provider_order": ["azurerm", "helm", "kubernetes", "random"],
  "ignore": ["modules/legacy/**"]
}

Only python3 is required (no Terraform binary). The --fix rewriter is round-trip-safe: it only reorders existing blocks and leaves any file it cannot parse cleanly untouched.

Usage Examples

Run Security Scanning Locally (via Makefile)

make setup      # Clone/update pipelines repo
make lint       # Run golangci-lint
make test       # Run Go tests with coverage
make security   # Run all security tools (CodeQL, Gitleaks, Hadolint, Trivy, Semgrep)

Configure Go Linting Globally

# Symlink the shared golangci-lint config for IDE integration
SCRIPTS_DIR=$HOME/Development/github.com/rios0rios0/pipelines
ln -s $SCRIPTS_DIR/global/scripts/languages/golang/golangci-lint/.golangci.yml ~/.golangci.yml

Container Images

Pre-built container images optimized for CI/CD environments:

Building Custom Images

# Build and push a custom container
make build-and-push NAME=awscli TAG=latest

# Local build for testing
docker build -t my-image -f global/containers/awscli.latest/Dockerfile global/containers/awscli.latest/

Makefile Integration

The recommended way to use this repository locally is through the includable .mk files. GNU Make's -include directive imports targets from the pipelines repository, so your project Makefile only needs to declare SCRIPTS_DIR and the includes:

Before (repeated in every project):

SCRIPTS_DIR = $(HOME)/Development/github.com/rios0rios0/pipelines

.PHONY: lint
lint:
	${SCRIPTS_DIR}/global/scripts/languages/golang/golangci-lint/run.sh --fix .

.PHONY: test
test:
	${SCRIPTS_DIR}/global/scripts/languages/golang/test/run.sh .

.PHONY: sast
sast:
	${SCRIPTS_DIR}/global/scripts/tools/codeql/run.sh "go"

After (include once, get all targets):

# Pipeline targets: setup, sast, lint, test
SCRIPTS_DIR ?= $(HOME)/Development/github.com/rios0rios0/pipelines
-include $(SCRIPTS_DIR)/makefiles/common.mk
-include $(SCRIPTS_DIR)/makefiles/golang.mk

build:
	go build -o bin/app .

run:
	go run .

This gives you the following targets for free:

Available language files:

The -include prefix means Make silently skips the includes if the repository is not cloned yet. Run make setup (or curl ... | bash) to bootstrap.

See the .docs/examples/ directory for complete per-provider examples including Makefiles.

Direct Script Usage

If you prefer calling scripts directly without Makefile includes:

export SCRIPTS_DIR=$HOME/Development/github.com/rios0rios0/pipelines

# Go linting
$SCRIPTS_DIR/global/scripts/languages/golang/golangci-lint/run.sh --fix

# Go tests
$SCRIPTS_DIR/global/scripts/languages/golang/test/run.sh

# Security scans
$SCRIPTS_DIR/global/scripts/tools/gitleaks/run.sh
$SCRIPTS_DIR/global/scripts/tools/codeql/run.sh go
$SCRIPTS_DIR/global/scripts/tools/hadolint/run.sh
$SCRIPTS_DIR/global/scripts/tools/trivy/run.sh
$SCRIPTS_DIR/global/scripts/tools/semgrep/run.sh

Testing Pipeline Changes

When developing pipeline modifications, you can test against development branches:

Switch to Development Branch

export BRANCH=your-feature-branch-name

# Update all pipeline references to use your branch
find . -type f -name "*.yaml" -exec sed -i.bak -E "s|(remote: 'https://raw.githubusercontent.com/rios0rios0/pipelines/)[^/]+(/.*)|\\1$BRANCH\\2|g" {} +

Test Your Changes

# Update your project's pipeline reference
# Before:
include:
  - remote: 'https://raw.githubusercontent.com/rios0rios0/pipelines/main/gitlab/golang/go-docker.yaml'

# After:
include:
  - remote: 'https://raw.githubusercontent.com/rios0rios0/pipelines/your-feature-branch/gitlab/golang/go-docker.yaml'

Troubleshooting

Common Issues & Solutions

Pipeline Failures

Issue: "No directories found to test it" (Go projects)

• Cause: Go project structure does not match the expected layout
• Solution: Ensure your project has cmd/, pkg/, or internal/ directories
• Alternative: Modify the test script to include your custom directories

Issue: "golangci-lint: command not found"

• Cause: golangci-lint not installed or not in PATH
• Solution: The script automatically downloads golangci-lint, ensure Docker is available

Issue: Docker build fails with SSL certificate errors

• Cause: Network restrictions in CI environment
• Solution: This is expected in restricted environments; contact your platform administrator

Security Tool Issues

Issue: CodeQL analysis fails

• Cause: CodeQL CLI not installed or language not supported
• Solution: Ensure network access to download CodeQL CLI bundle; supported languages: go, python, java, javascript, csharp, ruby (PHP is not supported)

Issue: Gitleaks takes too long or fails

• Cause: Large repository or network issues
• Solution: Increase timeout values, ensure network access to GitHub releases for the Gitleaks binary download

Issue: Semgrep timeout or hangs

• Cause: Large codebase, downloading security rules
• Solution: Allow 10+ minutes for completion, do not cancel the operation

Issue: Hadolint skips analysis

• Cause: No Dockerfiles found in the project
• Solution: This is expected for projects without Dockerfiles; Hadolint auto-skips gracefully

Issue: Trivy IaC scan finds false positives

• Cause: Trivy flags misconfigurations in Terraform, Kubernetes, or Dockerfiles
• Solution: Add entries to .trivyignore in the project root to suppress known false positives

Platform-Specific Issues

GitHub Actions:

• Issue: Workflow does not trigger
• Solution: Check repository permissions, ensure workflow file is in .github/workflows/

GitLab CI:

• Issue: "Remote file could not be fetched"
• Solution: Verify the remote URL is accessible, check branch name in URL

Azure DevOps:

• Issue: "Template not found"
• Solution: Ensure GitHub service connection is configured correctly

Environment Requirements

Minimum Requirements:

• Docker (for container builds and security tools)
• Git (for repository operations)
• Network access (for downloading tools and dependencies)

Language-Specific Requirements:

• Go: Go 1.18+ (automatically installed in CI)
• Python: Python 3.8+ (automatically managed in CI)
• Java: JDK 11+ (automatically managed in CI)
• Node.js: Node 16+ (automatically managed in CI)

Performance Expectations

Important: Never cancel operations that appear to be hanging - they may be downloading large Docker images or rule sets.

Contributing

Contributions are welcome. See CONTRIBUTING.md for guidelines.

License

This project is licensed under the MIT License.

Note: This repository provides pipeline templates and automation scripts, not a runnable application. Users consume these templates in their own projects to establish comprehensive CI/CD pipelines with security, quality, and testing automation.