02 - GitHub Actions Patterns

62 min read

GitHub Actions Patterns

How the main pattern categories fit together

GitHub Actions patterns fall into four categories. Each category builds on the previous one: composition patterns define what runs, data flow patterns move information between those units, control flow patterns decide when and whether they run, and architecture patterns combine all three into production-grade designs.


flowchart TB
  subgraph comp["Composition Patterns"]
    direction LR
    M["Matrix Strategy"]
    RW["Reusable Workflows"]
    CA["Composite Actions"]
    WC["Workflow Chaining"]
  end

  subgraph data["Data Flow Patterns"]
    direction LR
    O["Outputs & Env Vars"]
    A["Artifacts"]
    C["Caching"]
    S["Job Summaries"]
  end

  subgraph ctrl["Control Flow Patterns"]
    direction LR
    EX["Conditionals"]
    CC["Concurrency"]
    TO["Timeouts & Errors"]
    PF["Path Filtering"]
  end

  subgraph arch["Architecture Patterns"]
    direction LR
    MR["Monorepo"]
    RL["Release Automation"]
    SEC["Security"]
    COST["Cost Optimization"]
  end

  comp --> data
  data --> ctrl
  ctrl --> arch

  style comp fill:#292e42,stroke:#7aa2f7,color:#c0caf5
  style data fill:#292e42,stroke:#9ece6a,color:#c0caf5
  style ctrl fill:#292e42,stroke:#e0af68,color:#c0caf5
  style arch fill:#292e42,stroke:#f7768e,color:#c0caf5

Pattern categories form a layered architecture: composition defines the units of work, data flow connects them, control flow governs execution order, and architecture patterns combine all three for real-world systems. Each category in this page has a dedicated H2 section.

Ways to compose and reuse workflow logic

Composition patterns define how work is structured and reused across workflows. The four mechanisms differ in scope: matrix fans out a single job, reusable workflows share entire jobs, composite actions share step sequences, and workflow chaining coordinates independent workflows.

composition | matrix strategy

The matrix strategy creates multiple parallel instances of a job by computing the cross-product of variable lists. Each combination runs as an independent job with its own runner.

Define a static matrix with include and exclude

When a job must run against multiple versions, platforms, or configurations. It is typically triggered by any event — matrix applies at the job level regardless of trigger. Each matrix combination gets its own runner. All combinations share the same workflow run ID. Validate compatibility across Python versions, OS variants, or database backends without writing separate jobs.

Run a matrix build across Python 3.11 and 3.12, with include adding an experimental flag and exclude dropping 3.10.

name: "Demo: Matrix Strategy"
on:
  push:
    branches: [main]
    paths:
      - ".github/workflows/demo-matrix.yml"
  workflow_dispatch:
 
permissions:
  contents: read
 
jobs:
  matrix-test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      max-parallel: 3
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
        os: [ubuntu-latest]
        include:
          - python-version: "3.12"
            os: ubuntu-latest
            experimental: true
        exclude:
          - python-version: "3.10"
            os: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
 
      - name: Show matrix values
        run: |
          echo "=== Matrix combination ==="
          echo "Python: ${{ matrix.python-version }}"
          echo "OS: ${{ matrix.os }}"
          echo "Experimental: ${{ matrix.experimental }}"
          echo ""
          python --version
          echo ""
          echo "Matrix context (full):"
          echo '${{ toJSON(matrix) }}'

Workflow run output (run 24312297704, triggered by push to main):

✓ main Demo: Matrix Strategy · 24312297704
Triggered via push
 
JOBS
✓ matrix-test (3.11, ubuntu-latest) in 3s (ID 70983934648)
✓ matrix-test (3.12, ubuntu-latest) in 4s (ID 70983934660)

Job output for matrix-test (3.11):

=== Matrix combination ===
Python: 3.11
OS: ubuntu-latest
Experimental:

Job output for matrix-test (3.12):

=== Matrix combination ===
Python: 3.12
OS: ubuntu-latest
Experimental: true

Matrix explosion with too many dimensions

  • A matrix with 3 Python versions x 3 OS x 3 database backends = 27 parallel jobs
  • GitHub limits to 256 jobs per workflow run
  • Each job consumes runner minutes — a 3x3x3 matrix at 5 min/job = 135 minutes of billing

Control matrix size

  • Use max-parallel to limit concurrent runners
  • Use exclude to drop known-incompatible combinations
  • Use include to add specific combinations instead of full cross-products
  • Consider testing the full matrix only on PRs to main, not on every push
KeyTypeDefaultDescription
matrix.<variable>listrequiredValues to iterate — creates the cross-product
fail-fastbooleantrueCancel all jobs if any combination fails
max-parallelintegerunlimitedMaximum concurrent combinations
includelist of mapsnoneAdd variables to matching combinations or create new ones
excludelist of mapsnoneRemove matching combinations from the cross-product

composition | reusable workflows

A reusable workflow is a complete workflow file with on: workflow_call that another workflow can invoke as a job. The caller passes inputs and secrets; the callee returns outputs. This enables standardized CI/CD patterns across repositories.

Define a reusable workflow (callee)

When multiple repositories or workflows need the same job logic (e.g., deploy, test, validate). It is typically triggered by workflow_call — this workflow cannot be triggered directly, only by a caller. Runs on its own runner. Has access to the caller’s repository code and the caller’s GITHUB_TOKEN permissions. Centralize and standardize workflow logic so teams share a single tested implementation.

Define a reusable workflow that accepts environment, Python version, and a deploy token, then returns a deployment URL.

name: "Demo: Reusable Workflow (Called)"
on:
  workflow_call:
    inputs:
      environment:
        description: "Target environment name"
        required: true
        type: string
      python-version:
        description: "Python version to use"
        required: false
        type: string
        default: "3.12"
    outputs:
      deploy-url:
        description: "The deployment URL"
        value: ${{ jobs.deploy.outputs.url }}
    secrets:
      deploy-token:
        description: "Deployment authentication token"
        required: false
 
jobs:
  deploy:
    runs-on: ubuntu-latest
    outputs:
      url: ${{ steps.deploy.outputs.url }}
    steps:
      - uses: actions/checkout@v4
 
      - name: Show reusable workflow inputs
        run: |
          echo "=== Reusable workflow (called) ==="
          echo "Environment: ${{ inputs.environment }}"
          echo "Python version: ${{ inputs.python-version }}"
          echo "Deploy token provided: ${{ secrets.deploy-token != '' }}"
          echo ""
          echo "This workflow was invoked by a caller workflow."
          echo "It receives inputs, secrets, and can return outputs."
 
      - name: Simulate deployment
        id: deploy
        run: |
          URL="https://${{ inputs.environment }}.example.com"
          echo "url=$URL" >> "$GITHUB_OUTPUT"
          echo "Deployed to $URL"

Call a reusable workflow (caller)

When your workflow needs to invoke shared logic from another workflow file. It is typically triggered by any event on the caller side — push, workflow_dispatch, pull_request, etc. The uses: key at the job level (not step level) references the callee. Inputs go in with:, secrets in secrets:. Invoke the standardized deploy workflow with environment-specific parameters.

Call the reusable workflow with staging environment inputs, then read the returned deploy URL in a downstream job.

name: "Demo: Reusable Workflow (Caller)"
on:
  push:
    branches: [main]
    paths:
      - ".github/workflows/demo-reusable-caller.yml"
      - ".github/workflows/demo-reusable-called.yml"
  workflow_dispatch:
 
permissions:
  contents: read
 
jobs:
  call-reusable:
    uses: ./.github/workflows/demo-reusable-called.yml
    with:
      environment: staging
      python-version: "3.12"
    secrets:
      deploy-token: ${{ secrets.DEMO_SECRET }}
 
  show-result:
    needs: call-reusable
    runs-on: ubuntu-latest
    steps:
      - name: Show reusable workflow output
        run: |
          echo "=== Caller workflow ==="
          echo "Deploy URL from reusable workflow: ${{ needs.call-reusable.outputs.deploy-url }}"
          echo ""
          echo "The caller invokes the reusable workflow with 'uses:'"
          echo "and passes inputs via 'with:' and secrets via 'secrets:'."
          echo "Outputs flow back through needs.<job>.outputs.<name>."

Workflow run output (run 24312297695, triggered by push to main):

✓ main Demo: Reusable Workflow (Caller) · 24312297695
Triggered via push
 
JOBS
✓ call-reusable / deploy in 4s (ID 70983934723)
✓ show-result in 4s (ID 70983938911)

Callee job output:

=== Reusable workflow (called) ===
Environment: staging
Python version: 3.12
Deploy token provided: true
Deployed to https://staging.example.com

Caller show-result job output:

=== Caller workflow ===
Deploy URL from reusable workflow: https://staging.example.com
The caller invokes the reusable workflow with 'uses:'
and passes inputs via 'with:' and secrets via 'secrets:'.
Outputs flow back through needs.<job>.outputs.<name>.

secrets: inherit widens the blast radius

  • secrets: inherit passes all secrets from the caller to the callee — including secrets the callee does not need
  • If the callee is in another repository or maintained by another team, those secrets could be logged, exfiltrated, or leaked via a compromised action

Pass secrets explicitly

  • Always enumerate secrets by name: secrets: { deploy-token: ${{ secrets.DEPLOY_TOKEN }} }
  • This documents which secrets the callee needs and limits exposure
  • Use secrets: inherit only for same-repo callees where all secrets are relevant

Reusable workflow vs composite action

  • Reusable workflow: replaces an entire job — has its own runner, can use services, environment, and strategy. Maximum 4 levels of nesting. Called with uses: at the job level.
  • Composite action: replaces a sequence of steps — runs on the caller’s runner, sharing the same workspace. No service containers or environments. Called with uses: at the step level.
  • Rule of thumb: if the shared logic needs its own runner, environment, or matrix, use a reusable workflow. If it is a step-level utility (setup, validation, notification), use a composite action.
KeyScopeTypeDescription
on.workflow_call.inputs.<name>calleestring, number, booleanTyped input parameter with optional default
on.workflow_call.outputs.<name>calleestringValue returned to the caller
on.workflow_call.secrets.<name>calleesecretNamed secret the caller must provide
jobs.<id>.usescallerpath/refPath to the callee workflow file
jobs.<id>.withcallermapInput values matching callee’s declarations
jobs.<id>.secretscallermap or inheritSecret values or blanket inheritance
needs.<job>.outputs.<name>callerstringRead callee’s returned outputs

composition | composite actions

A composite action bundles multiple steps into a single reusable step defined in an action.yml file. Unlike reusable workflows, composite actions run on the caller’s runner and share the caller’s workspace.

Create a composite action

When multiple workflows repeat the same step sequence (setup, validation, notification). It is typically triggered by not triggered independently — invoked with uses: at the step level. Runs in the caller’s job, on the caller’s runner. Has access to the caller’s workspace, environment variables, and GITHUB_TOKEN. Encapsulate Python setup + pip cache + dependency install into a single reusable step.

Define a composite action that installs Python, restores pip cache, and installs dependencies.

name: "Setup Python Environment"
description: "Install Python, restore pip cache, and install dependencies from requirements.txt"
 
inputs:
  python-version:
    description: "Python version to install"
    required: false
    default: "3.12"
  requirements-file:
    description: "Path to requirements file"
    required: false
    default: "requirements.txt"
 
outputs:
  cache-hit:
    description: "Whether the pip cache was restored"
    value: ${{ steps.pip-cache.outputs.cache-hit }}
  python-path:
    description: "Path to the installed Python binary"
    value: ${{ steps.setup-python.outputs.python-path }}
 
runs:
  using: "composite"
  steps:
    - name: Set up Python
      id: setup-python
      uses: actions/setup-python@v5
      with:
        python-version: ${{ inputs.python-version }}
 
    - name: Cache pip packages
      id: pip-cache
      uses: actions/cache@v4
      with:
        path: ~/.cache/pip
        key: pip-${{ runner.os }}-py${{ inputs.python-version }}-${{ hashFiles(inputs.requirements-file) }}
        restore-keys: |
          pip-${{ runner.os }}-py${{ inputs.python-version }}-
 
    - name: Install dependencies
      shell: bash
      run: |
        python -m pip install --upgrade pip
        pip install -r ${{ inputs.requirements-file }}
 
    - name: Show environment summary
      shell: bash
      run: |
        echo "=== Python environment ready ==="
        echo "Python: $(python --version)"
        echo "Pip: $(pip --version)"
        echo "Cache hit: ${{ steps.pip-cache.outputs.cache-hit }}"
        echo "Packages installed: $(pip list --format=columns | tail -n +3 | wc -l)"

Use the composite action in a workflow

When a workflow needs the Python environment setup without repeating the step sequence. It is typically triggered by any workflow event — the composite action is called at the step level. Runs on the same runner as the calling job. The action’s steps appear in the calling job’s logs. Replace three manual steps (setup-python, cache, pip install) with a single uses: step.

Call the composite action with default inputs, then with custom Python 3.11.

name: "Demo: Composite Action"
on:
  push:
    branches: [main]
    paths:
      - ".github/actions/setup-python-env/**"
      - ".github/workflows/demo-composite-action.yml"
  workflow_dispatch:
 
permissions:
  contents: read
 
jobs:
  use-composite:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - name: Use composite action with defaults
        id: setup
        uses: ./.github/actions/setup-python-env
 
      - name: Show composite action outputs
        run: |
          echo "=== Composite action outputs ==="
          echo "Cache hit: ${{ steps.setup.outputs.cache-hit }}"
          echo "Python path: ${{ steps.setup.outputs.python-path }}"
          echo ""
          echo "The composite action encapsulated:"
          echo "  1. Python installation (setup-python)"
          echo "  2. Pip cache restore (actions/cache)"
          echo "  3. Dependency installation (pip install)"
          echo ""
          echo "Callers get a single 'uses:' step instead of three."
          python --version
          pip list --format=columns | head -10
 
  use-composite-custom:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - name: Use composite action with custom inputs
        id: setup
        uses: ./.github/actions/setup-python-env
        with:
          python-version: "3.11"
          requirements-file: "requirements.txt"
 
      - name: Verify custom version
        run: |
          echo "=== Custom inputs ==="
          echo "Requested Python 3.11, got: $(python --version)"
          echo "Cache hit: ${{ steps.setup.outputs.cache-hit }}"

Workflow run output (run 24313636452, triggered by push to main):

✓ main Demo: Composite Action · 24313636452
Triggered via push
 
JOBS
✓ use-composite in 9s (ID 70987459777)
✓ use-composite-custom in 11s (ID 70987459778)

Job output (use-composite, Python 3.12 defaults):

Successfully set up CPython (3.12.13)
Cache key: pip-Linux-py3.12-dc48ecd8592482cb31ac7d505da3fa4609591b66318c3c507821649953f1be8e

Job output (use-composite-custom, Python 3.11):

Successfully set up CPython (3.11.15)
KeyScopeDescription
nameaction.ymlDisplay name for the action
descriptionaction.ymlOne-line description shown in the marketplace
inputs.<name>action.ymlInput parameter with description, required flag, and optional default
outputs.<name>action.ymlOutput value surfaced to the caller via steps.<id>.outputs.<name>
runs.usingaction.ymlRuntime: "composite", "node20", or "docker"
runs.steps[].shellcomposite onlyRequired on every run: step — composite actions do not inherit defaults.run.shell

composition | workflow chaining

Workflow chaining coordinates independent workflows by triggering one workflow after another completes. The workflow_run event fires when a named workflow finishes, and workflow_dispatch enables manual or API-driven triggering with typed inputs.

Chain workflows with workflow_run

When a workflow should execute after another workflow completes (e.g., deploy after CI). It is typically triggered by workflow_run event — fires when the named workflow completes, succeeds, or fails. Always runs on the default branch (main), not the triggering branch. This is a security feature — the chained workflow uses trusted code from main. Decouple CI from deployment: let CI run on the PR branch, then trigger deployment from main after success.

Trigger a post-processing workflow after the push trigger workflow completes successfully.

name: "Demo: Workflow Run Trigger"
on:
  workflow_run:
    workflows: ["Demo: Push Trigger"]
    types: [completed]
 
permissions:
  contents: read
 
jobs:
  post-push:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    steps:
      - name: Show workflow_run context
        run: |
          echo "=== workflow_run trigger ==="
          echo "This workflow was triggered by the completion of another workflow."
          echo ""
          echo "Triggering workflow: ${{ github.event.workflow_run.name }}"
          echo "Triggering workflow ID: ${{ github.event.workflow_run.id }}"
          echo "Triggering conclusion: ${{ github.event.workflow_run.conclusion }}"
          echo "Triggering branch: ${{ github.event.workflow_run.head_branch }}"
          echo "Triggering SHA: ${{ github.event.workflow_run.head_sha }}"
          echo "Triggering actor: ${{ github.event.workflow_run.actor.login }}"
          echo ""
          echo "IMPORTANT: workflow_run always runs on the DEFAULT branch (main)."
          echo "It does NOT run the code from the triggering branch."
          echo "This has security implications — the called workflow is trusted code."
          echo ""
          echo "Common use case: run deployment after CI passes,"
          echo "or aggregate results from fork PR workflows."

Workflow run output (run 24313244786, triggered by workflow_run):

✓ main Demo: Workflow Run Trigger · 24313244786
Triggered via workflow_run
 
JOBS
✓ post-push in 2s (ID 70986424498)

Job output:

=== workflow_run trigger ===
Triggering workflow: Demo: Push Trigger
Triggering workflow ID: 24313241487
Triggering conclusion: success
Triggering branch: main
Triggering SHA: 03c544c113225f4310d28f09352a5cd6343d12b1
Triggering actor: alp78

workflow_run fires on completion, not just success

  • Without the if: guard, the chained workflow runs even when the upstream fails
  • Always add if: ${{ github.event.workflow_run.conclusion == 'success' }} unless you intentionally handle failure cases

Guard on conclusion

  • Use conclusion == 'success' for deploy-after-CI patterns
  • Use conclusion == 'failure' for failure notification or rollback patterns
  • Use types: [completed] (not [requested]) to ensure the upstream has finished

Chaining vs dependent jobs

  • Workflow chaining (workflow_run): the chained workflow is independent — it has its own trigger, permissions, and code version (always from main). Best for cross-concern boundaries (CI → deploy, PR → aggregate).
  • Dependent jobs (needs:): jobs within the same workflow share the trigger, run, and code version. Best for build → test → deploy within a single pipeline.
  • Use chaining when the second workflow should run different code (from main) than the first (from a PR branch).

Ways to move data across steps, jobs, and workflows

Data flow patterns move information between steps, jobs, and workflows. Each mechanism has different scope, persistence, and size limits.

data flow | outputs and environment variables

Steps within a job communicate through $GITHUB_OUTPUT (step outputs) and $GITHUB_ENV (dynamic environment variables). Cross-job communication uses job-level outputs read via the needs context.

Pass data between steps and jobs

When a step produces a value (version string, timestamp, computed path) that later steps or jobs need. It is typically triggered by any event — data flow is independent of the trigger. GITHUB_OUTPUT and GITHUB_ENV are scoped to the current job. Only values promoted to jobs.<id>.outputs are visible to downstream jobs via needs.<id>.outputs.<name>. Propagate build metadata from a producer job to a consumer job without using artifacts.

Set version and timestamp via GITHUB_OUTPUT, set BUILD_TAG via GITHUB_ENV, then read them in a downstream job.

name: "Demo: Outputs and Data Flow"
on:
  push:
    branches: [main]
    paths:
      - ".github/workflows/demo-outputs.yml"
  workflow_dispatch:
 
permissions:
  contents: read
 
jobs:
  producer:
    runs-on: ubuntu-latest
    outputs:
      version: ${{ steps.set_version.outputs.version }}
      timestamp: ${{ steps.set_timestamp.outputs.ts }}
    steps:
      - name: Set version via GITHUB_OUTPUT
        id: set_version
        run: |
          echo "version=1.2.3" >> "$GITHUB_OUTPUT"
          echo "Wrote version=1.2.3 to GITHUB_OUTPUT"
 
      - name: Set timestamp via GITHUB_OUTPUT
        id: set_timestamp
        run: |
          TS=$(date -u '+%Y%m%d-%H%M%S')
          echo "ts=$TS" >> "$GITHUB_OUTPUT"
          echo "Wrote ts=$TS to GITHUB_OUTPUT"
 
      - name: Set dynamic env via GITHUB_ENV
        run: |
          echo "BUILD_TAG=build-$(date -u '+%Y%m%d')" >> "$GITHUB_ENV"
          echo "Wrote BUILD_TAG to GITHUB_ENV"
 
      - name: Use dynamic env
        run: |
          echo "BUILD_TAG from GITHUB_ENV: $BUILD_TAG"
          echo "(This variable was set dynamically in the previous step)"
 
  consumer:
    needs: producer
    runs-on: ubuntu-latest
    steps:
      - name: Read job outputs via needs context
        run: |
          echo "=== Data received from producer job ==="
          echo "Version: ${{ needs.producer.outputs.version }}"
          echo "Timestamp: ${{ needs.producer.outputs.timestamp }}"
          echo "Producer result: ${{ needs.producer.result }}"
          echo ""
          echo "Key point: GITHUB_ENV does NOT cross job boundaries."
          echo "Only values declared in jobs.<id>.outputs and written"
          echo "to GITHUB_OUTPUT are available via needs.<id>.outputs."

Workflow run output (run 24312297707, triggered by push to main):

✓ main Demo: Outputs and Data Flow · 24312297707
Triggered via push
 
JOBS
✓ producer in 3s (ID 70983934709)
✓ consumer in 3s (ID 70983938140)

Producer job output:

Wrote version=1.2.3 to GITHUB_OUTPUT
Wrote ts=20260412-172939 to GITHUB_OUTPUT

Consumer job output:

=== Data received from producer job ===
Version: 1.2.3
Timestamp: 20260412-172939
Producer result: success
 
Key point: GITHUB_ENV does NOT cross job boundaries.
Only values declared in jobs.<id>.outputs and written
to GITHUB_OUTPUT are available via needs.<id>.outputs.

GITHUB_ENV does not cross job boundaries

  • Variables set via $GITHUB_ENV are only available to subsequent steps within the same job
  • A downstream job using needs: cannot read $GITHUB_ENV from the upstream job
  • Attempting to access it will silently return an empty string — no error is raised

Use GITHUB_OUTPUT for cross-job data

  • Write values to $GITHUB_OUTPUT and promote them via jobs.<id>.outputs
  • Downstream jobs read them via needs.<id>.outputs.<name>
  • For large payloads (>1 KB), use artifacts instead of outputs (outputs are limited to 1 MB total per job)
ChannelScopeSetterReaderCross-job
GITHUB_OUTPUTstep → step/jobecho "key=value" >> "$GITHUB_OUTPUT"steps.<id>.outputs.<key>Yes (via jobs.<id>.outputs)
GITHUB_ENVstep → stepecho "KEY=value" >> "$GITHUB_ENV"$KEY in subsequent stepsNo
env: (workflow/job)workflow/jobYAML declaration${{ env.KEY }} or $KEYNo
vars.*repo/env/orgGitHub Settings${{ vars.NAME }}Yes (static)
secrets.*repo/env/orgGitHub Settings${{ secrets.NAME }}Yes (static)

data flow | artifacts

Artifacts are files uploaded during a workflow run that persist beyond the job’s lifetime. They enable cross-job data passing and post-run inspection of build outputs, test reports, and manifests.

Upload and download artifacts across jobs

When a build job produces files that a deploy or test job needs, or when you need to preserve test reports for later inspection. It is typically triggered by any event — artifacts are a data flow mechanism. Artifacts are scoped to the workflow run. Cross-run artifact sharing requires the GitHub API. Different artifact names within the same run are independent. Pass build outputs from a build job to a deploy job, and preserve test reports with longer retention.

Upload build artifacts and test reports in a build job, download them in deploy and summary jobs.

name: "Demo: Artifacts"
on:
  push:
    branches: [main]
    paths:
      - ".github/workflows/demo-artifacts.yml"
  workflow_dispatch:
 
permissions:
  contents: read
 
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Generate build artifacts
        run: |
          mkdir -p dist reports
          echo '{"version": "1.0.0", "build": "${{ github.run_number }}"}' > dist/manifest.json
          echo "SELECT COUNT(*) FROM orders;" > dist/validation_query.sql
          echo "Test results: 42 passed, 0 failed" > reports/test-results.txt
          echo "Coverage: 87.3%" > reports/coverage.txt
          echo "Generated artifacts in dist/ and reports/"
          ls -la dist/ reports/
 
      - name: Upload build artifacts
        uses: actions/upload-artifact@v4
        with:
          name: build-output
          path: dist/
          retention-days: 5
 
      - name: Upload test reports
        uses: actions/upload-artifact@v4
        with:
          name: test-reports
          path: reports/
          retention-days: 30
 
  deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: Download build artifacts
        uses: actions/download-artifact@v4
        with:
          name: build-output
          path: ./downloaded-build
 
      - name: Verify downloaded artifacts
        run: |
          echo "=== Downloaded artifacts ==="
          ls -la downloaded-build/
          echo ""
          echo "Manifest contents:"
          cat downloaded-build/manifest.json
          echo ""
          echo "SQL contents:"
          cat downloaded-build/validation_query.sql
 
  summary:
    needs: [build, deploy]
    runs-on: ubuntu-latest
    steps:
      - name: Download all artifacts
        uses: actions/download-artifact@v4
        with:
          path: ./all-artifacts
 
      - name: Show all artifacts
        run: |
          echo "=== All downloaded artifacts ==="
          find ./all-artifacts -type f -exec echo {} \; -exec cat {} \; -exec echo "---" \;

Workflow run output (run 24312297699, triggered by push to main):

✓ main Demo: Artifacts · 24312297699
Triggered via push
 
JOBS
✓ build in 4s (ID 70983934624)
✓ deploy in 6s (ID 70983939017)
✓ summary in 4s (ID 70983945066)
 
ARTIFACTS
test-reports
build-output

Build job output:

Artifact build-output.zip successfully finalized. Artifact ID 6394260342
Artifact build-output has been successfully uploaded! Final size is 337 bytes.
Artifact test-reports.zip successfully finalized. Artifact ID 6394260393
Artifact test-reports has been successfully uploaded! Final size is 316 bytes.

Artifact retention defaults silently delete data

  • Default retention is 90 days — after that, artifacts are permanently deleted
  • For compliance-critical outputs (audit logs, signed attestations), 90 days may not be enough
  • Repository-level retention settings override per-upload retention-days if the per-upload value is higher

Set explicit retention and name artifacts clearly

  • Use retention-days: on every upload to match your operational needs
  • Use descriptive names: build-output-${{ github.sha }} instead of artifact
  • For long-term storage, upload to external storage (GCS, S3) instead of relying on GitHub artifact retention
KeyActionDefaultDescription
nameuploadartifactUnique name for the artifact within the run
pathuploadrequiredFile or directory to upload
retention-daysupload90Days to keep the artifact (max 400 public, configurable)
if-no-files-founduploadwarnBehavior when path matches no files: warn, error, or ignore
compression-levelupload6zlib compression level (0=none, 9=max)
overwriteuploadfalseReplace an existing artifact with the same name
namedownloadallDownload a specific artifact or all if omitted
pathdownload.Directory to extract to — each artifact gets a subdirectory

data flow | caching

Caching persists dependencies and build outputs across workflow runs to avoid redundant downloads. The cache is keyed by an exact string and scoped to the branch where it was created plus the default branch.

Design cache keys with fallback

When a workflow installs dependencies (pip, npm, Maven) or builds artifacts that rarely change. It is typically triggered by any event — caching applies at the step level. Caches are scoped to the branch and the default branch. A PR branch can read caches from main but not from other PR branches. Cache entries expire after 7 days of no access, with a 10 GB total limit per repository. Avoid downloading and installing the same pip packages on every run.

Cache pip packages with a hash-based key and OS-based fallback.

name: "Demo: Caching"
on:
  push:
    branches: [main]
    paths:
      - ".github/workflows/demo-cache.yml"
      - "requirements.txt"
  workflow_dispatch:
 
permissions:
  contents: read
 
jobs:
  cache-demo:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
 
      - name: Cache pip packages
        id: pip-cache
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: pip-${{ runner.os }}-${{ hashFiles('requirements.txt') }}
          restore-keys: |
            pip-${{ runner.os }}-
 
      - name: Show cache result
        run: |
          echo "=== Cache result ==="
          echo "Cache hit: ${{ steps.pip-cache.outputs.cache-hit }}"
          echo ""
          if [ "${{ steps.pip-cache.outputs.cache-hit }}" = "true" ]; then
            echo "Cache was restored from an exact key match."
          else
            echo "Cache miss — packages will be downloaded and cached after the job."
            echo "If a restore-key matched, a partial cache was restored."
          fi
 
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          echo ""
          echo "Installed packages:"
          pip list --format=columns | head -20

Workflow run output (run 24313241469, triggered by push to main):

✓ main Demo: Caching · 24313241469
Triggered via push
 
JOBS
✓ cache-demo in 9s (ID 70986415672)

Stale or poisoned caches in sensitive workflows

  • A cache entry from a previous run could contain outdated or malicious content
  • If a dependency is compromised and cached, the poisoned cache persists until the key changes or the cache expires
  • PR workflows on fork branches can read (but not write) caches from the default branch

Defensive cache patterns

  • Include the lock file hash in the cache key: key: pip-${{ runner.os }}-${{ hashFiles('requirements.txt') }}
  • Any change to dependencies produces a new key, forcing a fresh download
  • For security-sensitive workflows, add ${{ github.run_id }} to the key to prevent cache reuse entirely
  • Periodically delete stale caches via gh actions-cache delete
KeyDefaultDescription
pathrequiredDirectory or file to cache
keyrequiredExact cache key — a match skips the download
restore-keysnoneOrdered prefix fallbacks for partial matches
enableCrossOsArchivefalseAllow restoring caches created on a different OS
fail-on-cache-missfalseFail the step if no cache is found
lookup-onlyfalseCheck if a cache exists without downloading it
save-alwaysfalseSave the cache even if the job fails

data flow | job summaries and annotations

Job summaries write GitHub-flavored markdown to the workflow run page. Annotations attach notices, warnings, or errors to specific files and lines, visible in the PR diff.

Write a markdown job summary

When a workflow produces human-readable results (test reports, build stats, deployment URLs) that should be visible without downloading artifacts. It is typically triggered by any event. $GITHUB_STEP_SUMMARY is a file path that accepts markdown. Multiple steps can append to it. Maximum 1 MiB per step, 1 MiB total per job. Surface a build report with run metadata directly on the workflow run page.

Write a markdown table and additional notes to the job summary.

name: "Demo: Job Summary"
on:
  push:
    branches: [main]
    paths:
      - ".github/workflows/demo-summary.yml"
  workflow_dispatch:
 
permissions:
  contents: read
 
jobs:
  create-summary:
    runs-on: ubuntu-latest
    steps:
      - name: Write markdown summary
        run: |
          echo "## Build Report" >> "$GITHUB_STEP_SUMMARY"
          echo "" >> "$GITHUB_STEP_SUMMARY"
          echo "| Metric | Value |" >> "$GITHUB_STEP_SUMMARY"
          echo "|--------|-------|" >> "$GITHUB_STEP_SUMMARY"
          echo "| Run | #${{ github.run_number }} |" >> "$GITHUB_STEP_SUMMARY"
          echo "| Branch | \`${{ github.ref_name }}\` |" >> "$GITHUB_STEP_SUMMARY"
          echo "| Actor | ${{ github.actor }} |" >> "$GITHUB_STEP_SUMMARY"
          echo "| Status | :white_check_mark: Passed |" >> "$GITHUB_STEP_SUMMARY"
          echo "" >> "$GITHUB_STEP_SUMMARY"
          echo "> GITHUB_STEP_SUMMARY accepts full GitHub-flavored markdown." >> "$GITHUB_STEP_SUMMARY"
          echo "> Multiple steps can append to the same summary." >> "$GITHUB_STEP_SUMMARY"
          echo "" >> "$GITHUB_STEP_SUMMARY"
          echo "Summary written to GITHUB_STEP_SUMMARY."
 
      - name: Append to summary
        run: |
          echo "### Additional Notes" >> "$GITHUB_STEP_SUMMARY"
          echo "- Each step appends to the same file" >> "$GITHUB_STEP_SUMMARY"
          echo "- The summary renders on the workflow run page in GitHub" >> "$GITHUB_STEP_SUMMARY"
          echo "- Maximum size: 1 MiB per step, 1 MiB total per job" >> "$GITHUB_STEP_SUMMARY"
          echo "Appended additional notes to summary."

Workflow run output (run 24312297701, triggered by push to main):

✓ main Demo: Job Summary · 24312297701
Triggered via push
 
JOBS
✓ create-summary in 5s (ID 70983934669)

Job output:

Summary written to GITHUB_STEP_SUMMARY.
Appended additional notes to summary.

When summaries replace artifacts

  • Use summaries for human-readable reports that reviewers need to see immediately (test results, plan output, deployment URLs)
  • Use artifacts for machine-readable files that downstream jobs consume (manifests, binaries, coverage JSON)
  • Summaries render directly on the run page — no download required

Add annotations to files

Annotations attach messages to specific files and lines, appearing inline in the PR diff and on the workflow run page. Use the ::notice, ::warning, and ::error workflow commands.

Add an annotation to a specific file and line.

      - name: Annotate code
        run: |
          echo "::notice file=src/main.py,line=42::This function needs documentation"
          echo "::warning file=src/config.py,line=10::Deprecated configuration key"
          echo "::error file=src/auth.py,line=5::Missing input validation"
CommandSeverityEffect
::notice file=F,line=L::msginfoBlue badge on the run, inline annotation in PR diff
::warning file=F,line=L::msgwarningYellow badge on the run, inline annotation in PR diff
::error file=F,line=L::msgerrorRed badge on the run, inline annotation in PR diff
::group::title / ::endgroup::groupingCollapsible section in the log output

Ways to control execution, timing, and conditions

Control flow patterns determine when and whether jobs and steps execute. They cover conditional logic, concurrency management, timeouts, error handling, and path filtering.

control flow | conditional expressions

The if: key on jobs and steps accepts expressions that evaluate to a boolean. Expressions can test context values, use status check functions, and perform string operations.

Use status check functions

Status check functions test the aggregate result of all previous steps in a job. They are the primary mechanism for conditional execution after failures.

FunctionReturns true whenDefault behavior
success()All previous steps succeededImplicit — every step has if: success() unless overridden
failure()Any previous step failedStep only runs after a failure
always()Always — even if the run is cancelledStep runs regardless of any outcome, including cancellation
cancelled()The workflow run was cancelledStep only runs on cancellation

always() runs even when the workflow is cancelled

  • if: always() means the step executes even when a user cancels the run or a concurrency group cancels it
  • This may not be desired for cleanup steps that should only run on success or failure

Use failure() || cancelled() instead of always()

  • For cleanup that should run on failure but not on cancellation: if: failure()
  • For cleanup that should run on both failure and cancellation but not on success: if: failure() || cancelled()
  • Reserve always() for steps that truly must run in every case (e.g., releasing a lock)

Guard with context-based conditions

Expressions can test any context value to conditionally run steps or jobs based on the event, branch, actor, or labels.

Common conditional patterns.

      # Run only on main branch
      - if: github.ref == 'refs/heads/main'
 
      # Run only on pull requests
      - if: github.event_name == 'pull_request'
 
      # Run only for a specific actor
      - if: github.actor == 'dependabot[bot]'
 
      # Run only when a PR has a specific label
      - if: contains(github.event.pull_request.labels.*.name, 'deploy')
 
      # Ternary pattern using && / ||
      - run: echo "env=${{ github.ref == 'refs/heads/main' && 'production' || 'staging' }}"

Type coercion gotchas in expressions

  • The string "false" is truthy — only the boolean false, the number 0, and null are falsy
  • Version numbers like 3.10 are parsed as the float 3.1 unless quoted: always use "3.10" in matrix values
  • fromJSON() silently returns null on invalid JSON — no error is raised, and downstream comparisons may pass unexpectedly

Defensive expression patterns

  • Always quote version numbers in matrix values: ["3.10", "3.11", "3.12"]
  • Test fromJSON() results for null before using them: if: fromJSON(steps.data.outputs.config) != null
  • Use == true or == 'true' explicitly instead of relying on truthy evaluation

control flow | concurrency

The concurrency: key serializes or cancels workflow runs sharing the same group identifier. It prevents conflicting deployments and reduces waste from redundant runs.

Configure concurrency groups

When parallel runs of the same workflow on the same branch would conflict (e.g., deploying to the same environment). It is typically triggered by any event — concurrency applies at the workflow or job level. Concurrency groups are global to the repository. The group identifier is a string that can include expressions. Cancel redundant CI runs on push, or serialize deployments to prevent conflicts.

Configure concurrency to cancel redundant runs on the same branch.

name: "Demo: Concurrency"
on:
  push:
    branches: [main]
    paths:
      - ".github/workflows/demo-concurrency.yml"
  workflow_dispatch:
 
concurrency:
  group: demo-${{ github.ref }}
  cancel-in-progress: true
 
permissions:
  contents: read
 
jobs:
  deploy-simulation:
    runs-on: ubuntu-latest
    steps:
      - name: Show concurrency info
        run: |
          echo "=== Concurrency demo ==="
          echo "Concurrency group: demo-${{ github.ref }}"
          echo "cancel-in-progress: true"
          echo ""
          echo "If another push happens to this branch while this run is active,"
          echo "this run will be cancelled and replaced by the new one."
          echo ""
          echo "Simulating a 30-second deployment..."
 
      - name: Simulate deploy
        run: |
          for i in $(seq 1 6); do
            echo "Deploy step $i/6..."
            sleep 5
          done
          echo "Deployment simulation complete."

Workflow run output (run 24312297705, triggered by push to main):

✓ main Demo: Concurrency · 24312297705
Triggered via push
 
JOBS
✓ deploy-simulation in 33s (ID 70983934713)

cancel-in-progress can kill deployment jobs mid-flight

  • If a deployment is in progress and a new push triggers the same concurrency group with cancel-in-progress: true, the running deployment is cancelled
  • This can leave the target environment in a partially deployed state
  • The cancelled run does not automatically roll back

Separate CI and deploy concurrency groups

  • Use cancel-in-progress: true for CI (lint, test) — cancelling redundant checks is safe
  • Use cancel-in-progress: false for deployments — let the running deploy finish, queue the new one
  • Scope deploy groups to the environment: group: deploy-${{ inputs.environment }}
PatternGroup expressioncancel-in-progressUse case
Per-branch CIci-${{ github.ref }}trueCancel stale CI on new push
Per-environment deploydeploy-${{ inputs.environment }}falseSerialize deploys, never cancel
Per-PRpr-${{ github.event.pull_request.number }}trueCancel stale PR checks
Global deploydeploy-productionfalseOne production deploy at a time

control flow | timeouts and error handling

Timeouts prevent runaway jobs from consuming runner hours. The continue-on-error flag controls whether a failed step or job blocks downstream execution.

Configure timeouts and continue-on-error

When a job or step could hang indefinitely, or when a step’s failure should not block the rest of the job. It is typically triggered by any event. Default timeout is 360 minutes (6 hours). Step-level timeout overrides job-level for that step. continue-on-error applies independently of timeouts. Prevent stuck jobs from burning runner hours and allow flaky steps to fail without blocking the pipeline.

Set job-level and step-level timeouts.

name: "Demo: Timeout"
on:
  push:
    branches: [main]
    paths:
      - ".github/workflows/demo-timeout.yml"
  workflow_dispatch:
 
permissions:
  contents: read
 
jobs:
  timeout-demo:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - name: Show timeout config
        run: |
          echo "=== Timeout demo ==="
          echo "Job timeout: 5 minutes (set via timeout-minutes at job level)"
          echo ""
          echo "Default timeout: 360 minutes (6 hours) if not specified."
          echo "Maximum: 360 minutes for GitHub-hosted runners."
          echo ""
          echo "Best practice: always set an explicit timeout."
          echo "A stuck job at default timeout burns 6 hours of runner time."
 
      - name: Step with timeout
        timeout-minutes: 1
        run: |
          echo "This step has a 1-minute timeout."
          echo "Step-level timeout overrides job-level for this step."
          echo "Simulating a quick task..."
          sleep 2
          echo "Task completed within timeout."

Workflow run output (run 24312297693, triggered by push to main):

✓ main Demo: Timeout · 24312297693
Triggered via push
 
JOBS
✓ timeout-demo in 6s (ID 70983934668)

Demonstrate step-level and job-level continue-on-error with outcome vs conclusion.

name: "Demo: Continue on Error"
on:
  push:
    branches: [main]
    paths:
      - ".github/workflows/demo-continue-on-error.yml"
  workflow_dispatch:
 
permissions:
  contents: read
 
jobs:
  step-level:
    runs-on: ubuntu-latest
    steps:
      - name: Failing step (with continue-on-error)
        id: flaky
        continue-on-error: true
        run: |
          echo "This step will fail but the job continues."
          exit 1
 
      - name: Check outcome after failure
        run: |
          echo "=== Step-level continue-on-error ==="
          echo "flaky.outcome: ${{ steps.flaky.outcome }}"
          echo "flaky.conclusion: ${{ steps.flaky.conclusion }}"
          echo ""
          echo "outcome = raw result: failure"
          echo "conclusion = after continue-on-error applied: success"
          echo ""
          echo "The job status is still 'success' because"
          echo "continue-on-error converted the failure."
 
      - name: Conditional cleanup
        if: ${{ steps.flaky.outcome == 'failure' }}
        run: echo "Running cleanup because the flaky step failed."
 
  job-level:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - name: Fail the entire job
        run: |
          echo "This job will fail, but downstream jobs still run"
          echo "because continue-on-error is set at the JOB level."
          exit 1
 
  downstream:
    needs: job-level
    runs-on: ubuntu-latest
    steps:
      - name: Check upstream job
        run: |
          echo "=== Job-level continue-on-error ==="
          echo "job-level.result: ${{ needs.job-level.result }}"
          echo ""
          echo "The upstream job failed, but this job still ran because"
          echo "the upstream had continue-on-error: true at the job level."
          echo ""
          echo "WARNING: Job-level continue-on-error makes the overall"
          echo "workflow show as 'success' even when jobs fail."
          echo "Use sparingly — it hides real failures."

Workflow run output (run 24312297700, triggered by push to main):

✓ main Demo: Continue on Error · 24312297700
Triggered via push
 
JOBS
X job-level in 4s (ID 70983934633)
✓ step-level in 2s (ID 70983934634)
✓ downstream in 2s (ID 70983938933)

Step-level job output:

=== Step-level continue-on-error ===
flaky.outcome: failure
flaky.conclusion: success
 
outcome = raw result: failure
conclusion = after continue-on-error applied: success

Job-level continue-on-error hides real failures

  • The overall workflow reports success even though the job failed
  • Required status checks on PRs will pass despite the failure
  • Use job-level continue-on-error only for genuinely optional jobs (e.g., experimental matrix combinations)

Prefer step-level continue-on-error

  • Step-level gives you outcome vs conclusion to inspect the raw result
  • Use if: steps.<id>.outcome == 'failure' for targeted cleanup or notification
  • Keep job-level continue-on-error for optional matrix combinations marked with experimental: true

control flow | path filtering

Path filters limit workflow triggers to changes in specific files or directories. This is essential for monorepos where unrelated changes should not trigger unrelated workflows.

Filter by paths on the trigger

Path filtering at the trigger level prevents the entire workflow from running when changes are outside the specified paths.

Filter a workflow to only run on changes in the services/api directory.

on:
  push:
    branches: [main]
    paths:
      - "services/api/**"
      - "shared/lib/**"
    paths-ignore:
      - "**/*.md"
      - "docs/**"

Detect changes with dorny/paths-filter

For per-job path filtering within a single workflow (monorepo pattern), use dorny/paths-filter to detect which directories have changes and conditionally run downstream jobs.

Detect changes per service and conditionally run jobs.

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      api: ${{ steps.filter.outputs.api }}
      web: ${{ steps.filter.outputs.web }}
      infra: ${{ steps.filter.outputs.infra }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            api:
              - 'services/api/**'
            web:
              - 'services/web/**'
            infra:
              - 'terraform/**'
 
  test-api:
    needs: detect-changes
    if: needs.detect-changes.outputs.api == 'true'
    runs-on: ubuntu-latest
    steps:
      - run: echo "Running API tests..."
 
  test-web:
    needs: detect-changes
    if: needs.detect-changes.outputs.web == 'true'
    runs-on: ubuntu-latest
    steps:
      - run: echo "Running web tests..."

Required status checks and skipped jobs

  • When a job is skipped via if: on a path filter, its status check is not reported — not failed, not passed, just absent
  • If that job is a required status check on the PR, the PR cannot be merged
  • This is the single most common frustration with path filtering in monorepos

Report skipped jobs as passing

  • Add an if: always() workaround job that reports success when the conditional job is skipped:
  api-status:
    needs: test-api
    if: always()
    runs-on: ubuntu-latest
    steps:
      - run: |
          if [ "${{ needs.test-api.result }}" = "failure" ]; then
            exit 1
          fi
  • Make api-status the required check instead of test-api

Architecture patterns for larger workflow systems

Architecture patterns combine composition, data flow, and control flow into production-grade designs for specific organizational needs.

architecture | monorepo workflows

Monorepo workflows use path filtering and change detection to run only the CI/CD that is relevant to the changed code. The key challenge is making required status checks work with conditionally skipped jobs.

The path filtering and dorny/paths-filter patterns in the Control Flow section above are the building blocks for monorepo workflows. The complete monorepo architecture additionally requires:

  • Per-service workflow files with path-scoped triggers
  • A shared library of reusable workflows for common patterns (lint, test, deploy)
  • Branch protection rules with the api-status workaround pattern for required checks
  • CODEOWNERS to route PR reviews to the team owning each service directory

architecture | release automation

Release automation patterns standardize how versions are bumped, changelogs are generated, tags are created, and packages are published.

Automate releases with Release Please

After merging Conventional Commits to main — Release Please creates a release PR that bumps the version and generates a changelog. It is typically triggered by push to main (for the Release Please action) and release: published (for publishing). Requires contents: write and pull-requests: write permissions. Remove manual version management — merge PRs with conventional commit messages, and Release Please handles the rest.

Release Please + PyPI publish with OIDC trusted publishing.

name: Release
on:
  push:
    branches: [main]
 
permissions:
  contents: write
  pull-requests: write
 
jobs:
  release-please:
    runs-on: ubuntu-latest
    outputs:
      release_created: ${{ steps.release.outputs.release_created }}
      tag_name: ${{ steps.release.outputs.tag_name }}
    steps:
      - uses: googleapis/release-please-action@v4
        id: release
        with:
          release-type: python
 
  publish:
    needs: release-please
    if: needs.release-please.outputs.release_created == 'true'
    runs-on: ubuntu-latest
    permissions:
      id-token: write
    environment: pypi
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install build && python -m build
      - uses: pypa/gh-action-pypi-publish@release/v1

OIDC trusted publishing for PyPI

  • PyPI supports OIDC: no API token needed — the GitHub Actions runner proves its identity via OIDC
  • Configure the trusted publisher on pypi.org under your project settings
  • Requires permissions: id-token: write and an environment: pypi with protection rules

architecture | self-hosted runner routing

Self-hosted runners provide custom hardware, pre-installed tools, network access to internal resources, and cost control. Label-based routing directs jobs to the appropriate runner.

Route jobs by runner labels

Use label arrays to target specific runner configurations.

jobs:
  build:
    runs-on: [self-hosted, linux, gpu]  # requires all three labels
 
  test:
    runs-on: ubuntu-latest  # GitHub-hosted for portability
 
  deploy:
    runs-on: [self-hosted, linux, production]  # production-network runner

Self-hosted runners with public repos

  • Public repository forks can trigger workflows on your self-hosted runners
  • A malicious fork PR could execute arbitrary code on your internal network
  • GitHub-hosted runners are ephemeral and isolated; self-hosted runners may persist state between runs

Secure self-hosted runner usage

  • Use self-hosted runners only with private repositories or organization-scoped runner groups
  • Enable --ephemeral mode so the runner accepts one job and then re-registers (JIT pattern)
  • Use pull_request_target with explicit checkout controls for fork PR workflows
  • Restrict runner groups to specific repositories via organization settings
Patternruns-onUse case
GitHub-hosted onlyubuntu-latestDefault for all CI — no setup overhead
Self-hosted by label[self-hosted, linux, gpu]GPU training, internal network access
Dynamic selectionMatrix with ${{ matrix.runner }}Hosted for PRs, self-hosted for deploys
Ephemeral/JIT--ephemeral flag on registrationOne-shot runners for security isolation
Larger runnersubuntu-latest-8-coresCompute-intensive builds on GitHub’s infra

architecture | security patterns

Security patterns protect the CI/CD pipeline from supply-chain attacks, credential leaks, and over-privileged workflows.

Pin actions by SHA

Reference actions by their full commit SHA instead of a mutable tag.

      # Mutable tag — can be replaced by the action author
      - uses: actions/checkout@v4
 
      # Immutable SHA pin — locked to a specific commit
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683  # v4.2.2

Mutable action tags are a supply-chain risk

  • Version tags like v4 can be moved to point to a different commit at any time
  • A compromised action author could replace the tag with malicious code
  • Every workflow run using the tag would execute the compromised version

SHA pin all third-party actions

  • Use the full 40-character commit SHA: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
  • Add a comment with the version: # v4.2.2
  • Use Dependabot or Renovate to automate SHA pin updates when new versions are released
  • First-party actions (actions/checkout, actions/cache) are lower risk but should still be pinned

Configure least-privilege permissions

Set explicit permissions at the workflow level — all unspecified scopes default to none.

name: "Demo: Permissions"
on:
  push:
    branches: [main]
    paths:
      - ".github/workflows/demo-permissions.yml"
  workflow_dispatch:
 
permissions:
  contents: read
 
jobs:
  least-privilege:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
      - name: Show permissions info
        run: |
          echo "=== Permissions demo ==="
          echo "This job has read-only access to repository contents."
          echo ""
          echo "When 'permissions' is set at workflow level:"
          echo "  - All unspecified scopes default to 'none'"
          echo "  - This is least-privilege: only grant what you need"
          echo ""
          echo "Without a permissions block:"
          echo "  - GITHUB_TOKEN gets the repo's default permissions"
          echo "  - For public repos: read for all scopes"
          echo "  - For private repos: read/write for most scopes"
          echo ""
          echo "SECURITY: Always set explicit permissions."
          echo "Over-broad defaults are a common attack vector."
 
      - name: Verify read-only access
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          echo "Attempting to read repo info (should succeed)..."
          gh api repos/${{ github.repository }} --jq '.full_name' || echo "Read failed"
          echo ""
          echo "The GITHUB_TOKEN for this job cannot write to the repo."

Workflow run output (run 24312297703, triggered by push to main):

✓ main Demo: Permissions · 24312297703
Triggered via push
 
JOBS
✓ least-privilege in 3s (ID 70983934659)
ScopeReadWriteCommon use
contentscheckout, read filespush commits, create releasesMost workflows
pull-requestsread PR datacomment, approve, labelPR automation
issuesread issuescreate, comment, labelIssue automation
packagesread packagespublish packagesPackage publishing
id-token—request OIDC tokenCloud authentication
actionsread workflow datamanage cache, approve runsWorkflow management
deploymentsread deploymentscreate deployment statusDeployment tracking
security-eventsread alertsupload SARIF resultsSecurity scanning
statusesread commit statusescreate commit statusesCustom status checks
attestations—create attestationsSupply-chain provenance

architecture | notifications and status

Notification patterns inform teams about workflow outcomes through external channels (Slack, email, PR comments) or GitHub-native mechanisms (commit statuses, check runs).

Notify Slack on workflow failure

Send a Slack notification when any job in the workflow fails.

  notify-failure:
    needs: [build, test, deploy]
    if: failure()
    runs-on: ubuntu-latest
    steps:
      - uses: slackapi/slack-github-action@v2.1.0
        with:
          webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
          webhook-type: incoming-webhook
          payload: |
            {
              "text": ":x: Workflow *${{ github.workflow }}* failed on `${{ github.ref_name }}`",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": ":x: *${{ github.workflow }}* failed\n*Branch:* `${{ github.ref_name }}`\n*Actor:* ${{ github.actor }}\n*Run:* <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Run>"
                  }
                }
              ]
            }

Comment deployment status on PRs

Post a deployment URL as a PR comment after a successful preview deployment.

      - uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `## Preview Deployment\n\n:white_check_mark: Deployed to: ${process.env.DEPLOY_URL}\n\nCommit: \`${context.sha.substring(0, 7)}\``
            })
        env:
          DEPLOY_URL: ${{ steps.deploy.outputs.url }}

architecture | cost optimization

Cost optimization patterns reduce runner minutes, cache usage, and API calls without sacrificing CI quality.

PatternMechanismSavings
Timeoutstimeout-minutes: on every jobPrevents 6-hour runaway jobs
Concurrency cancellationcancel-in-progress: trueKills redundant CI runs on rapid pushes
Path filteringon.push.paths or dorny/paths-filterSkips irrelevant workflows entirely
Skip CIif: !contains(github.event.head_commit.message, '[skip ci]')Lets doc-only commits skip CI
Aggressive cachingactions/cache for pip, npm, Docker layersAvoids re-downloading dependencies
Runner sizingubuntu-latest for most, larger runners only when neededAvoids paying for unused CPU/RAM
Matrix pruningexclude + max-parallelReduces combination count and concurrent runners
Artifact cleanupLow retention-days, avoid uploading large binariesReduces storage costs

Quick wins for data-engineering workflows

  • Cache pip and dbt packages — ~/.cache/pip and ~/.dbt/packages
  • Set timeout-minutes: 15 on BigQuery validation jobs — they should never run more than a few minutes
  • Use path filtering to skip pipeline CI when only docs or dashboards change
  • Run expensive warehouse tests (full table scans, backfill validation) only on PRs to main, not on every push

Pattern choices for common data-engineering workflows

This table maps patterns from this page to common data-engineering CI/CD needs. Each scenario references the pattern category and specific section.

ScenarioPatternSection
Test Python pipelines across 3.11 and 3.12Matrix strategyComposition: matrix
Standardize dbt CI across 10 reposReusable workflowsComposition: reusable
Setup Python + pip cache in one stepComposite actionComposition: composite
Deploy staging after CI passesWorkflow chainingComposition: chaining
Pass dbt manifest from build to deployArtifactsData flow: artifacts
Cache pip and dbt packagesCachingData flow: caching
Surface BQ dry-run cost in PRJob summariesData flow: summaries
Skip CI on config-only changesPath filteringControl flow: paths
Serialize expensive BQ backfill jobsConcurrencyControl flow: concurrency
CI for services + infra + pipelinesMonorepoArchitecture: monorepo
Auto-version dbt packagesRelease automationArchitecture: release
Notify on pipeline failureNotificationsArchitecture: notifications
SHA-pin all third-party actionsSecurityArchitecture: security
Limit BigQuery job timeoutsCost optimizationArchitecture: cost

Common pattern failures and how to fix them

ProblemSymptomFix
Matrix combination skippedJob shows “skipped” in UICheck exclude rules — the combination may be excluded
Reusable workflow not founderror: .github/workflows/X.yml not foundVerify the file path and ref — cross-repo needs @ref
Composite action missing shellA shell must be specifiedAdd shell: bash to every run: step in composite actions
Cache never hitscache-hit always falseCheck cache key — hashFiles() is case-sensitive, paths must match exactly
Cache scope isolationPR can’t read main’s cacheCaches are branch-scoped — PRs can read from the default branch
Concurrency cancels wrong runExpected run cancelledCheck concurrency group expression — overly broad groups cancel unrelated runs
Required check missingPR blocked, check never reportsSkipped jobs don’t report status — use the if: always() workaround
Artifact not foundUnable to find artifactArtifact names are case-sensitive — verify exact match
Timeout too aggressiveJob cancelled mid-deployIncrease timeout-minutes — measure actual duration first
continue-on-error hides failureWorkflow green despite failureMove to step-level and check outcome instead of conclusion
fromJSON returns nullUnexpected conditional behaviorValidate JSON input — fromJSON() silently returns null on parse error
secrets: inherit leaks secretsAll caller secrets passed to calleeSwitch to explicit secret enumeration
Mutable tag compromisedAction behavior changed unexpectedlySHA-pin the action and add a version comment
Self-hosted runner persists stateTest pollution between runsEnable --ephemeral mode or clean workspace in pre-job hook
Summary exceeds limitSummary truncated or missingEach step: 1 MiB max. Split content across steps or use artifacts

Operating rules for production-ready workflow patterns

  1. Pin all third-party actions by SHA — use Dependabot or Renovate to keep them updated.
  2. Set explicit permissions: on every workflow — never rely on repository defaults.
  3. Set timeout-minutes: on every job — the 6-hour default wastes money on stuck jobs.
  4. Use cancel-in-progress: true for CI, false for deploys — redundant CI is waste, cancelled deploys are dangerous.
  5. Prefer composite actions for step-level reuse, reusable workflows for job-level reuse — do not use reusable workflows to share a 3-step setup sequence.
  6. Cache dependencies aggressively — include the lock file hash in the key and the OS as a prefix.
  7. Pass secrets explicitly, never use secrets: inherit cross-repo — document which secrets each callee needs.
  8. Use path filtering in monorepos — both at the trigger level and with dorny/paths-filter for per-job control.
  9. Write job summaries instead of downloading artifacts for human-readable output — summaries render inline.
  10. Test with workflow_dispatch before relying on push triggers — manual dispatch lets you iterate without pushing commits.

Key pattern primitives at a glance

PatternKeyWherePurpose
Matrixstrategy.matrixjobFan out across combinations
Fail-faststrategy.fail-fast: falsejobLet all matrix jobs complete
Max parallelstrategy.max-parallel: NjobLimit concurrent matrix jobs
Includestrategy.matrix.includejobAdd variables to specific combos
Excludestrategy.matrix.excludejobRemove specific combos
Reusable calleeon: workflow_callworkflowAccept inputs/secrets, return outputs
Reusable callerjobs.<id>.uses:jobInvoke a reusable workflow
Composite actionruns.using: "composite"action.ymlBundle steps into one action
Workflow chainon: workflow_runworkflowTrigger after another workflow
Manual triggeron: workflow_dispatchworkflowEnable UI/API triggering
Step output$GITHUB_OUTPUTstepPass data between steps
Dynamic env$GITHUB_ENVstepSet env vars for subsequent steps
Job outputjobs.<id>.outputsjobExpose data to downstream jobs
Upload artifactactions/upload-artifact@v4stepPersist files across jobs
Download artifactactions/download-artifact@v4stepRetrieve uploaded files
Cacheactions/cache@v4stepPersist dependencies across runs
Job summary$GITHUB_STEP_SUMMARYstepWrite markdown to run page
Annotation::warning file=F,line=L::msgstepAttach messages to code
Conditionalif:job/stepGate execution on expressions
Status functionsuccess() / failure()if:Test prior step/job results
Concurrency groupconcurrency.group:workflow/jobSerialize or cancel runs
Cancel in progressconcurrency.cancel-in-progressworkflow/jobCancel stale runs
Timeouttimeout-minutes:job/stepPrevent runaway execution
Continue on errorcontinue-on-error: truejob/stepAllow failure without blocking
Path filteron.push.paths:workflowTrigger only on file changes
Change detectiondorny/paths-filter@v3stepPer-directory change flags
SHA pinuses: action@<sha>stepLock to immutable commit
Permissionspermissions:workflow/jobLeast-privilege GITHUB_TOKEN
Self-hostedruns-on: [self-hosted, label]jobTarget specific runners
Release Pleasegoogleapis/release-please-actionstepAutomated version + changelog
Slack notifyslackapi/slack-github-actionstepChannel notifications

Graph View

Backlinks