Housekeeping

• Beth is here to help me!
• We’re taking a picture of us near the end of class. Let Beth know in chat if you’d like to opt out
• Find us in the Maven course community to get help
• Reply to any email help@bretfisher.com

Schedule

Week 1

• Workshop 1: Monday. GitHub Actions for Docker and best practices
• Workshop 2: Wednesday. GitHub Actions reusable workflows
• Q&A 1: Friday. Discussion on implementing GHA

–

Week 2

• Workshop 3: Monday. Argo CD setup
• Workshop 4: Wednesday. Argo CD app deployments
• Q&A 2: Friday. Discussion on implementing Argo CD

Topic links

Why GitHub Actions

1. Software lives on GitHub. GHA is the new standard for automation
2. Over 18k Actions in marketplace. De facto for tool creators
3. GHA growing rapidly as #2. Jenkins (#1) usage declining
4. Rapid improvements
5. Repo-centric forces you to think like dev, IaC-style
6. GHA is free. Runner minutes are what cost
7. Open source repos have unlimited minutes
8. 100% free with self-hosted runners

GitHub Actions weaknesses

• Repo-centric (forces good design, but…)
• No built-in org or project-wide action dashboards
• Hard to find where to manually run one (called a "workflow_dispatch")
• Doesn’t replace Backstage dev portal (but can enhance it)
• Scheduled workflows stop firing if no commits for 60 days
• Local running of workflows isn’t an official feature and can be bumpy with (act)
• CD features aren’t great yet (That’s OK, we’re using GitOps)

GitHub Action workflows

Stick a YAML file in .github/workflows/ for each workflow with this basic structure:

name: learn-github-actions # must be unique in repo
on: [push, pull_request] # or dozens of other events
jobs:
  check-bats-version: # Multiple jobs run in parallel by default
    runs-on: ubuntu-latest # change this to windows-latest or macos-latest or your own runners
    steps:
      - uses: actions/checkout@v3 # change this to x.x.x to pin to a specific version
      - uses: actions/setup-node@v3 # pin to commit hash for highest security
        with:
          node-version: '14' # `with` lets you pass options to the action
      - run: npm install -g bats # easily run shell commands with `run`
      - run: bats -v

Bret’s GHA example repos

• GitHub Actions DevOps Templates
• Docker Build GitHub Action Template
• GitHub Actions for Docker, step by step: examples and video walkthrough
• Super Linter GitHub Action Template

Introductions!

1. Introduce yourself to the group (Keep it short, 60 seconds or less, please)

2. Where are you Zooming from in the world?

3. Question: What GHA workflows are you wanting to build or migrate from old CI tools?

What are we building and deploying?

• Wordsmith: multi-container solution to randomly display words

What are we building and deploying?

• 2 repos: API and Web (plus a postgres db)

Add simple Docker build via PR

Let’s focus on wordsmith-web for now

1. Copy my sample "Docker Simple" YAML here
2. Navigate to the Actions tab in wordsmith-web
3. Click New workflow, then click setup a workflow yourself
4. Name the new file .github/workflows/docker-build.yml
5. Paste in our sample workflow
6. One thing to change: update tags to our repo (last line)
   • ghcr.io/<orgname>/wordsmith-web:latest
   • ⚠️ <orgname> must be lowercase to work with Docker!
7. Commit changes to a new branch
8. Create a PR for that branch. This will kick off the workflow
9. Let’s watch the Action run (don’t merge yet)

This is a good start, but…

• Only pushes an image when merged to the main branch
• Doesn’t build for multi-arch (Intel + Arm)
  • Click your Zoom Reaction if you have Apple Silicon Mac’s in your team
• Doesn’t use image layer caching
• Only has one (hardcoded) tag :latest
• Doesn’t have image labels like

  org.opencontainers.image.title=wordsmith-api
  org.opencontainers.image.source=https://github.com/MostlyDevOps/wordsmith-api
  org.opencontainers.image.version=gha-4715332302
  org.opencontainers.image.created=2023-04-16T21:16:33.149Z
  org.opencontainers.image.revision=f217725e1bf9e1032debb15df43b06513907d58b

• Requires us to dig into build logs to know the image tag
• We can do better!

Add BuildKit, caching, and multi-arch builds

• Walk through a more advanced example that uses most of the Docker Action features
  • You saw these features in the course prep video on "CI Examples"
• Edit our docker-build.yml and replace all text with our advanced example
• This time change the image name in the docker-meta step to ghcr.io/<orgname>/wordsmith-web
• Commit to the PR branch and watch builds
• If PR Checks are green, merge!

Notes on image tags

Tagging in a production team is an artform

• Docker’s Metadata Action is awesome for auto-tagging and labels
• It’s removed many long shell scripts for teams with custom tagging
• Let’s dig into those tagging rules real quick

tags: |
  type=raw,value=latest,enable=${{ endsWith(github.ref, github.event.repository.default_branch) }}
  type=ref,event=pr
  type=ref,event=branch
  type=semver,pattern={{version}}

Add permissions to every workflow

• By default, Actions might have R/W to everything in repo
• This is set in Settings > Actions at repo or org level
• Let’s ensure we start with a "least privilege" mindset in wordsmith-web

1. Go to org Settings > Actions > General > Workflow Permissions
2. Ensure it’s set to Read repository contents and packages permissions
3. Manually add permissions needed to each workflow (next slide)
4. Permissions can be per workflow or per job (see all perms)

Permission options for GITHUB_TOKEN

• Add the ones you need, it’ll set the rest to none (docs)
• You can do this at root or per-job level

permissions:
  actions: read|write|none
  checks: read|write|none
  contents: read|write|none
  deployments: read|write|none
  id-token: read|write|none
  issues: read|write|none
  discussions: read|write|none
  packages: read|write|none
  pages: read|write|none
  pull-requests: read|write|none
  repository-projects: read|write|none
  security-events: read|write|none
  statuses: read|write|none

• Sometimes you need none (when your workflow doesn’t need to access its repo)

permissions: {}

Improve supply chain security with GitHub settings

More automation means you need more guardrails

• In GitHub repo settings, add a main branch protection rule
• In GitHub Security tab, enable everything
• For bigger teams, add a CODEOWNERS file
• (Shameless plug) 📧 get on my newsletter to get updates on my security steps guide

Real-world image tagging

• Production tags should be immutable
• <something>-<date> or <something>-<git-sha> are good
• Even <something>-<date>-<git-sha> is good
  • This helps humans trace when it was built and what commit it came from
• Keep offering latest for friendly dev use, but don’t use it in prod
• Build and store PR images with a pr-<number> tag
• Maybe, split "CI builds" and "Production-ready builds" into separate repos or registries
• Docker Metadata Action can do all this

Dependabot helps us keep things up to date

• Dependabot is known for updating your app dependencies
• But it can also be configured via YAML to update GH Action versions
• You likely want it to check monthly or weekly

# .github/dependabot.yml in every repo with GHAs
version: 2
updates:
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"

You can do this in the extra credit slides later

• It supports updating your repos for lots of package ecosystems (full list)
• It can update your Dockerfile FROM base image versions
• It can update image versions in Kubernetes manifests & Helm charts

Concurrency limits can save you money

• Avoid multiple commits in the same branch/PR from running at the same time

• Add this at workflow level, before jobs, in every workflow

# cancel any previously-started, yet still active runs of this workflow on the same branch
concurrency:
  group: ${{ github.ref }}-${{ github.workflow }}
  cancel-in-progress: true

You can do this in the extra credit slides later

Workflow design and parallelization

• Generally, one workflow per purpose (build and test image, lint code, deploy)
• Jobs are isolated from each other on different runners
• Jobs run in parallel by default, but:
• Jobs can depend on each other to control order
• Workflows can depend on another one finishing, or call another workflow directly
• It’s OK to only have one job per workflow
• ProTip: Creating a Matrix can simplify parallel testing (split up testing, or test on multiple OSes)
• If your job has dependency services, use service containers or Docker Compose
• Service containers are simpler. Docker Compose means you can easily reuse locally
• Over time, optimize for speed (parallelization) and cost (less minutes). It’s a balance
• Reusable workflows (Wednesday!) will greatly affect your workflow design

Ways to run a Workflow

• Normally a GitHub API event triggers a workflow (push, pull_request, issues)
• You can also run a workflow manually from the Actions tab, CLI, VS Code plugin, or webhook
• Workflows can start by depending on another one finishing

Keep YAML D.R.Y.

• You can have a Job call a "Reusable" Workflow in another repo (learn this Wednesday)
• You can make your own Actions and call them from a Job step in another repo
• You can make your own "Starter" Workflow templates in your Orgs .github repo (not as D.R.Y.)

Workshop 1 GHA further reading for the curious

(full GHA resources list on bretfisher.com)

1. Follow all the latest updates on GitHub Actions:
   1. https://github.blog/changelog/label/actions/
   2. https://github.blog/changelog/label/githubactions/
   3. https://twitter.com/GHchangelog (all changes)
   4. Follow the Roadmap Project. For example, here’s all dependabot feature releases
2. Docker Official Actions Docs
3. How do GitHub’s Public Runners work, and what’s installed on them?
4. List of “Awesome” Actions. Good for ideas on what to run: awesome-actions, (outdated but useful) and useful-actions, Official GitHub Actions, all Verified Actions

Add a PR comment with labels and tags

This will add a popular PR Commenter Action

• Very handy for helping humans know what tags were created

• First, add this highlighted line to the job permissions in docker-build.yml

  permissions:
    contents: read
    packages: write # needed to push docker image to ghcr.io
    pull-requests: write # needed to create and update comments in PRs

• Then, add the Actions steps I created a template for to the end of your Docker build workflow.

• Commit to a new PR and watch builds. Notice a new comment in the PR!

  

Concurrency limits can save you money and time

• Avoid multiple commits in the same branch/PR from running at the same time

• Usually, when you push a new commit, you don’t care if the last commit passed Actions

• This snippet will cancel any previously-started, yet still active runs of this workflow on the same branch

• Add this at workflow root level, usually after on: but before jobs:, in every workflow

# cancel any previously-started, yet still active runs of this workflow on the same branch
concurrency:
  group: ${{ github.ref }}-${{ github.workflow }}
  cancel-in-progress: true

Dependabot creates PRs for outdated Actions

Let’s add a Dependabot config to our wordsmith-web repo

• Create a new file in the .github folder called dependabot.yml

---
# Please see the documentation for all configuration options:
# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates

version: 2
updates:
  # Maintain dependencies for GitHub Actions
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "daily"

Then you’ll see PRs like this:

Create PRs for Docker image updates

This only works for image tags that use semver tags

• Add another package-ecosystem to the dependabot.yml file

  - package-ecosystem: "docker"
    directory: "/"
    schedule:
      interval: "weekly"
    commit-message:
      prefix: "[docker] "

Then you’ll see PRs like this:

This also works for Kubernetes and Helm! Repeat the above YAML for each directory you want to update.

Note: Dependabot can update many dependencies, check it out

What is Super-Linter?

Extra, Extra Credit: Watch my Super-Linter walkthrough live stream

Why lint everything?

1. It doesn’t just change code, it changes culture
2. Helps avoid "one person’s opinion" about style
3. Educates on best practices
4. Avoid "code smells" that lead to bugs
5. Avoid unseen errors before deploys
6. Linters now do security checks too

Add Super-Linter to wordsmith-web

1. In your wordsmith-web repo, go to the Actions tab
2. Don’t use suggestions, click "set up a workflow yourself"
3. Paste in the Super-Linter workflow template
4. Commit to a new PR and study the output/logs of how it works
5. Advanced Reusable Mode. We’ll learn more about Reusable Workflows Wednesday, but if you want to take the leap now: Fork and implement BretFisher/super-linter-workflow

Popcorn Time!

What was your lightbulb moment today about GitHub Actions?

After you speak, pick the next raised hand to speak

Wrap-up

• That’s it for today
• Tonight you’ll get an email with homework for Wednesday
• Wednesday we’ll scale our actions up to multiple repos and add more Workflows!
• Remember extra credit (in Maven) if you have time
• Please throw questions in the Maven class community, or topics you want to cover in Zoom
• For schedule, logistics, and feedback: help@bretfisher.com

The problem with Monday’s GHA approach

• Each Workflow only works in its repo
  • (we’d have to copy/paste to every repo)
• No way to set standards, or track "workflow drift" between repos
• On every Action version update, we have to update every repo
  • (Even with Dependabot, that’s a lot of PRs to review)

Option 1: GitHub template repos

• Cons: Too "brute force" for workflows, and doesn’t update existing repos

Option 2: Actions Starter workflows

• Create an org repo called .github and put workflows in there
• When you create a new Action from the repo web UI, you’ll see them:

• Con: Better, but still doesn’t centrally control workflows across repos
• Con: Only works in private repos if you have GitHub Enterprise Cloud

Option 3: Actions required org workflows

• Setup by a org admin for all repos or a set of repos per workflow

• Pro: can enforce standards org-wide or granularly
• Con: easily becomes a blocker because PRs can’t merge if they fail

Option 4: Create custom Actions

• Like public Actions, but in your own repo for your own org
• Pro: Create in a central repo and consumed in other repos
• Pro: Lots of flexibility, can be built with shell commands, Node.js, or containers
• Con: More complex to setup and maintain
• Con: It can only run one Job per Action
• Big Con: Repo bloat! One repo per custom Action (and you might end up with dozens)

Option 5: Actions reusable workflows!

This is my fav

• Run one workflow from the job of another workflow

name: Docker Build
on:
push:
  branches:
    - main
  pull_request:
jobs:
  call-docker-build:
    name: Call Docker Build
    uses: mostlydevops/actions/.github/workflows/reusable-docker-build.yaml@main
    permissions:
      packages: write
      pull-requests: write
    with:
      image-name: ghcr.io/mostlydevops/wordsmith-web

Advantages of reusable workflows

• Reusable workflows can be centrally managed (one or more repos)
  • (a different team can own them)
• Works across public and private repos
• Ideally, one place to manage action versions and control complexity
  • (depends on how you call them with @semver, @branch, or @sha)
• You can call a workflow in the same repo or a different repo
• You can call multiple reusable workflows from a single calling workflow (20 max)
• You can nest reusable workflows in reusable workflows (4 levels max)
• You can make starter workflows that call reusable workflows
  • (if you’re open source or GitHub Enterprise)
• Allows junior devs to use advanced workflows w/o knowing the internals
• You can design inputs so calling workflows can customize the reusable workflow
• Con: Inputs (how you pass data between workflows) will add a bit of complexity
• Con: Troubleshooting workflows not running gets a bit more complex

Create a central public actions repo

1. Create a new public repo called actions
2. Add a .github/dependabot.yml file with this content:

version: 2
updates:
 - package-ecosystem: "github-actions"
   directory: "/"
   schedule:
     interval: "daily"

Note: If the repo was private, this setting would show up. We would need to change it:

Settings > Actions > General > Access

Extra Credit Recap

Did you do any of the extra credit? Do you have questions?

Pre-Workshop Homework Recap

Did you watch the Reusable Workflow video?

Did you make the actions repo?

Housekeeping

• I updated Monday’s Further reading slide
  • More links to following updates: Public Roadmap
  • More links to finding good Actions

Examples of reusable workflows

Imagine all these being centrally controlled and automated for all software repos in your org

• The advanced Docker Build
• Image CVE Scanning with the Trivy Action
• Linting Action with Super-Linter (50+ linters in one Action)
• Complex testing designs (unit + integration + E2E in one Workflow)
• Smoke test any PR commit by deploying to a k8s cluster and waiting for probes to pass
• Static analysis (code quality, security, etc) with CodeQL
• Create PR preview environments with Uffizzi or Okteto. Built on PR open and destroyed on PR close
• PR bots that auto-assign reviewers, assign labels, etc
• Automatically rebase PRs when they get out of date from main
• Sync labels (names, colors, descriptions) across repos
• Create a OpenSSF scorecard of your repos
• Watch GitHub runners for bad behavior while your workflows run

Process overview: implement a reusable workflow

1. Add a workflow to a repo .github/workflows that you want to reuse elsewhere
   1. This is the "Reusable Workflow"
2. Add a on:workflow_call event to the workflow
3. Add any required or optional inputs and secrets to that event
   1. These act similarly to environment variables during the workflow run
4. Add those input and secret values into the workflow steps
5. Create a "Calling Workflow" job in another repo (a calling job calls a reusable workflow)
6. Add the inputs and secrets to the calling workflows

Add a reusable workflow to the actions repo

<org>/actions/.github/workflows/reusable-trivy.yml

This Trivy workflow will scan our images for vulnerabilities

1. Create a new workflow called reusable-trivy.yml with this content:

name: Trivy Scan Image
on:
  workflow_call:
jobs:
  build:
    name: CVE Image Scan
    runs-on: ubuntu-latest
    steps:
      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@0.11.2
        with:
          image-ref: ghcr.io/${{ github.repository }}:latest
        env:
          TRIVY_USERNAME: ${{ github.actor }}
          TRIVY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}

Save and run that workflow…

2. Commit it to a new branch and make a PR
3. Let’s watch it run in the PR
4. Hmm, it’s not running. Why not?
5. Remember, what events is it watching for?
6. Only workflow_call events. It will ignore push and pull_request events
7. We need to call it from another workflow

Call the workflow from the wordsmith-web repo

1. Create the file .github/workflows/call-trivy.yml and paste this in
2. Change <org>. Then change <pr-branch-name> to the branch you just created in the actions repo
3. ProTip: This is how you test reusable workflows before merging them in either repo
4. GitHub Best Practice: Make this PR draft, because you’ll change it to main branch before merging

name: Call Trivy
on:
 push:
   branches: [main]
 pull_request:
jobs:
 scan:
   name: Scan
   uses: <org>/actions/.github/workflows/reusable-trivy.yml@<pr-branch-name>

• NOTE: This Action run may not succeed if you have uppercase in org or repo name
• Also, how do we change the tag to the one being built in a commit or PR?
• That brings us to inputs!

Popcorn Time!

Question 1: What workflow will you likely build first in your work projects?

Question 2: What workflow would most impress your boss?

After you speak, pick the next raised hand to speak

The problem with reusable workflows

• The calling workflow usually needs to pass data to the reusable workflow
• GitHub limits the ways we can do that. Because security
• GHA calls these workflow inputs
• env/config data can be passed via with and secrets
• Example:

jobs:
  call-docker-build:
    name: Call Docker Build
    uses: bretfisher/docker-build-workflow/.github/workflows/reusable-docker-build.yaml@main
    secrets:
      dockerhub-username: ${{ secrets.DOCKERHUB_USERNAME }}
      dockerhub-token: ${{ secrets.DOCKERHUB_TOKEN }}
    with:
      dockerhub-enable: true
      image-names: ghcr.io/${{ github.repository }}
      tag-rules: |
        type=ref,event=pr
        type=raw,value=gha-${{ github.run_id }}

Passing data to reusable workflows

Reusable workflows accept data via inputs

1. First, let’s add an inputs array to our reusable workflow on:workflow_call: event
2. Next, we pass those input values to the Action steps: ${{ inputs.<input-name> }}
3. Then we’ll add matching with and secrets key:values to our calling workflow

Add input to the Trivy workflow in

<org>/actions/.github/workflows/reusable-trivy.yml

• Add these inputs lines under the workflow_call: event

on:
  workflow_call:
    inputs:
      image:
        description: Image to scan
        required: true
        type: string

• Then pass that value to the trivy action step at bottom

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@0.11.2
        with:
          image-ref: ${{ inputs.image }}

Add input to the wordsmith-web workflow PR

<org>/wordsmith-web/.github/workflows/call-trivy.yml

1. Change the <org> to yours

jobs:
  scan:
    name: Scan
    uses: <org>/actions/.github/workflows/reusable-trivy.yml@<pr-branch-name>
    with:
      image: 'ghcr.io/<org>/wordsmith-web:latest'

2. Commit that and watch the Action run in the wordsmith-web repo

3. We now have a working reusable workflow

But we’re only scanning latest

A CVE scan is more valuable if it’s scanning each PR commit

1. We’ll need to ensure our Docker Build creates a unique image tag for each PR commit

2. Also, we’ll need to wait for Docker Build to finish before trying to scan

• We’ll enhance the Docker Build in today’s Homework!

Bret’s Reusable Workflow Conventions

• Store all reusable workflows in a single repo
  • actions, gha-reusable, or reusable-workflows are all fine names
• Standardize naming of "calling" versus "callable" workflow files
  • Called/callable reusable workflows: reusable-*.yml
  • Calling workflows: call-*.yml
• Permissions should be set in both calling --> reusable
  • Reusable workflow (at workflow level) sets required permissions to run
  • Calling workflow (at job level) sets permissions it gives to reusable
• Don’t mix workflow_call with push or pull_request events in the same file
• Reusable workflow repos should have a local calling workflow to test it
  • This may mean adding a sample Dockerfile and other file types as test data
• Reusable workflow repos- should have a ./template/ full of calling workflow examples
• Example of these conventions bretfisher/docker-build-workflow

Permissions in reusable workflows!

• Below is a "calling" workflow. Notice the Job-based permissions
• We always need to pass necessary permissions to the reusable workflow
• The reusable "called" workflow gets the calling "context", including GITHUB_TOKEN and these perms
• ProTip: Also set required perms in Reusable Workflow. Then, it’ll fail earily if calling doesn’t give them

name: Docker Build
on:
push:
  branches:
    - main
  pull_request:
jobs:
  call-docker-build:
    name: Call Docker Build
    uses: mostlydevops/actions/.github/workflows/reusable-docker-build.yaml@main
    permissions:
      packages: write
      pull-requests: write
    with:
      image-name: ghcr.io/mostlydevops/wordsmith-web

We need to evolve our Docker Build

• In your actions repo

  1. We’ll use my advanced docker-build-workflow template
  2. In your actions repo, create a new file .github/workflows/reusable-docker-build.yml
  3. Copy the contents of this file and commit to main branch
     1. (Since no repo is calling it yet, we’ll shortcut the PR testing process)

• In your wordsmith-web and wordsmith-api repos, replace the workflow

  1. Delete the old workflow file .github/workflows/docker-build.yml
  2. Create a new file .github/workflows/call-docker-build.yml and add this content
  3. Change line 30 uses: to point to your new reusable workflow file
  4. Commit that to a PR branch and watch the Action run in a wordsmith-web PR
  5. You may have minor issues. Check errors. Check Actions tab (troubleshooting required)
  6. Once it checks green, merge the PR and add the same calling workflow to wordsmith-api

Bret’s Production-Ready Docker Build Workflow

Once you got that workflow working, you’ll be ready for Extra Credit and Argo CD next week

  If you didn't already, read through the comments in the calling and called workflows

Features of note:

1. Lots of inputs for maximum flexibility
2. Lights up all the Docker BuildKit features (caching, multi-platform, SBOM, etc)
3. Adds two new tag rules
   • 1st for PRs that are a unique per-GHA-run tag (that’ll help with Trivy scan)
   • 2nd for after merge, a typical production tag of stable-<date>-<sha>
4. Uses GHA Outputs of the unique image tag so we can chain jobs (like a CVE Scan) after it finishes

Popcorn Time!

With unlimited time and resources, what workflow would you build for your team?

Workshop 2 GHA further reading for the curious

(full GHA resources list on bretfisher.com)

Tools

• act - Best (only?) way to run Actions locally
• GitHub Actions for VS Code
• gh CLI - GitHub CLI tool that runs GHAs remotely

GHA Official Docs

• Reusable workflows (I reference this often)
• Events that can trigger an action, including workflow_call
• Reusable workflow inputs

Problems with our current Trivy workflow

<org>/wordsmith-web/.github/workflows/call-trivy.yml

1. We need a unique tag for every image build in order to scan it
2. Jobs run on different servers. We need images pushed to GHCR before we can scan them
3. We need to wait for the image to be built before we start a scan

Problem 1: unique tag per GHA workflow run

Solved by implementing our homework: Docker Build Reusable Workflow

• Our advanced Docker Build Reusable Workflow makes a unique tag:
  • wordsmith-web:gha-${{ github.run_id }}
• That tag is unique for every commit in a PR
• We’re only going to use that tag for CI testing
• The Reusable Workflow also has an Output that we can send to the Trivy Job

outputs:
  # only outputs the unique gha- image tag that's unique to each GHA run
  image-tag: ${{ steps.image-tag.outputs.image-tag }}

Problem 2: push images to GHCR before scanning

Solved by implementing our homework: Docker Build Reusable Workflow

• In the real world, you will need an image registry to store CI images
• Once you build the image, you’ll need to re-use it for CVE scanning, testing, smoke testing, etc
• Splitting all this into separate Jobs is great for parallelization
• Sometimes, humans will also want to use this PR image and will need registry access
• Our Reusable Docker Build Workflow is set to always push successful PR builds to GHCR
• Many teams will use a different registry or a different image repo for production builds
• That is called "image promotion". Promoting an image to stable/prod registry once it’s merged

Problem 3: wait for the image to be built

Solved by adding the Trivy job to the Docker Build workflow with a needs: key

• We can chain two Jobs together in the same Workflow when there’s a dependency
• We should already have the reusable-trivy.yml workflow in our actions repo
• Let’s edit the call-docker-build.yml workflow in our wordsmith-web repo
• Add this new job at the end of file
  • Make sure it’s indented correctly
  • Change <org> to your org name
• Commit it to a new branch and PR, and be sure it runs correctly before merging

# 2nd job in Workflow, 2 spaces indented
  scan:
    name: Scan Image
    needs: call-docker-build
    uses: <org>/actions/.github/workflows/reusable-trivy.yml@main
    with:
      image: 'ghcr.io/<org>/wordsmith-web:${{ needs.call-docker-build.outputs.image-tag }}'

Starter Templates

Provide easy access to your org’s workflow templates

Once added, this UI will show up when creating a new workflow in the GitHub UI

It saves time needed to hunt down your actions repo to find the latest templates

Setting up a Starter Template

Be sure you did the Docker Build Reusable Workflow homework first

Workflow starter templates must go in <org>/.github/workflow-templates/

1. Create the public repo .github in your org
2. Copy your calling Docker workflow from wordsmith-web into this new repo at ./workflow-templates/call-docker-build.yml
3. Add call-docker-build.properties.json to the same folder with this content:

{
    "name": "Call Docker Build",
    "description": "Call the reusable workflow in the actions repo",
    "iconName": "octicon smiley",
    "categories": [
        "Dockerfile"
    ],
    "filePatterns": [
        "^Dockerfile"
    ]
}

Testing our starter template

We can assume the actual workflow will run correctly for this example

Let’s add it to wordsmith-api through the GitHub Actions UI

1. Go to the wordsmith-api repo, and click the "Actions" tab
2. Click the new workflow button and check for your custom Starter workflow
3. Click Configure and ensure the correct file is copied in

Examples of reusable workflows

Imagine all these being centrally controlled and automated for all software repos in your org

• The advanced Docker Build
• Image CVE Scanning with the Trivy Action
• Linting Action with Super-Linter (50+ linters in one Action)
• Complex testing designs (unit + integration + E2E in one Workflow)
• Smoke test any PR commit by deploying to a k8s cluster and waiting for probes to pass
• Static analysis (code quality, security, etc) with CodeQL
• Create PR preview environments with Uffizzi or Okteto. Built on PR open and destroyed on PR close
• PR bots that auto-assign reviewers, assign labels, etc
• Automatically rebase PRs when they get out of date from main
• Sync labels (names, colors, descriptions) across repos
• Create a OpenSSF scorecard of your repos
• Watch GitHub runners for bad behavior while your workflows run

Wrap-up

• That’s it for today
• Friday is in Zoom as catch up, extra credit, and Q&A
• If you’ve done all that: Try these GHAs as Extra Credit
• You could also try one on a real repo at work
• Take tech topic questions to the Maven class community
• For schedule, logistics, and feedback: help@bretfisher.com

Housekeeping

• You now have more Extra Credit Labs for Wednesday
• I created an article with all external reference links that I’m expanding and keeping current

Workshop 2 homework recap

• Did everyone get the Docker reusable workflow working in web and api?
• You’ll need those images built for next week
• I threw in a few minor issues like…
  • Finding the right uses: path for the reusable workflow
  • .yaml vs .yml in file paths
  • secrets: and with: were empty arrays and invalid YAML. Comment them out
• If you don’t see a workflow run in PR like you expect, check the Actions tab
• If it’s done correctly, github.com/orgs/<org-name>/packages shows two images
  • Both repositories should have an image with tags similar to stable-20230714-b3cc954
• Check HardlyDevOps for a working example

Week 1 recap

• You had pre-class videos on GHA Basics and CI Examples
• We added some basic GHAs, and then watched a video on Reusable Workflows
• We added a Docker build that turned into an advanced Reusable Workflow
• We had Extra Credit Labs from Monday or Wednesday

🧑‍💻 Prep for next week

• Next week we’re shifting to Argo CD on Kubernetes
• You need a Kubernetes cluster for Argo CD. I’ll be using Docker Desktop
• Any K8s is fine, even k3d (Kubernetes in Docker), minikube, or kind
• Let me know if you need help with setup or a K8s cluster to use (single node)
• Wednesday’s reusable Docker builds homework is REQUIRED for next week
  • (we need those latest and stable-<date>-<sha> images to deploy)
• You’ll get a video on GitOps and Argo CD before Monday to prep the basics

Potential discussion topics for today

• Any questions about GHA or content so far
• Implementing these learnings in your real projects

–

• Testing in Docker Build Stages vs. GitHub Actions
• Preview environments with GitHub Actions in PRs
• "Image Promotion", using multiple registries or repos for images

Wrap-up

• That’s it for today
• Monday is in Zoom on Argo CD setup
• Remember you’ll get a short GitOps video before Monday as prep
• Take tech topic questions to the Maven class community
• For schedule, logistics, and feedback: help@bretfisher.com

The problem with traditional Kubernetes deploys

• Previously we used kubectl apply or helm install
• This is a "push" model
• Even if we automate, traditional CI/CD is typically, "one and done"
• It has limits:
  • Status is hard to get. CI/CD will usually show "success" or "failure"
  • CI/CD-aware rollbacks are unlikely
  • Drift (out-of-band change after deploy) is hard to detect or nonexistent
  • CI/CD now has admin cluster access. To all clusters! 😱

Definition according to Weaveworks / OpenGitOps

Weaveworks coined the term GitOps in 2017, now documented in the OpenGitOps project

• Declarative
  • A system managed by GitOps must have its desired state expressed declaratively.
• Versioned and Immutable
  • Desired state is stored in a way that enforces immutability, versioning, and a complete version history.
• Pulled Automatically
  • Software agents automatically pull the desired state declarations from the source.
• Continuously Reconciled
  • Software agents continuously observe actual system state and attempt to apply the desired state.

What isn’t GitOps?

• Letting humans admin K8s clusters directly (kubectl)
  • (once you’ve fully adopted GitOps, daily kubectl use should stop)
• Using Git to store IaC, and letting GHA push to K8s
  • (IaC is a prerequisite for GitOps, but not enough on its own)
• Pushing changes to K8s clusters
  • (GitOps is a pull model from within the system that is getting updated)
• Any CI/CD system that doesn’t self-heal the cluster after deployment
  • (If you kubectl delete deploy/wordsmith-api in your cluster, GitOps should restore it)

PSA: No tool is GitOps

Just like no tool is DevOps

• GitOps is a set of principles, not a tool
• Argo CD is a CD tool that helps you implement GitOps principles
• Argo CD can be used imperatively 😒
• Argo CD can let you change things outside git, causing drift 😠
• Argo CD lets you do imperative things without history or rollback 😭
• It’s up to YOU to restrict access to the Argo CD UI and CLI 🚔

Why Argo CD?

• Applies GitOps principles to kubectl apply
• Provides a useful web UI for monitoring and (optionally) managing deployments
• Argo is a CNCF "Graduated" project
• The most popular GitOps tool for Kubernetes, and a top CNCF project
• That means other tools often have Argo CD support built-in
• Also means there’s lots of help and videos out there
• ArgoCon (videos) and GitOpsCon are great resources of content

More about Argo’s history

• Project started in 2017, grew within Intuit, and exploded in popularity
• Intuit open-sourced it in 2018. Today Intuit uses it for:
  • 14,000 apps in 3,300 git repos
  • 5k developers doing 1,000 deployments per day
  • On 350 clusters with 15,000 nodes
• The parent project, Argo, also has "Workflows", "Events", and "Rollouts"
• Akuity is a startup by a few of the Argo founders
• Akuity was on my show last year, and is a SaaS platform for Argo

Argo CD core concepts

Argo CD concept dictionary

Argo CD setup architecture

• A full Argo CD install has 7 pods (non HA)
• Simple to start with, but can get complex fast (TLS, secrets, user management, SSO)
• Luckily, if you follow GitOps principles, no backups needed (if breaks, just delete and redeploy)
• HA is possible, but not required for most use cases
• SSO is encouraged, unless only a few users

Argo CD internal architecture

A full Argo CD install has 7 pods (non HA)

Scaling Argo CD

• Argo CD in a cluster scales horizontally and vertically
• A default install should handle 100s of app stacks fine (1000s of containers)
• Certain values can be changed based on different scaling needs. Needs like:
  • Large number of clusters in a single Argo CD
  • Large number of applications deployed in a single Argo CD
  • Large number of commits on any App repo Argo CD is watching (esp. monorepos)
  • Large number of repositories to poll/webhook in a single Argo CD
  • Implementing Argo CD High-Availability (docs)
• Many questions answered by maintainers "Prod Best Practices" video

Argo CD architecture setup options

• Namespace scoped: One Argo CD per namespace
  • In enterprise: Dev’s manage their own Argo CD in their namespace. HA not required
• Workload clusters: One Argo CD per cluster
  • Most popular. Mgmt overhead grows with # of clusters. HA not required
• Control plane cluster: One Argo CD for multiple clusters
  • Inevitable without tight IaC control. Reduces overhead, but SPOF . HA highly encouraged
• Hybrid: Argo of Argo’s. Central Argo CD deploys Argo CD to each cluster
• Akuity SaaS: Argo of Argo’s with more automation and easier management

Argo CD GitOps repo design options

• Argo CD is flexible about how you organize your Git repos
• Assume YAML is in a different repo from app code (a 12-Factor principle) and many other reasons
• Repo design often/should reflect org structure (Conway’s Law)

Minimal: Shared ownership

• 1 ring repo to rule them all: Kustomize + Helm + Argo CD App + Argo CD Config

Scalable: Split app configs from Argo CD configs 👈 my favorite

• 1 Kustomize/Helm repo per solution: (Dev and DevOps share ownership)
• 1 Argo CD repo: Argo CD App + Argo CD Config (DevOps or Ops team owns)
• Optional: involves DevOps/Ops PR approval when first adding an app to Argo CD (since they own repo)
• ApplicationSet now enables Ops cluster control while Devs add/remove apps at will
• Day to day, Devs can PR to their own Kustomize/Helm repos w/o Ops involvement

Dev self-service

• By default, all Argo CD apps are managed within the same argocd namespace
• This requires App YAML authors (Devs) to have Argo CD write perms, basically cluster admin
• You can mitigate this with branch protection PR reviews by DevOps/Ops of app YAML (which is what I do)
• That can slow down Devs, which now need a DevOps PR approval to add new Apps
• What if Dev teams could manage their own apps in their own namespaces?

Repo design needs to be split further

• Kustomize/Helm repo per "solution" or "area of responsibility" (Dev and DevOps share ownership)
• Argo CD App repo (Dev teams own)
• Argo CD Config repo (DevOps/Ops team owns)

Self-service opt 1: "Applications in any namespace"

beta

• Tell Argo CD config which namespaces to allow self-service in (wildcards allowed)
• Give Argo CD cluster-admin permission
• Dev teams then use GHA to push kind: Application to their namespace
• (This seems anti-GitOps, but not quite. This isn’t the app itself)
• Then Argo CD sees that new resource, and pulls the app

Self-service design: ApplicationSet controller

ApplicationSet is fantastic, and my favorite way to manage apps

• Until now, you had to define a Application resource YAML for each app to deploy
• ApplicationSet tells Argo CD where to find app YAML elsewhere
• Deploys one YAML file to many multiple places (namespaces or clusters) at once
• Includes other use cases including self-service via Generators
• Can even generate apps based on PRs and other external sources
• The Generators lets Ops limit what App settings a Dev can change

–

1. Ops set "Argo CD Ops" repo to only allow Devs to change repoURL and path of ApplicationSet
2. Next, Devs PR to "Argo CD Ops" repo once, pointing to their Argo CD repo
3. Then, Devs can config their Argo CD App in their repo w/o Ops involvement

Quiz!

More questions on last week’s content

Popcorn Time!

• Raise your hand in Zoom reactions to share your answer
• After you speak, pick the next raised hand to speak

Q1: Some of you are already using Argo CD. What does your setup and infra look like for Argo CD today? cluster or multi-cluster? self-service or PR-gates? Do you give dev’s access to the Web UI? Do you lock them to read-only?

Next question:

Q2: For everyone: What design decisions do you think you’ll implement? Will you implement self-service or are PR-gates enough?

The Argo CD init problem: Cluster bootstrapping

• Argo CD can be complex to setup manually (TLS certs, admin password, user accounts, GIT_TOKEN, etc)
• The argocd CLI tool makes it easier, but it’s not declarative
• The (eventual) goal is to have a fully-declarative Argo CD setup
• Being able to recreate Argo CD with a simple kubectl apply is powerful! (no backups needed)

A few setup options

1. kubectl apply plus argocd CLI
   • Basically follow the Getting Started Docs
   • This is the one I’ll be using this week for demos
2. The argocd-autopilot project
   • A CLI that is fully GitOps automated
   • Uses App of Apps pattern on Argo CD itself
   • Seems to be in maintenance mode, and lacks features I would expect
3. Argo CD Operator
   • Helps installing, upgrading, backing up, restoring, more
   • Not declarative AFAIK (as most operators aren’t)
4. 😍 Hard mode: Config all the YAML yourself per docs "Declarative Setup"
   • The best way to build your production design
5. (and of course) SaaS option (Akuity, maybe others)

How we’re going to deploy it today

• This is your homework
• Choose an option from the last slide, #1 or #2 are recommended
• Install Argo CD on your personal K8s. Docker Desktop, Minikube, etc.
• Maybe do #1 just to have that experience, then do #2, which is a completely different CLI experience
• Goal: Have a working Argo CD, with UI accessible (self-signed cert is fine) by Wednesday’s class
• Doesn’t have to be fully declarative yet, or fancy. You’ll get there 😉

📚 GitOps further reading for the curious

My favorite links for understanding GitOps

• Weaveworks’ history of GitOps is a good read on its origins
• Their Guide to GitOps are the most comprehensive out there for the "why" and "how"
• Their GitOps FAQ is fantastic
• Awesome GitOps

There’s lots of confusion about GitOps

• For example, GitLab says you can GitOps in a push method, which is just automated IaC, not GitOps
• GitHub says we need to lock in a GitOps definition before it becomes a vague buzzword like DevOps

Do "GitOps tools" do GitOps for their own config?

Viktor Farcic wrote a good post, and then another on "GitOps tools" not following their own principles

📚 Workshop 3 Argo CD admin further reading

Tools

• Check out the lab projects for Argo

Docs

• Awesome Argo’s list of resources
• What’s next for Argo CD? Check the public roadmap
• Argo CD "Operator Manual" details lots of enterprise features like HA, pre/post deploy hooks, SSO, metrics, etc

Videos

• KubeCon EU 2023 Multi-Tenancy with Vcluster, Crossplane, and Argo CD
• Argo Proj channel on YouTube https://www.youtube.com/@argoproj7170
• Argo CD Prod Best Practices https://www.youtube.com/watch?v=ESQLqjbM8h0

Secrets in Argo CD

• Argo CD itself needs secrets
  • keys for app git repos
  • keys for its own config repo
  • Web UI TLS certs (If you don’t use K8s Certificates Manager)
• Read up on Argo CD’s options for Secrets Management
• Argo CD needs to deploy your apps secrets too (GHA Secrets aren’t an option)
• Tool options I would consider include:
  • Sealed Secrets in Git. Great for small teams with only a few clusters. Hard to scale across dozens of clusters
  • Argo CD Vault Plugin is a specific solution in Argo CD that avoids CRDs (simpler deployment) but only has "all or nothing" access for humans (Supports Vault,AWS,GCP,Azure,SOPS,1Password,Yandex,Keeper,IBM)
  • External Secrets Operator I learned about at GitOpsCon in At KubeCon 2022. Seems great for all.

More on External Secrets Operator

The goal is to add one

Docs on External Secrets Operator

Videos

• Short comparison of Secret solutions: How to Keep a Secret in GitOps, Without Keeping it in Git (15min)
• Manage Kubernetes Secrets With External Secrets Operator by my friend Viktor Farcic (12min)
• KubeCon EU 2023: Protecting Your Crown Jewels with External Secrets Operator (38min)

Wrap-up

• That’s it for today
• Homework: Install Argo CD on your own cluster by Wednesday
• Extra Credit for the over achievers
• Wednesday we’ll deploy our app multiple ways with Argo CD!
• Take tech topic questions to the Maven class community
• For schedule, logistics, and feedback: help@bretfisher.com

File formats for deploying your apps

AKA "Ways Argo CD can convert something into Kubernetes manifest YAML"

• 👉 Kustomize 👈 my fav for internally-built apps
• 👉 Helm 👈 my fav for externally-built apps (Official Helm Charts, Artifact Hub)
• Vanilla manifests (for when using external templating tools)
• Jsonnet (only used if you already use it)
• Extensions (I’ve never seen them used)

Key concept: How Argo CD knows what to deploy

• Argo CD won’t auto-update anything it’s not watching
• Argo CD gives K8s new CRDs so Argo CD can respond to three main things:
  • Where to watch for app changes (git, PRs, and registries)
  • Where to deploy those changes (K8s API endpoints and namespaces)
  • (optional) Grouping those settings and controlling permissions
• What are these CRDs?

The kind: Application resource

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: wordsmith-staging01
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: 'https://github.com/<org>/wordsmith-k8s.git'
    path: environments/staging01
    targetRevision: <branch>
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: wordsmith-staging01
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Deploy Wordsmith with that YAML

• Create the file wordsmith-staging01.yaml on your local machine (where kubectl is)
• Copy the last slide into it, changing <org> and <branch>

–

• Now we need to tell Argo CD about it with kubectl
• Run this command against your K8s:

kubectl apply -f wordsmith-staging01.yaml

We could also create it without YAML via argocd app create... or Web UI

Let’s check what we deployed

• If you did Monday’s homework and have Argo CD working, let’s check Web UI
• Let’s see which image we deployed for web Deployment

–

• It says bretfisher/wordsmith-web, that’s not right
• Let’s check our Kustomize YAML in environments/staging01/kustomization.yaml

–

Oh boy, we need to fix that override YAML to use our new stable images from last week

Edit our Kustomize YAML for staging01 env

• First let’s look up the image tags we created last week
• Go find them in either org-level > Packages, or repo-level > Packages
• Look for the stable-* tag for each image wordsmith-web and wordsmith-api
• Edit that kustomization.yaml file and let’s change each image name and tag

images:
  - name: bretfisher/wordsmith-api
    newName: ghcr.io/<org>/wordsmith-api
    newTag: stable-<your-unique-tag>
  - name: bretfisher/wordsmith-web
    newName: ghcr.io/<org>/wordsmith-web
    newTag: stable-<your-unique-tag>

When you commit this file to your default branch, Argo CD will auto-update the app

By the way, this images: array is a Kustomize "Build-ins" feature called ImageTagTransformer

3 minutes to sync repos? How can we sync faster?

• The autosync feature defaults to checking git repos every 3 minutes
• We can click the "Sync" button in Web UI to force a sync
• We can change this to 1 minute (be aware of rate limiting, 5k-15k per hour)
• We can implement a webhook to trigger a sync (GitHub.com needs Argo CD API access)
• We can use GHA to trigger a sync with Argo CD CLI (GHA Runner needs Argo CD API access)

We did it! Let’s add a 2nd environment… but…

• Problem: If we need to update the kind: Application YAML, we have to kubectl apply again
• Argo CD doesn’t monitor that Application definition because we applied it out-of-band (kubectl)
• The YAML told Argo CD to monitor something else downstream, not the Application YAML
• Same issue if we create a new YAML Application for a 2nd environment, we’d need to kubectl apply it
• We need a way to tell Argo CD to watch for new environments in our repo without needing yaml
• Enter the ApplicationSet resource

+ Mic Time!

• Raise your hand in Zoom reactions to share your answer

Which Argo CD setup design do you think you’ll use?

• One instance per cluster?
• One multi-cluster instance?
• Argo-of-Argos where a central Argo CD manages an Argo-per-cluster design (Akuity)?

Let ApplicationSet create our Application

ApplicationSet has many advantages over Application

• The new ApplicationSet resource’s job is to create many Application resources
• Eliminates the need to copy-paste Argo Application YAML for each app
• Enables self-service, where devs can just create Helm/Kustomize in other repos
• Can deploy the same app YAML to multiple clusters
• Handles monorepos and more than just git repos. 9 Application YAML Generators and counting…
• Generators search for Helm/Kustomize apps based on org’s/repo’s/dir’s/file’s/pr’s we define

Create ApplicationSet

• Create a new file in your wordsmith-k8s repo: applicationsets/wordsmith-k8s.yaml
• A ApplicationSet YAML will have two main parts, generators and the template
• The generator tells Argo where to find the apps Helm/Kustomize YAML
• The template is how Argo CD will create a kind: Application for each app it finds
• This next slide uses the Git Generator to create one Application per subdir
• Commit it to the default branch
• We’ll kubectl apply -f it to our cluster in a minute

What about when changing the ApplicationSet?

An idea for allowing Argo CD to monitor ApplicationSet changes

• We created the ApplicationSet resource out-of-band with kubectl
  • Sooo… same problem as before. Argo CD won’t see changes to it
• Aggg, will this nested YAML ever end?!
• We can make a single, simple, "root" Application resource that watches a dir
• As long as we put all ApplicationSet YAML in that dir, it’ll watch them all
• With great power… messing this up can break all your apps

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  name: applicationsets
  namespace: argocd
spec:
  destination:
    namespace: argocd
    server: https://kubernetes.default.svc
  project: default
  source:
    path: applicationsets
    repoURL: https://github.com/mostlydevops/wordsmith-k8s.git
  syncPolicy:
    automated:
      allowEmpty: true
      prune: true
      selfHeal: true
    syncOptions:
    - allowEmpty=true

ApplicationSet is the future of Argo CD

The more you learn about it, the more you’ll realize how flexible it is

• New feature releases for it are fast and furious
• Use SCM Provider Generator to find repos in your org to make apps from
• Use Pull Request Generator to find PRs in your repos to make apps from
• Use Matrix Generator to combine multiple generators into one
• (those are neat, but Git and List Generators are the most common)
• If cluster Ops team manages permissions and resources with AppProject, and point ApplicationSet to Dev’s repos, then devs can self-service their own apps w/o any Argo CD YAML

My WIP Examples of Application and ApplicationSet

WIP: Work in Progress

https://github.com/BretFisher/gitops-argocd

design-01 is a day-1 super-simple file structure

design-02 is more complex, yet offers way more flexibility

design-03 would be the most advanced and automated (WIP)

https://github.com/MostlyDevOps/wordsmith-k8s

that’s a working example of what we did today

We can skip humans with the Image Updater

I like this for dev or staging environments

• Rather than watching a git repo, it watches an OCI registry
• Watches a registry for new images (with various strategies)
  • Semver (1.2.3)
  • Latest (metadata) date with a reusable (mutable) tag (dev)
  • Last tag name, alphabetically (stable-YYYY-MM-DD-mm-ss)
• Auto patches your Argo CD apps with the new image and deploys
• Optionally, auto commits an override file to your App repo:
  • Argo CD knows to look for this file, called .argocd-source.yaml
  • It can also use a Kustomize mode that runs kustomize edit set image
• Only works with Kustomize and Helm apps
• Regex can be used in addition to the above strategies
• It can write to a new branch

Quiz!

Questions on Monday’s content

+ Mic Time!

• Raise your hand in Zoom reactions to share your answer

What Argo CD ApplicationSet Generators do you think you’ll use (or use now)?

• Git Generator? (like we did today, folders in a repo)
• PR Generator?
• SCM Provider Generator? (search GitHub org for repos matching a regex)
• Matrix Generator? (combine two of the above into one)
• (others)

Bridge the gap between CI and GitOps

• Last week, automation ended with a new image tag in a registry
• This week, app deployment starts with updating Helm/Kustomize to new image tag
• Let’s automate that gap! (I don’t know what to call it…)

A new reusable workflow

BretFisher/github-actions-templates/.github/workflows/reusable-gitops-pr.yaml

• It starts after app repo PRs are merged and images are pushed
• It runs in that apps repo, and updates the app’s Helm/Kustomize prod env with new tag
• Now all a app dev needs to do after they merge their app PR is to accept GitOps PR
• For staging clusters, we could use Argo CD Image Updater to auto-deploy new tags
• We just implemented GitOps and only added a single PR review to their workflow

Troubleshooting Argo CD

• ApplicationSet Controller only makes Application resources. Read about that

1. Problem: Argo CD isn’t seeing my Application or ApplicationSet YAML
   1. Make sure it’s in the argocd namespace: kubectl get applicationset -A
   2. See what Applications exist: kubectl get application -A
   3. Use describe to get details: kubectl describe -n argocd appset/wordsmith
   4. Ensure all three resources were added to K8s API: kubectl api-resources | grep argo
   5. Check logs of Argo CD pods, which are detailed kubectl get -n argocd pods
      1. argocd-server pod for server config
      2. argocd-application-controller for Application resources
      3. argocd-applicationset-controller for ApplicationSet resources
      4. repo-server for git repo access and polling
      5. argocd-dex-server for SSO auth
      6. argocd-notifications-controller for notification config and triggers
      7. argocd-redis for caching repo and resource data

Troubleshooting Argo CD

2. Problem: My YAML isn’t formatted correctly
   1. In VS Code, ensure you have the Kubernetes extension installed
   2. It’ll check syntax as you type, including Argo resources, with popup definitions
   3. ProTip: You can add it as a suggested extension in your K8s YAML repos
   4. Require GHA K8s linter to pass before PRs can merge. (Super-linter) also runs that
   5. Manually try kubectl apply --dry-run=server -f file.yaml on K8s with Argo CD
      1. ProTip: you could do this in GHA as part of a "smoke test"
         1. Setup K3d Action
         2. Install Argo CD with one-liner and maybe a config file, if needed
         3. Apply yamls from PR diff, or maybe only if specific YAML dirs changed

📚 Workshop 4 Argo CD app deploys further reading

Tools

• Argo CD Interlace: Add Providence to deploys (Supply Chain Security)
• Awesome Argo’s list of resources

Videos

• ArgoCon Scaling Argo CD with ApplicationSet (25m)
• GitOpsCon Application Sets at Adobe (36m)
  • Blog post: Overview of Argo CD Architectures in 2023

Docs

• Argo CDs "User Guide" is about features for app deployers
• Use pre/post hooks for advanced workflows (db migrations, webhooks to external systems)
• What’s next for Argo CD? Check the public roadmap

Wrap-up

• That’s it for today
• Homework: catch up with today’s labs
• More Extra Credit coming tomorrow
• Take tech topic questions to the Maven class community
• For schedule, logistics, and feedback: help@bretfisher.com

Housekeeping

• Graduation Day! You’ll get a certificate (tomorrow?) from Maven
• Download slides (will also be in email later)
• BONUS: Schedule a 1-on-1 with Bret in two weeks to discuss your progress
  • What implementation questions do you have weeks later?
• Course survey: It’s important to us. Let’s take 5 minutes to fill it out now
• You’ll get emails on all this later

Potential discussion topics for today

• Any questions about Argo CD, ApplictionSet, or GitOps?
• Implementing these learnings in your real projects

–

• Bringing GitHub Actions and Argo CD together with automated YAML PRs
• Preview environments with GitHub Actions or Argo CD in PRs

Wrap-up

• It’s been an honor to help you!
• You’ll get a few more emails to ensure you’ve got everything you need
• Maven chat still works! We’ll respond to any questions there or at help@bretfisher.com
• Have a great weekend, and good luck in your implementation!