CS Fundamentals Trainer · Guides

Git internals and CI/CD pipelines

Git as a content-addressed object store (hashes, packfiles, bisect, rebase), branching strategies, continuous integration and delivery, and how canary, blue/green and rolling releases limit the damage of a bad release.

37 min read

Git looks like a set of commands to memorise, and it becomes much easier once you know that it is a small content-addressed database with a handful of object types, and that branches are just names for objects. CI/CD (continuous integration and continuous delivery) is then the discipline of putting every change through the same automated path, from a commit to running in production, in small steps that can be undone. This guide opens up the repository with the real commands and measures what it stores, and then covers pipelines, branching strategies and the deployment strategies that limit the damage of a bad release. Everything in the Git half was run on a real repository; the script is in the appendix.

It fits with containers and Kubernetes (what the pipeline deploys), observability (how you know a release is healthy) and security (secrets in history, supply chain).

1. Git is a content-addressed object store

Git stores four kinds of objects, each named by the hash of its content (SHA-1 by default; SHA-256 in repositories created with --object-format=sha256, see below) (with a short header):

ObjectHoldsAnalogy
blobthe bytes of one file (no name, no permissions)file content
treea list of names, modes and the ids of blobs and other treesa directory
committhe id of a tree, the ids of its parent commits, author, committer, time and messagea snapshot with history
taga named, optionally signed pointer to an objecta label

I checked the naming rule by hand. For a file containing hello and a newline, git hash-object printed ce013625030ba8dba906f756967f9e9ca394464a, and hashing the string blob 6, a zero byte and hello with plain sha1sum printed the same id. The name of the object is its content, which has three consequences:

Diagram: A commit points to a tree, and the tree to blobs; the branch is only a name

Both trees point to one blob, because the file did not change.

Branches, HEAD and the index

A branch is a file under .git/refs/heads/ containing one commit id: in the experiment it was exactly 41 bytes, 40 hexadecimal characters and a newline. Creating a branch writes 41 bytes, which is why branching is cheap in Git; and committing moves the current branch's pointer to the new commit. HEAD says which branch (or commit) you are on. The index (staging area) is the proposed next tree: git add writes blobs and updates the index, and git commit turns the index into a tree and a commit.

Commits are snapshots, not diffs. Differences are computed when needed (and a commit shown by git show is a difference between two snapshots).

Packfiles: snapshots stored as deltas

If every version were a full copy, history would be huge. New objects are first stored "loose", one compressed file each; git gc (or a push) bundles them into a packfile, where similar objects are stored as deltas against each other. I created a 5,000-line file (108 KB) and committed 200 small edits to it:

StorageSize
201 full copies, uncompressedabout 21,873 KB
after git gc --aggressive: one pack with 608 objectsabout 102 KB

A reduction of more than 200 times, because each edit changed one line. This is why a repository with long history is smaller than you expect, and why large binary files are the exception: a re-encoded video or image changes almost every byte, so deltas do not help, and every version is kept forever. Use Git LFS or an artifact store for them.

The same fact is behind the biggest Git mistake: a file committed once is in the history forever. Deleting it in a later commit removes it from the tree, not from the old commits. A leaked key must be revoked first (it is compromised the moment it is pushed); rewriting history to remove it (git filter-repo) is cleanup, not a fix. See the security guide.

SHA-1 and SHA-256

Git was designed with SHA-1 for identity, not security. After practical SHA-1 collisions were demonstrated in 2017, Git added collision detection, and it supports SHA-256 repositories as a transition. The important point for design is that content addressing, as in event logs and container image digests, gives integrity and deduplication together.

SHA-256 repositories in practice

Everything above says "SHA-1", but a Git repository can use SHA-256 instead: git init --object-format=sha256 (Git 2.29 and later; the Git project no longer calls it experimental since 2.42). The object model is identical; only the hash function, and therefore every id, changes. I created the same one-file, one-commit repository in both formats (Git 2.43, with fixed dates so that the only difference is the format):

SHA-1 repositorySHA-256 repository
id of the hello blobce013625030ba8dba906f756967f9e9ca394464a (40 hex characters)2cf8d83d9ee29543b34a87727421fdecb7e3f3a183d337639025de576db9ebb4 (64)
id of the first commitbc31b899…58f1538c… (a different id for the same content)
git log --onelinebc31b89 first58f1538 first (abbreviations are still short)
repository configcore.repositoryformatversion = 0, no extensioncore.repositoryformatversion = 1, extensions.objectformat = sha256
naming rule by handsha1("blob 6" + NUL + "hello\n")sha256("blob 6" + NUL + "hello\n"), which I computed with sha256sum and got the id above

What follows from it:

2. Merge, rebase and rewriting history

Two branches that have diverged can be combined in two ways:

Diagram: Merge keeps both lines of history; rebase replays one line on top of the other

The rule that follows from immutability: never rewrite commits that others have already based work on (rebase or amend something pushed to a shared branch). Their copies still have the old ids, and the two histories now diverge, which is how a team ends up with duplicate commits and painful merges. Rewrite your own unpublished work freely; when you must update a branch that you share, say so, and use --force-with-lease instead of --force so that you do not overwrite a colleague's push.

A squash merge turns a branch into one commit on the main line. It gives a clean history and a one-to-one link between pull request and commit, at the cost of losing the individual steps (which matters for bisect inside a big change). Choose one policy per repository and enforce it.

Recovering: the reflog

Branch pointers move, and old commits remain in the object store until garbage collection (by default, unreachable objects are kept for weeks). The reflog records where HEAD and each branch pointed over time, so after a bad reset or rebase, git reflog shows the old id and git reset --hard <id> or git branch rescue <id> brings it back. Most "lost" work is not lost; work that was never committed is.

3. Finding the commit that broke something: bisect

git bisect is binary search over history (see complexity). You mark one good and one bad commit; Git checks out the middle, you (or a script) say good or bad, and it halves the range. I made 100 commits in which a test starts failing at c77, and ran git bisect run with a script that exits 0 for good and non-zero for bad:

Commits in the rangeTest runs neededResult
1007c77 named as the first bad commit

log2(100) is about 6.6, so seven runs; for 10,000 commits it is 14 runs. This works best with small, buildable commits (so every commit can be tested) and a fast, deterministic test, which is one more argument for the trunk-based practices below.

4. Branching strategies

StrategyHow it worksFits
Trunk-based developmenteveryone integrates into one main branch at least daily; branches live for hours, incomplete work is hidden behind feature flagsteams that deploy continuously; the practice the DORA research links to high performance
GitHub flowshort-lived feature branches, a pull request with review and CI, merge to main, deploy from mainmost web teams
Gitflowlong-lived develop and release branches plus feature, hotfix branchessoftware shipped in versions that must be maintained in parallel (installed products, libraries)
Release branchesbranch at release, backport fixesthe same, more lightly

The reason short branches win is merge cost: the cost of combining two lines of work grows faster than their size, since conflicts grow with the number of changes and with how long each side has forgotten the other's context. Long-lived branches postpone that cost and make it bigger. Feature flags let you merge unfinished code safely and then turn a feature on for 1% of users, which separates deploying code from releasing a feature.

5. Continuous integration and delivery

Diagram: A typical pipeline: fast checks first, one artifact promoted through environments

Principles that make a pipeline trustworthy:

A concrete pipeline: provenance, pinned dependencies, secret scanning, migrations and rollback

The principles above become checks. Here is one pipeline, with the failure each stage prevents and a runnable demonstration of the parts that can be shown without a CI server (the scripts are in the appendix). The pipeline definition at the end is a sketch of how it fits together; the CI products and scanners it names are not demonstrated here.

Diagram: A pipeline with supply-chain and rollback controls

1. Dependency pinning. A version range means the same commit builds different code on different days. In the demonstration lib>=2.0 installs 2.31.0 in January and 3.0.0 in June. A lock file with hashes (pip install --require-hashes, package-lock.json, go.sum) fixes both the version and the bytes: a download whose sha-256 differs from the lock is refused (REFUSED lib==2.31.0: hash 42c962426d1f… does not match the lock file's fec2012abbd8…). Commit the lock file, update it deliberately (a bot opens the pull request, the pipeline tests it), and run an audit of the locked set on every build (pip-audit, npm audit, OSV-Scanner; the scan examples are in the security guide). Pin third-party CI actions to a full commit id, not to a moving tag.

2. Build once, reproducibly, and record where it came from. Building the same source twice with default settings gave two different archives (75978fa7e9035769… and c9b605c62ba3ef43…), because of embedded timestamps and file order. Normalising them (sorted names, fixed modification time and owner) gave the identical ea511b3da0f25700… both times. A reproducible build lets anyone check an artifact by rebuilding it, and makes the digest a stable identity. Then the build job signs a provenance statement: the artifact's digest, the repository and commit, the branch, the builder and the inputs. The deploy step verifies it before it runs anything. In the demonstration (Ed25519 signatures; a trusted key held by the deploy step):

CaseDeploy step's verdict
genuine build from mainACCEPT: built by ci-runner-1 from 9f1c2ab0 on main
artifact swapped after the buildREJECT: the artifact is not the one that was built
statement edited to claim another commitREJECT: signature does not verify
signed with a key the deploy step does not trustREJECT: signature does not verify
genuine build, but from a feature branchREJECT: not a protected branch

In real pipelines you do not write this yourself: use the platform's artifact attestations or Sigstore/cosign, in the SLSA provenance format, and verify with the same tools in the deploy step or in a Kubernetes admission policy. The point is the policy: only artifacts built by our pipeline from a protected branch run in production. Tag container images by digest (image@sha256:…), not by a mutable tag.

3. Secret scanning, before and after the fact. I built a repository where a .env file with an AWS example key was committed and then deleted in the next commit. A scan of the files now in the working tree found 0 problems. A scan of every commit's added lines found the access key id and the secret key in the commit "add config". git log --all -S AKIAIOSFODNN7EXAMPLE lists both commits, and git show of the old one prints the file: deleting a secret does not remove it. A pre-commit check on the staged changes blocked a commit that added TOKEN=ghp_…. So: scan staged changes locally (pre-commit hook), scan the pull request and the full history in CI (gitleaks, TruffleHog), and turn on the hosting platform's push protection where it exists. When something leaks, the order is: revoke and rotate first, then clean history if you must, then find out whether it was used. Tune the scanner (entropy thresholds, allowlists for test fixtures, an expiry on every suppression): in the demonstration it did not flag a Git commit id, a UUID or a placeholder password in a notes file, which is what keeps people from ignoring it.

4. Database migrations and rollback. During a rolling deploy, the old and the new application versions run against the same database, and a rollback runs the old version again. A schema change must therefore work for both. I tested renaming a column (name to full_name) with SQLite and two application versions:

Schema after the stepOld app (v1)New app (v2, tolerates both columns)
start (name)okFAIL (the new column is not there yet)
naive: rename in one stepFAILok
expand: add full_nameokok
backfill: copy name into full_nameokok
contract: drop nameFAILok
contract without a backfillFAILFAIL (old rows have no value)

The naive rename breaks every pod that is still running v1, and breaks the rollback. The safe sequence is expand (additive, runs before the deploy), deploy v2 (reads either column, writes both), backfill (in small batches), and, only after v1 is gone and the rollback window has closed, a later release that contracts. Rules that go with it:

A sketch of the pipeline as configuration (GitHub Actions syntax; not run here):

name: release
on: { push: { branches: [main] } }
permissions: { contents: read, id-token: write, attestations: write }       # least privilege; the signing identity comes from the platform
jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<full-commit-sha>                            # pinned to a commit, not a tag
      - run: gitleaks detect --redact                                        # secrets in the history
      - run: pip install --require-hashes -r requirements.lock && pip-audit -r requirements.lock
      - run: python -m unittest discover                                     # unit and security regression tests
  build:
    needs: verify
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<full-commit-sha>
      - run: docker build -t app:${{ github.sha }} . && docker push registry/app:${{ github.sha }}
      - uses: actions/attest-build-provenance@<full-commit-sha>              # signed provenance for the image digest
  deploy:
    needs: build
    environment: production                                                  # required reviewers, protected secrets
    steps:
      - run: ./migrate.sh expand                                             # additive schema changes only
      - run: cosign verify-attestation … && ./deploy.sh --digest "$DIGEST" --canary 10%
      - run: ./wait_for_health.sh --error-rate-below 1% || ./deploy.sh --rollback   # automatic rollback to the previous digest

6. Deployment strategies: limiting the blast radius

Every release is a risk, and what you control is how many users meet a problem before you notice it.

StrategyHowRollbackCost and catch
Recreatestop the old version, start the newredeploydowntime
Rollingreplace instances gradually (see Kubernetes)roll forward or back, slowlyboth versions run together, so they must be compatible
Blue/greenrun a full second environment, test it, then switch the routerswitch back (seconds)double the capacity during the release; the database is shared, so schema changes still need care
Canarysend a small share of real traffic to the new version, compare its metrics, then increasesend the share back to 0needs good metrics and an automated gate
Feature flag / dark launchship the code turned off, enable per user groupturn the flag offflag debt: remove old flags
Shadow (mirror)copy production traffic to the new version and discard its responsesnot needed (users never see it)doubles load; side effects must be avoided

A canary release is a staged rollout with a gate between stages. Step through one with stages of 5%, 25%, 50% and 100%, a limit of 1% errors, and then turn on a version with a bug:

Interactive animation: canary release

In the model, a healthy version passes four stages. The version that fails 8% of requests is stopped at the first stage: it affected 4 of every 1,000 requests in that window (4 errors in the 50 requests it received) instead of 80 of every 1,000 had everybody been switched at once, a 20-fold reduction, which is exactly the share it was allowed to take (5%). A canary only protects you if:

Whatever the strategy, the old and new versions run side by side for a while, so API and database changes must be backward compatible for at least one version (REST versioning, microservices).

7. Infrastructure as code and GitOps

Environments built by hand drift apart and cannot be reproduced. Infrastructure as code describes servers, networks and clusters in files that are reviewed, versioned and applied by tools. GitOps applies the same idea to deployment: the Git repository is the desired state of the environment, and an agent in the cluster continuously compares reality with the repository and fixes the difference, the same reconcile loop as Kubernetes controllers. A deployment is a commit, a rollback is a revert, and the audit log is the Git history. Keep secrets out of the repository (use a secrets manager or encrypted values).

8. Common mistakes

9. Things to mention in an interview

Quick recap

Appendix: the experiment code

One shell script that needs only git and Python 3. It creates a fresh repository in a new directory, ignores your global Git configuration, and prints each result. The object ids and timestamps differ on every run (they depend on the time), but the structure, the hash of the hello blob (which is always ce013625030ba8dba906f756967f9e9ca394464a), the sizes and the bisect step count are repeatable. The pack size comes from git gc --aggressive on one machine and Git version and will differ slightly on others. The canary figures are computed by the animation's model with expected values (no randomness), not measured on a real service.

git.sh: objects, packs, bisect and rebase

#!/bin/bash
# Git internals experiments. Needs only git and python3.
set -eu
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null
export GIT_AUTHOR_NAME=a GIT_AUTHOR_EMAIL=a@x GIT_COMMITTER_NAME=a GIT_COMMITTER_EMAIL=a@x
rm -rf r && git init -q -b main r && cd r
echo "--- 1. a blob's name is the SHA-1 of 'blob <size>\\0<content>'"
printf 'hello\n' > a.txt
echo "git hash-object: $(git hash-object a.txt)"
echo "by hand (sha1):  $(printf 'blob 6\0hello\n' | sha1sum | cut -d' ' -f1)"
git add a.txt; git commit -qm one
echo "--- 2. the objects: commit -> tree -> blob"
git cat-file -p HEAD; git cat-file -p 'HEAD^{tree}'
echo "--- 3. the same content stored twice is one object"
cp a.txt b.txt; git add b.txt; git commit -qm two
git ls-files -s
echo "--- 4. a branch is a file holding one commit id (40 hex characters and a newline)"
cat .git/refs/heads/main; wc -c < .git/refs/heads/main
echo "--- 5. delta compression: a 5000-line file edited 200 times"
python3 - <<'PY'
import random
r=random.Random(1); open('big.txt','w').write(''.join(f"line {i} {r.random():.10f}\n" for i in range(5000)))
PY
git add big.txt; git commit -qm big
for i in $(seq 1 200); do sed -i "$((i*20))s/.*/edited $i/" big.txt; git commit -qam "edit $i"; done
size=$(wc -c < big.txt)
echo "file: $((size/1024)) KB; 201 full copies would be $((201*size/1024)) KB"
git gc -q --aggressive
git count-objects -vH | grep -E 'size-pack|in-pack'
echo "--- 6. bisect: 100 commits, the test starts failing at c77"
git checkout -q -b bisect-demo
for i in $(seq 1 100); do echo "$i" > counter.txt; [ "$i" -ge 77 ] && echo bug > broken.txt; git add -A; git commit -qm "c$i"; done
cat > test.sh <<'T'
#!/bin/sh
[ ! -f broken.txt ]
T
chmod +x test.sh; cp test.sh /tmp/gt.sh
first=$(git rev-list --max-parents=0 HEAD | tail -1)
git bisect start HEAD main >/dev/null
git bisect run /tmp/gt.sh | grep -E "is the first bad commit|running" | wc -l
git bisect log | grep -c '^git bisect \(good\|bad\)'
git bisect log | grep "first bad" || git show -s --format='first bad commit: %s' refs/bisect/bad
git bisect reset >/dev/null 2>&1
echo "--- 7. merge keeps commit ids; rebase rewrites them"
git checkout -q main; git checkout -q -b feature; echo f > f.txt; git add f.txt; git commit -qm feature
git checkout -q main; echo m > m.txt; git add m.txt; git commit -qm mainline
git checkout -q feature; echo "feature commit before rebase: $(git rev-parse --short HEAD)"
git rebase -q main; echo "feature commit after rebase:  $(git rev-parse --short HEAD)   (same change, new parent, new id)"

SHA-1 and SHA-256 repositories side by side

A shell script that needs git 2.29 or newer, bc and Python 3. It creates throwaway repositories and removes them.

#!/bin/bash
# SHA-1 and SHA-256 repositories side by side: identifiers, size, speed, and what does not interoperate. Needs git 2.29 or newer.
set -e
W=$(mktemp -d); cd "$W"
export GIT_AUTHOR_DATE='2026-01-01T00:00:00Z' GIT_COMMITTER_DATE='2026-01-01T00:00:00Z' GIT_AUTHOR_NAME=t GIT_AUTHOR_EMAIL=t@t GIT_COMMITTER_NAME=t GIT_COMMITTER_EMAIL=t@t
for f in sha1 sha256; do
  git init -q --object-format=$f r_$f; cd r_$f
  printf 'hello\n' > a.txt; git add a.txt; git commit -q -m first
  echo "== $f: blob $(git hash-object a.txt)"
  echo "   commit $(git rev-parse HEAD) ($(git rev-parse HEAD | tr -d '\n' | wc -c) hex characters), git log --oneline shows: $(git log --oneline)"
  echo "   config: repositoryformatversion=$(git config core.repositoryformatversion) extensions.objectformat=$(git config extensions.objectformat || echo '(not set: SHA-1)')"
  cd ..
done
echo "== the naming rule by hand: sha256 of 'blob 6<NUL>hello<LF>' = $(printf 'blob 6\0hello\n' | sha256sum | cut -d' ' -f1)"
echo "== the same content gets a different commit id: the format is part of the identity, so ids from one repository mean nothing in the other"
echo "== fetch across formats:"; (cd r_sha256 && git fetch ../r_sha1 2>&1 | head -1) || true
# a history of 300 commits touching 20 files each, in both formats
for f in sha1 sha256; do
  git init -q --object-format=$f big_$f; cd big_$f
  python3 - <<'P'
import random, subprocess, os
r = random.Random(1)
for c in range(300):
    for k in range(20):
        open(f"f{(c*7+k) % 400}.txt", "w").write("".join(r.choice("abcdefgh \n") for _ in range(4000)))
    subprocess.run(["git", "add", "-A"], check=True); subprocess.run(["git", "commit", "-q", "-m", f"c{c}"], check=True)
P
  T0=$(date +%s.%N); git gc -q --aggressive; echo "   $f: git gc --aggressive took $(echo "$(date +%s.%N) - $T0" | bc) s"
  echo "   $f: objects $(git count-objects -v | awk '/in-pack/ {print $2}'), pack size $(du -sk .git/objects/pack | cut -f1) KB, total .git $(du -sk .git | cut -f1) KB"
  T0=$(date +%s.%N); git log --stat >/dev/null; echo "   $f: git log --stat over 300 commits took $(echo "$(date +%s.%N) - $T0" | bc) s"
  cd ..
done
rm -rf "$W"

Pinned dependencies, a reproducible build and signed provenance

Python 3 with the cryptography package for Ed25519 signatures. Local data stands in for package downloads and a build runner.

"""Supply-chain basics for a pipeline: pinned and hash-checked dependencies, a reproducible build, and signed provenance that the deploy step verifies.
Standard library plus `cryptography` (Ed25519). Local files stand in for downloads and for a build runner."""
import gzip, hashlib, io, json, os, tarfile, time
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
from cryptography.hazmat.primitives import serialization
from cryptography.exceptions import InvalidSignature

sha = lambda b: hashlib.sha256(b).hexdigest()

# ------------------------------------------------------------------ 1. pinning with hashes
def pin_demo():
    print("1. Dependencies: a range changes under you, a pin with a hash cannot")
    index = {"2026-01-01": ["2.30.0", "2.31.0"], "2026-06-01": ["2.30.0", "2.31.0", "2.32.0", "3.0.0"]}   # versions available on two dates
    pick = lambda day, rule: max((v for v in index[day] if rule(v)), key=lambda v: tuple(map(int, v.split("."))))
    rule = lambda v: tuple(map(int, v.split("."))) >= (2, 0, 0)
    print(f"   'lib>=2.0' installs {pick('2026-01-01', rule)} in January and {pick('2026-06-01', rule)} in June: the same commit builds with different code")
    wheel = b"PK-not-really-a-wheel: lib 2.31.0 contents"                          # the file that was reviewed and locked
    lock = {"lib": ("2.31.0", sha(wheel))}
    def install(name, content):
        ver, want = lock[name]; got = sha(content)
        return f"installed {name}=={ver}" if got == want else f"REFUSED {name}=={ver}: hash {got[:12]}… does not match the lock file's {want[:12]}…"
    print("   lock file, genuine download:    ", install("lib", wheel))
    print("   lock file, tampered download:   ", install("lib", wheel + b" + a backdoor"))

# ------------------------------------------------------------------ 2. a reproducible build
def build_archive(files, deterministic, mtime_hint):
    raw = io.BytesIO()
    with tarfile.open(fileobj=raw, mode="w") as tar:
        for name in (sorted(files) if deterministic else list(files)):
            data = files[name]; info = tarfile.TarInfo(name); info.size = len(data)
            info.mtime = 0 if deterministic else mtime_hint; info.uid = info.gid = 0 if deterministic else 1000; info.mode = 0o644
            tar.addfile(info, io.BytesIO(data))
    out = io.BytesIO()
    with gzip.GzipFile(fileobj=out, mode="wb", mtime=0 if deterministic else int(mtime_hint)) as g: g.write(raw.getvalue())
    return out.getvalue()

def repro_demo():
    print("2. A build you can check: the same source twice")
    files = {"app/main.py": b"print('hello')\n", "app/util.py": b"X = 1\n", "VERSION": b"1.4.2\n"}
    a = build_archive(files, False, time.time()); time.sleep(1.1); b = build_archive(files, False, time.time())
    print(f"   default build, run twice:        {sha(a)[:16]}…  {sha(b)[:16]}…  identical: {a == b}")
    a = build_archive(files, True, 0); b = build_archive(dict(reversed(list(files.items()))), True, 0)
    print(f"   timestamps and order normalised: {sha(a)[:16]}…  {sha(b)[:16]}…  identical: {a == b}")
    return a

# ------------------------------------------------------------------ 3. signed provenance, verified before deploy
def provenance_demo(artifact):
    print("3. Provenance: who built these bytes, from which commit, and does the deploy step check?")
    ci_key = Ed25519PrivateKey.generate(); ci_pub = ci_key.public_key()                 # in real life the key never leaves the build system (or is keyless, as in Sigstore)
    other_key = Ed25519PrivateKey.generate()
    def attest(art, commit, branch, builder="ci-runner-1", key=ci_key):
        stmt = json.dumps({"subject": {"name": "app.tar.gz", "sha256": sha(art)}, "source": {"repo": "example/app", "commit": commit, "branch": branch},
                           "builder": builder, "materials": {"requirements.lock": "sha256:…"}}, sort_keys=True).encode()
        return stmt, key.sign(stmt)
    ALLOWED_BUILDERS, PROTECTED = {"ci-runner-1", "ci-runner-2"}, {"main"}
    def verify(art, stmt, sig):
        try: ci_pub.verify(sig, stmt)
        except InvalidSignature: return "REJECT: signature does not verify"
        p = json.loads(stmt)
        if p["subject"]["sha256"] != sha(art): return "REJECT: the artifact is not the one that was built"
        if p["builder"] not in ALLOWED_BUILDERS: return f"REJECT: builder {p['builder']} not allowed"
        if p["source"]["branch"] not in PROTECTED: return f"REJECT: built from branch {p['source']['branch']}, not a protected branch"
        return f"ACCEPT: built by {p['builder']} from {p['source']['commit'][:8]} on {p['source']['branch']}"
    s, sig = attest(artifact, "9f1c2ab07d", "main")
    print("   genuine build from main:                     ", verify(artifact, s, sig))
    print("   artifact swapped after the build:            ", verify(artifact + b"x", s, sig))
    print("   statement edited to claim another commit:    ", verify(artifact, s.replace(b"9f1c2ab07d", b"deadbeef00"), sig))
    s2, sig2 = attest(artifact, "9f1c2ab07d", "main", key=other_key)
    print("   signed with a key the deploy step does not trust:", verify(artifact, s2, sig2))
    s3, sig3 = attest(artifact, "77aa00bb11", "feature/x")
    print("   genuine build, but from a feature branch:    ", verify(artifact, s3, sig3))

if __name__ == "__main__":
    pin_demo(); art = repro_demo(); provenance_demo(art)

A minimal secret scanner over a throwaway repository

Python 3 and git. The 'secrets' are the AWS documentation's example values or made up. Real scanners (gitleaks, TruffleHog) have far larger rule sets.

"""A minimal secret scanner run over a throwaway git repository, to show what scanning the current files misses and what the history keeps.
All 'secrets' are the well-known example values from the AWS documentation or made up; the real tools (gitleaks, TruffleHog, platform push protection) have far larger rule sets."""
import math, os, re, shutil, subprocess, tempfile

RULES = {"AWS access key id": re.compile(r"\bAKIA[0-9A-Z]{16}\b"), "private key": re.compile(r"-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----"),
         "generic secret assignment": re.compile(r"(?i)(?:secret|token|password|api[_-]?key)\w*\s*[:=]\s*['\"]?([A-Za-z0-9/+_\-]{16,})")}
def entropy(s): return -sum(s.count(c) / len(s) * math.log2(s.count(c) / len(s)) for c in set(s))

def scan_text(text, min_entropy=3.5):
    hits = []
    for line in text.splitlines():
        for name, rx in RULES.items():
            m = rx.search(line)
            if m and (name != "generic secret assignment" or entropy(m.group(1)) >= min_entropy): hits.append((name, line.strip()[:70]))
    return hits

def git(*a, cwd): return subprocess.run(["git", *a], cwd=cwd, capture_output=True, text=True, check=True).stdout

def main():
    d = tempfile.mkdtemp(); g = lambda *a: git(*a, cwd=d)
    g("init", "-q"); g("config", "user.email", "t@t"); g("config", "user.name", "t")
    open(f"{d}/app.py", "w").write("print('hello')\n"); g("add", "."); g("commit", "-qm", "first")
    open(f"{d}/.env", "w").write("DB_HOST=db\nAWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE\nAWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\n")
    g("add", "-A"); g("commit", "-qm", "add config")                                     # the mistake
    os.remove(f"{d}/.env"); g("add", "-A"); g("commit", "-qm", "remove .env (oops)")      # the 'fix'
    open(f"{d}/notes.md", "w").write("build id: 3f786850e387550fdab836ed7e6dc881de23001b\napi_key = changeme-changeme-changeme\nrequest id 123e4567-e89b-12d3-a456-426614174000\n")
    g("add", "-A"); g("commit", "-qm", "notes")
    print("1. scanning only the files in the working tree now:")
    found = [h for f in os.listdir(d) if os.path.isfile(f"{d}/{f}") for h in scan_text(open(f"{d}/{f}").read())]
    print(f"   {len(found)} findings" + "".join(f"\n      {n}: {l}" for n, l in found))
    print("2. scanning every commit's added lines (what a history scan does):")
    seen = set()
    for c in g("rev-list", "--reverse", "HEAD").split():
        for n, l in scan_text("\n".join(x[1:] for x in g("show", "-U0", "--format=", c).splitlines() if x.startswith("+") and not x.startswith("+++"))):
            if (n, l) not in seen: seen.add((n, l)); print(f"   commit {c[:8]} ({g('log', '-1', '--format=%s', c).strip()}): {n}: {l}")
    print("3. the file is gone from HEAD, but still in the repository:")
    print("   git log --all --oneline -S AKIAIOSFODNN7EXAMPLE ->", g("log", "--all", "--oneline", "-S", "AKIAIOSFODNN7EXAMPLE").strip().replace("\n", " | "))
    first = g("log", "--all", "--format=%h", "-S", "AKIAIOSFODNN7EXAMPLE").split()[-1]
    print(f"   git show {first}:.env contains the key: {'AKIAIOSFODNN7EXAMPLE' in g('show', f'{first}:.env')}")
    print("4. a pre-commit check on the staged changes:")
    open(f"{d}/deploy.sh", "w").write("export TOKEN=ghp_16C7e42F292c6912E7710c838347Ae178B4a\n"); g("add", "."); staged = g("diff", "--cached", "-U0")
    hits = scan_text("\n".join(x[1:] for x in staged.splitlines() if x.startswith("+") and not x.startswith("+++")))
    print("   staged diff:", f"BLOCK the commit: {hits[0][0]} in {hits[0][1]!r}" if hits else "clean")
    shutil.rmtree(d)

if __name__ == "__main__": main()

Renaming a column across a rolling deploy

Python 3 with sqlite3.

"""Database migrations and rollback: which versions of the application work with which schema step. SQLite; the 'application' versions are two small classes.
Goal: rename a column (name -> full_name) while old and new code run together during a rolling deploy, and keep rollback possible."""
import sqlite3

class V1:                                   # the version in production: knows only `name`
    label = "v1 (reads/writes name)"
    def write(self, db, i, n): db.execute("insert into users(id, name) values (?, ?)", (i, n))
    def read(self, db, i): return db.execute("select name from users where id=?", (i,)).fetchone()[0]
class V2:                                   # the new version: writes both columns when it can, reads full_name with a fallback to name
    label = "v2 (reads/writes full_name, tolerates both)"
    def write(self, db, i, n):
        cols = {r[1] for r in db.execute("pragma table_info(users)")}
        if "name" in cols and "full_name" in cols: db.execute("insert into users(id, name, full_name) values (?, ?, ?)", (i, n, n))
        elif "full_name" in cols: db.execute("insert into users(id, full_name) values (?, ?)", (i, n))
        else: db.execute("insert into users(id, full_name) values (?, ?)", (i, n))        # the column is not there yet: this will fail
    def read(self, db, i):
        cols = {r[1] for r in db.execute("pragma table_info(users)")}
        sel = "coalesce(full_name, name)" if {"name", "full_name"} <= cols else "full_name"
        return db.execute(f"select {sel} from users where id=?", (i,)).fetchone()[0]

STEPS = {
    "S0 start: column name":                        lambda db: None,
    "S1 NAIVE: rename name -> full_name":          lambda db: db.execute("alter table users rename column name to full_name"),
    "S2 expand: add full_name (nullable)":         lambda db: db.execute("alter table users add column full_name text"),
    "S3 backfill: copy name into full_name":       lambda db: db.execute("update users set full_name = name where full_name is null"),
    "S4 contract: drop name":                      lambda db: db.execute("alter table users drop column name"),
}
def fresh():
    db = sqlite3.connect(":memory:"); db.execute("create table users(id integer primary key, name text not null)"); db.execute("insert into users values (1, 'Ada')"); return db

def works(app, steps):
    db = fresh()
    for s in steps: STEPS[s](db)
    try:
        app.write(db, 2, "Grace")
        return "ok" if (app.read(db, 1), app.read(db, 2)) == ("Ada", "Grace") else "FAIL"      # the old row and the new row must both read correctly
    except (sqlite3.Error, TypeError): return "FAIL"

if __name__ == "__main__":
    names = list(STEPS)
    def run(path, title):
        print(title)
        print(f"   {'schema after step':<44}{'v1':>6}{'v2':>6}")
        applied = []
        for s in path:
            applied.append(s); print(f"   {s:<44}{works(V1(), applied):>6}{works(V2(), applied):>6}")
    run([names[0], names[1]], "Naive: rename in one step")
    run([names[0], names[2], names[3], names[4]], "\nExpand, backfill, contract")
    run([names[0], names[2], names[4]], "\nExpand and contract, but the backfill was forgotten")
    print("\nReading it: v1 must keep working until the last v1 pod is gone AND the rollback window has closed, so 'contract' (S4) is its own later release.")
    print("A row written by v2 at S4 has no `name`; after the column is dropped, rolling back to v1 cannot work: the data and the code must be rolled back together, or not at all.")

Sources

Practise this topic: DevOps & Cloud

More guides