CS Fundamentals Trainer · Guides
Git internals and CI/CD pipelines
Git as a content-addressed object store (hashes, packfiles, bisect, rebase), branching strategies, continuous integration and delivery, and how canary, blue/green and rolling releases limit the damage of a bad release.
37 min read
Git looks like a set of commands to memorise, and it becomes much easier once you know that it is a small content-addressed database with a handful of object types, and that branches are just names for objects. CI/CD (continuous integration and continuous delivery) is then the discipline of putting every change through the same automated path, from a commit to running in production, in small steps that can be undone. This guide opens up the repository with the real commands and measures what it stores, and then covers pipelines, branching strategies and the deployment strategies that limit the damage of a bad release. Everything in the Git half was run on a real repository; the script is in the appendix.
It fits with containers and Kubernetes (what the pipeline deploys), observability (how you know a release is healthy) and security (secrets in history, supply chain).
1. Git is a content-addressed object store
Git stores four kinds of objects, each named by the hash of its content (SHA-1 by default; SHA-256 in repositories created with --object-format=sha256, see below) (with a short header):
| Object | Holds | Analogy |
|---|---|---|
| blob | the bytes of one file (no name, no permissions) | file content |
| tree | a list of names, modes and the ids of blobs and other trees | a directory |
| commit | the id of a tree, the ids of its parent commits, author, committer, time and message | a snapshot with history |
| tag | a named, optionally signed pointer to an object | a label |
I checked the naming rule by hand. For a file containing hello and a newline, git hash-object printed ce013625030ba8dba906f756967f9e9ca394464a, and hashing the string blob 6, a zero byte and hello with plain sha1sum printed the same id. The name of the object is its content, which has three consequences:
- Identical content is stored once. I added a copy of the file under another name: both entries in the index pointed to the same blob id, so the repository gained no new blob.
- An object cannot change without changing its name. A commit's id covers its tree and its parents, which cover everything below them, so a commit id verifies the entire history behind it (a Merkle tree or hash chain, as in replication and blockchains). Tampering with an old commit changes every id after it.
- Objects are immutable; "changing" a commit means creating a new one.
Diagram: A commit points to a tree, and the tree to blobs; the branch is only a name
Both trees point to one blob, because the file did not change.
Branches, HEAD and the index
A branch is a file under .git/refs/heads/ containing one commit id: in the experiment it was exactly 41 bytes, 40 hexadecimal characters and a newline. Creating a branch writes 41 bytes, which is why branching is cheap in Git; and committing moves the current branch's pointer to the new commit. HEAD says which branch (or commit) you are on. The index (staging area) is the proposed next tree: git add writes blobs and updates the index, and git commit turns the index into a tree and a commit.
Commits are snapshots, not diffs. Differences are computed when needed (and a commit shown by git show is a difference between two snapshots).
Packfiles: snapshots stored as deltas
If every version were a full copy, history would be huge. New objects are first stored "loose", one compressed file each; git gc (or a push) bundles them into a packfile, where similar objects are stored as deltas against each other. I created a 5,000-line file (108 KB) and committed 200 small edits to it:
| Storage | Size |
|---|---|
| 201 full copies, uncompressed | about 21,873 KB |
after git gc --aggressive: one pack with 608 objects | about 102 KB |
A reduction of more than 200 times, because each edit changed one line. This is why a repository with long history is smaller than you expect, and why large binary files are the exception: a re-encoded video or image changes almost every byte, so deltas do not help, and every version is kept forever. Use Git LFS or an artifact store for them.
The same fact is behind the biggest Git mistake: a file committed once is in the history forever. Deleting it in a later commit removes it from the tree, not from the old commits. A leaked key must be revoked first (it is compromised the moment it is pushed); rewriting history to remove it (git filter-repo) is cleanup, not a fix. See the security guide.
SHA-1 and SHA-256
Git was designed with SHA-1 for identity, not security. After practical SHA-1 collisions were demonstrated in 2017, Git added collision detection, and it supports SHA-256 repositories as a transition. The important point for design is that content addressing, as in event logs and container image digests, gives integrity and deduplication together.
SHA-256 repositories in practice
Everything above says "SHA-1", but a Git repository can use SHA-256 instead: git init --object-format=sha256 (Git 2.29 and later; the Git project no longer calls it experimental since 2.42). The object model is identical; only the hash function, and therefore every id, changes. I created the same one-file, one-commit repository in both formats (Git 2.43, with fixed dates so that the only difference is the format):
| SHA-1 repository | SHA-256 repository | |
|---|---|---|
id of the hello blob | ce013625030ba8dba906f756967f9e9ca394464a (40 hex characters) | 2cf8d83d9ee29543b34a87727421fdecb7e3f3a183d337639025de576db9ebb4 (64) |
| id of the first commit | bc31b899… | 58f1538c… (a different id for the same content) |
git log --oneline | bc31b89 first | 58f1538 first (abbreviations are still short) |
| repository config | core.repositoryformatversion = 0, no extension | core.repositoryformatversion = 1, extensions.objectformat = sha256 |
| naming rule by hand | sha1("blob 6" + NUL + "hello\n") | sha256("blob 6" + NUL + "hello\n"), which I computed with sha256sum and got the id above |
What follows from it:
- The format is part of the repository's identity. The same content has different ids in the two formats, so ids, tags and signatures from one repository mean nothing in the other.
git fetchfrom a SHA-1 repository into a SHA-256 one failed at once withfatal: mismatched algorithms: client sha256; server sha1. You cannot convert a repository in place with the Git I tested; migrating means creating a new repository and importing the history (for example withgit fast-export/fast-import, which rewrites every id, and with it every signature, every reference to a commit id in issues, CI configuration and documents). The Git project documents a transition plan with interoperability between the two formats, but treat it as a plan, not as something to depend on. - The cost is small. On a generated history of 300 commits and 6,600 objects, the packed repository was 12,500 KB with SHA-1 and 12,688 KB with SHA-256 (1.5% more, from the longer ids);
git gc --aggressivetook 33.2 s and 34.7 s andgit log --stat1.17 s and 0.99 s, differences within the noise of a shared machine. Hashing raw data ran at about 1,494 MB/s (SHA-1) and 1,345 MB/s (SHA-256) in Python on this CPU, which has hardware support for both; on a CPU without SHA instructions SHA-256 is slower, and neither is the bottleneck of everyday Git commands. - Why move at all. SHA-1 collisions were demonstrated in 2017 (the SHAttered attack), and since Git 2.13 it uses a collision-detecting SHA-1 that rejects the known attack patterns. For ordinary use the risk is low, but a commit id is the root of trust of a signed tag or commit, so a cryptographically stronger hash is the sensible long-term answer.
- Ecosystem support is the real obstacle. In the sources I could read (a preview of Git 3.0 published in 2026; check the current state before deciding): SHA-256 is not Git's default and is planned for Git 3.0, which has no release date; Forgejo and Dulwich support it, GitLab, go-git and libgit2 are experimental, and GitHub and Bitbucket did not support SHA-256 repositories. A repository your hosting provider cannot host is not usable for a team.
- What to do now. Stay on SHA-1 unless you have a specific requirement and your whole toolchain supports SHA-256; sign commits and tags (authenticity does not depend on the object hash alone); and stop hard-coding 40: a regular expression
[0-9a-f]{40}, aCHAR(40)column, a fixed-width log format or a tool that parsesgit rev-parseoutput will break on a 64-character id. Accept both lengths, so the day you switch is a configuration change.
2. Merge, rebase and rewriting history
Two branches that have diverged can be combined in two ways:
Diagram: Merge keeps both lines of history; rebase replays one line on top of the other
- A merge creates a commit with two parents and keeps the history as it really happened. Existing commits do not change.
- A rebase takes your commits and replays them on top of another commit, creating new commits with new parents. In the experiment, a one-commit feature branch had id
dba6fbebeforegit rebase mainand0cae772after: the same change, but a new object, since its parent is part of its identity. The result is a straight line that is easy to read and to bisect.
The rule that follows from immutability: never rewrite commits that others have already based work on (rebase or amend something pushed to a shared branch). Their copies still have the old ids, and the two histories now diverge, which is how a team ends up with duplicate commits and painful merges. Rewrite your own unpublished work freely; when you must update a branch that you share, say so, and use --force-with-lease instead of --force so that you do not overwrite a colleague's push.
A squash merge turns a branch into one commit on the main line. It gives a clean history and a one-to-one link between pull request and commit, at the cost of losing the individual steps (which matters for bisect inside a big change). Choose one policy per repository and enforce it.
Recovering: the reflog
Branch pointers move, and old commits remain in the object store until garbage collection (by default, unreachable objects are kept for weeks). The reflog records where HEAD and each branch pointed over time, so after a bad reset or rebase, git reflog shows the old id and git reset --hard <id> or git branch rescue <id> brings it back. Most "lost" work is not lost; work that was never committed is.
3. Finding the commit that broke something: bisect
git bisect is binary search over history (see complexity). You mark one good and one bad commit; Git checks out the middle, you (or a script) say good or bad, and it halves the range. I made 100 commits in which a test starts failing at c77, and ran git bisect run with a script that exits 0 for good and non-zero for bad:
| Commits in the range | Test runs needed | Result |
|---|---|---|
| 100 | 7 | c77 named as the first bad commit |
log2(100) is about 6.6, so seven runs; for 10,000 commits it is 14 runs. This works best with small, buildable commits (so every commit can be tested) and a fast, deterministic test, which is one more argument for the trunk-based practices below.
4. Branching strategies
| Strategy | How it works | Fits |
|---|---|---|
| Trunk-based development | everyone integrates into one main branch at least daily; branches live for hours, incomplete work is hidden behind feature flags | teams that deploy continuously; the practice the DORA research links to high performance |
| GitHub flow | short-lived feature branches, a pull request with review and CI, merge to main, deploy from main | most web teams |
| Gitflow | long-lived develop and release branches plus feature, hotfix branches | software shipped in versions that must be maintained in parallel (installed products, libraries) |
| Release branches | branch at release, backport fixes | the same, more lightly |
The reason short branches win is merge cost: the cost of combining two lines of work grows faster than their size, since conflicts grow with the number of changes and with how long each side has forgotten the other's context. Long-lived branches postpone that cost and make it bigger. Feature flags let you merge unfinished code safely and then turn a feature on for 1% of users, which separates deploying code from releasing a feature.
5. Continuous integration and delivery
- Continuous integration (CI): every change is merged into the main line often (at least daily) and each merge is built and tested automatically. The goal is to find integration problems within minutes, not at the end of a release.
- Continuous delivery: the main line is always releasable; every change that passes the pipeline can be deployed by pressing a button.
- Continuous deployment: every change that passes is deployed to production automatically, without a button.
Diagram: A typical pipeline: fast checks first, one artifact promoted through environments
Principles that make a pipeline trustworthy:
- Fast feedback first. Run the cheapest, most informative checks first and fail early (the order in this repository's own CI is type check, lint, tests, build). A pipeline that takes an hour is skipped or worked around.
- Build once, deploy the same artifact everywhere. The binary or image tested in staging must be byte-for-byte the one in production (identified by its digest or commit id), with differences only in configuration. Rebuilding per environment tests something different from what ships (twelve-factor: build, release, run).
- Keep configuration and secrets out of the artifact and inject them at deploy time.
- A test pyramid: many fast unit tests, fewer integration tests, few end-to-end tests. Slow, flaky end-to-end tests at the top are expensive; a flaky test that is retried until green is worse than none, because it teaches everyone to ignore red.
- Schema changes and code changes ship separately, as additive steps (expand, deploy, then contract), since during a rollout both versions run against one database; see replication and event versioning.
- Protect the pipeline: it can deploy to production, so its credentials are the keys to the system. Use short-lived credentials, least privilege, pinned dependencies and actions, and signed artifacts (see supply-chain security in the security guide).
- Measure it with the four DORA metrics: deployment frequency, lead time for changes, change failure rate and time to restore service. Frequent, small releases improve all four, because a small change is easy to understand, test and revert.
A concrete pipeline: provenance, pinned dependencies, secret scanning, migrations and rollback
The principles above become checks. Here is one pipeline, with the failure each stage prevents and a runnable demonstration of the parts that can be shown without a CI server (the scripts are in the appendix). The pipeline definition at the end is a sketch of how it fits together; the CI products and scanners it names are not demonstrated here.
Diagram: A pipeline with supply-chain and rollback controls
1. Dependency pinning. A version range means the same commit builds different code on different days. In the demonstration lib>=2.0 installs 2.31.0 in January and 3.0.0 in June. A lock file with hashes (pip install --require-hashes, package-lock.json, go.sum) fixes both the version and the bytes: a download whose sha-256 differs from the lock is refused (REFUSED lib==2.31.0: hash 42c962426d1f… does not match the lock file's fec2012abbd8…). Commit the lock file, update it deliberately (a bot opens the pull request, the pipeline tests it), and run an audit of the locked set on every build (pip-audit, npm audit, OSV-Scanner; the scan examples are in the security guide). Pin third-party CI actions to a full commit id, not to a moving tag.
2. Build once, reproducibly, and record where it came from. Building the same source twice with default settings gave two different archives (75978fa7e9035769… and c9b605c62ba3ef43…), because of embedded timestamps and file order. Normalising them (sorted names, fixed modification time and owner) gave the identical ea511b3da0f25700… both times. A reproducible build lets anyone check an artifact by rebuilding it, and makes the digest a stable identity. Then the build job signs a provenance statement: the artifact's digest, the repository and commit, the branch, the builder and the inputs. The deploy step verifies it before it runs anything. In the demonstration (Ed25519 signatures; a trusted key held by the deploy step):
| Case | Deploy step's verdict |
|---|---|
genuine build from main | ACCEPT: built by ci-runner-1 from 9f1c2ab0 on main |
| artifact swapped after the build | REJECT: the artifact is not the one that was built |
| statement edited to claim another commit | REJECT: signature does not verify |
| signed with a key the deploy step does not trust | REJECT: signature does not verify |
| genuine build, but from a feature branch | REJECT: not a protected branch |
In real pipelines you do not write this yourself: use the platform's artifact attestations or Sigstore/cosign, in the SLSA provenance format, and verify with the same tools in the deploy step or in a Kubernetes admission policy. The point is the policy: only artifacts built by our pipeline from a protected branch run in production. Tag container images by digest (image@sha256:…), not by a mutable tag.
3. Secret scanning, before and after the fact. I built a repository where a .env file with an AWS example key was committed and then deleted in the next commit. A scan of the files now in the working tree found 0 problems. A scan of every commit's added lines found the access key id and the secret key in the commit "add config". git log --all -S AKIAIOSFODNN7EXAMPLE lists both commits, and git show of the old one prints the file: deleting a secret does not remove it. A pre-commit check on the staged changes blocked a commit that added TOKEN=ghp_…. So: scan staged changes locally (pre-commit hook), scan the pull request and the full history in CI (gitleaks, TruffleHog), and turn on the hosting platform's push protection where it exists. When something leaks, the order is: revoke and rotate first, then clean history if you must, then find out whether it was used. Tune the scanner (entropy thresholds, allowlists for test fixtures, an expiry on every suppression): in the demonstration it did not flag a Git commit id, a UUID or a placeholder password in a notes file, which is what keeps people from ignoring it.
4. Database migrations and rollback. During a rolling deploy, the old and the new application versions run against the same database, and a rollback runs the old version again. A schema change must therefore work for both. I tested renaming a column (name to full_name) with SQLite and two application versions:
| Schema after the step | Old app (v1) | New app (v2, tolerates both columns) |
|---|---|---|
start (name) | ok | FAIL (the new column is not there yet) |
| naive: rename in one step | FAIL | ok |
expand: add full_name | ok | ok |
backfill: copy name into full_name | ok | ok |
contract: drop name | FAIL | ok |
| contract without a backfill | FAIL | FAIL (old rows have no value) |
The naive rename breaks every pod that is still running v1, and breaks the rollback. The safe sequence is expand (additive, runs before the deploy), deploy v2 (reads either column, writes both), backfill (in small batches), and, only after v1 is gone and the rollback window has closed, a later release that contracts. Rules that go with it:
- Rolling back code is easy; rolling back data is not. A "down" migration that drops a column destroys what v2 wrote there. Prefer forward fixes and feature flags, keep destructive steps for last, and take a backup or snapshot before them.
- Run migrations as their own pipeline step, once (not on every pod start), with a lock, a timeout and logging, and make them idempotent. Check how long they take on production-sized data: some changes lock a table for the duration (in PostgreSQL, adding a nullable column is quick, and an index should be created
CONCURRENTLY; see the database documentation for your version). - Rollback of the application is "redeploy the previous artifact by digest", which only works if that artifact is still stored and the schema is still compatible with it. Retain artifacts and test the rollback path regularly, as part of releases.
- Gate promotion on health signals (error rate, latency) and let the pipeline roll back on its own (section 6 on deployment strategies, and observability).
A sketch of the pipeline as configuration (GitHub Actions syntax; not run here):
name: release
on: { push: { branches: [main] } }
permissions: { contents: read, id-token: write, attestations: write } # least privilege; the signing identity comes from the platform
jobs:
verify:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@<full-commit-sha> # pinned to a commit, not a tag
- run: gitleaks detect --redact # secrets in the history
- run: pip install --require-hashes -r requirements.lock && pip-audit -r requirements.lock
- run: python -m unittest discover # unit and security regression tests
build:
needs: verify
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@<full-commit-sha>
- run: docker build -t app:${{ github.sha }} . && docker push registry/app:${{ github.sha }}
- uses: actions/attest-build-provenance@<full-commit-sha> # signed provenance for the image digest
deploy:
needs: build
environment: production # required reviewers, protected secrets
steps:
- run: ./migrate.sh expand # additive schema changes only
- run: cosign verify-attestation … && ./deploy.sh --digest "$DIGEST" --canary 10%
- run: ./wait_for_health.sh --error-rate-below 1% || ./deploy.sh --rollback # automatic rollback to the previous digest
6. Deployment strategies: limiting the blast radius
Every release is a risk, and what you control is how many users meet a problem before you notice it.
| Strategy | How | Rollback | Cost and catch |
|---|---|---|---|
| Recreate | stop the old version, start the new | redeploy | downtime |
| Rolling | replace instances gradually (see Kubernetes) | roll forward or back, slowly | both versions run together, so they must be compatible |
| Blue/green | run a full second environment, test it, then switch the router | switch back (seconds) | double the capacity during the release; the database is shared, so schema changes still need care |
| Canary | send a small share of real traffic to the new version, compare its metrics, then increase | send the share back to 0 | needs good metrics and an automated gate |
| Feature flag / dark launch | ship the code turned off, enable per user group | turn the flag off | flag debt: remove old flags |
| Shadow (mirror) | copy production traffic to the new version and discard its responses | not needed (users never see it) | doubles load; side effects must be avoided |
A canary release is a staged rollout with a gate between stages. Step through one with stages of 5%, 25%, 50% and 100%, a limit of 1% errors, and then turn on a version with a bug:
Interactive animation: canary release
In the model, a healthy version passes four stages. The version that fails 8% of requests is stopped at the first stage: it affected 4 of every 1,000 requests in that window (4 errors in the 50 requests it received) instead of 80 of every 1,000 had everybody been switched at once, a 20-fold reduction, which is exactly the share it was allowed to take (5%). A canary only protects you if:
- the traffic share is large enough to detect the problem: with 50 requests an error rate of 0.1% cannot be told from zero, so each stage needs enough requests (or time) for the statistics to mean something;
- the gate compares the right signals: error rate, latency percentiles, saturation, and business metrics such as the checkout rate, against the old version running at the same time, not against yesterday (observability);
- the rollback is automatic and tested. The best moment to find out that rollback does not work is not during an incident.
Whatever the strategy, the old and new versions run side by side for a while, so API and database changes must be backward compatible for at least one version (REST versioning, microservices).
7. Infrastructure as code and GitOps
Environments built by hand drift apart and cannot be reproduced. Infrastructure as code describes servers, networks and clusters in files that are reviewed, versioned and applied by tools. GitOps applies the same idea to deployment: the Git repository is the desired state of the environment, and an agent in the cluster continuously compares reality with the repository and fixes the difference, the same reconcile loop as Kubernetes controllers. A deployment is a commit, a rollback is a revert, and the audit log is the Git history. Keep secrets out of the repository (use a secrets manager or encrypted values).
8. Common mistakes
- Long-lived branches and big-bang merges.
- Rebasing or force-pushing shared branches.
- Committing secrets,
.envfiles or large binaries, then "deleting" them in a later commit. - A red main branch that people merge on top of; broken builds not fixed first.
- Flaky tests that are re-run until they pass.
- Rebuilding the artifact for each environment, or deploying
latest. - Manual steps in the release process that "only Anna knows".
- Rollback that has never been tried, or that is impossible because a migration cannot be undone.
- A canary without a metric gate, or judged on too little traffic.
- Pipeline credentials with broad, permanent access to production.
- Commits that mix refactoring, formatting and a behaviour change, which cannot be reviewed, bisected or reverted separately.
9. Things to mention in an interview
- Git stores blobs, trees, commits and tags named by the hash of their content; a branch is a 41-byte pointer; commits are snapshots; packfiles store deltas.
- Why identical files are stored once, why history is tamper-evident, and why a leaked secret is never removed by a new commit.
- Merge against rebase, the golden rule of not rewriting shared history, and the reflog for recovery.
git bisect: binary search, about log2(n) steps.- Trunk-based development against Gitflow, and feature flags to separate deploy from release.
- CI against continuous delivery against continuous deployment; build once, promote the artifact; test pyramid; DORA metrics.
- Rolling, blue/green, canary, feature flags and shadow traffic, and what each needs (compatibility, capacity, metrics).
- Backward-compatible schema changes as expand and contract.
Quick recap
- Git is a content-addressed store: the name of an object is the SHA-1 of its content, so identical content is stored once, history is tamper-evident, and objects never change. The experiment confirmed the hash by hand, a branch of 41 bytes, and a saving of more than 200 times from packfile deltas (about 21,873 KB to about 102 KB).
- Rebase creates new commits (new ids); merge keeps old ones. Do not rewrite shared history; the reflog recovers local mistakes.
- Bisect found the bad commit among 100 in 7 test runs.
- Short-lived branches and feature flags keep integration cheap; CI/CD gives fast feedback, builds once and promotes one artifact, and is measured by the DORA metrics.
- Canary, blue/green and rolling releases limit the blast radius; in the model a 5% canary turned 80 failures per 1,000 requests into 4. They rely on backward-compatible changes, honest metrics and a rollback that is automatic.
Appendix: the experiment code
One shell script that needs only git and Python 3. It creates a fresh repository in a new directory, ignores your global Git configuration, and prints each result. The object ids and timestamps differ on every run (they depend on the time), but the structure, the hash of the hello blob (which is always ce013625030ba8dba906f756967f9e9ca394464a), the sizes and the bisect step count are repeatable. The pack size comes from git gc --aggressive on one machine and Git version and will differ slightly on others. The canary figures are computed by the animation's model with expected values (no randomness), not measured on a real service.
git.sh: objects, packs, bisect and rebase
#!/bin/bash
# Git internals experiments. Needs only git and python3.
set -eu
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null
export GIT_AUTHOR_NAME=a GIT_AUTHOR_EMAIL=a@x GIT_COMMITTER_NAME=a GIT_COMMITTER_EMAIL=a@x
rm -rf r && git init -q -b main r && cd r
echo "--- 1. a blob's name is the SHA-1 of 'blob <size>\\0<content>'"
printf 'hello\n' > a.txt
echo "git hash-object: $(git hash-object a.txt)"
echo "by hand (sha1): $(printf 'blob 6\0hello\n' | sha1sum | cut -d' ' -f1)"
git add a.txt; git commit -qm one
echo "--- 2. the objects: commit -> tree -> blob"
git cat-file -p HEAD; git cat-file -p 'HEAD^{tree}'
echo "--- 3. the same content stored twice is one object"
cp a.txt b.txt; git add b.txt; git commit -qm two
git ls-files -s
echo "--- 4. a branch is a file holding one commit id (40 hex characters and a newline)"
cat .git/refs/heads/main; wc -c < .git/refs/heads/main
echo "--- 5. delta compression: a 5000-line file edited 200 times"
python3 - <<'PY'
import random
r=random.Random(1); open('big.txt','w').write(''.join(f"line {i} {r.random():.10f}\n" for i in range(5000)))
PY
git add big.txt; git commit -qm big
for i in $(seq 1 200); do sed -i "$((i*20))s/.*/edited $i/" big.txt; git commit -qam "edit $i"; done
size=$(wc -c < big.txt)
echo "file: $((size/1024)) KB; 201 full copies would be $((201*size/1024)) KB"
git gc -q --aggressive
git count-objects -vH | grep -E 'size-pack|in-pack'
echo "--- 6. bisect: 100 commits, the test starts failing at c77"
git checkout -q -b bisect-demo
for i in $(seq 1 100); do echo "$i" > counter.txt; [ "$i" -ge 77 ] && echo bug > broken.txt; git add -A; git commit -qm "c$i"; done
cat > test.sh <<'T'
#!/bin/sh
[ ! -f broken.txt ]
T
chmod +x test.sh; cp test.sh /tmp/gt.sh
first=$(git rev-list --max-parents=0 HEAD | tail -1)
git bisect start HEAD main >/dev/null
git bisect run /tmp/gt.sh | grep -E "is the first bad commit|running" | wc -l
git bisect log | grep -c '^git bisect \(good\|bad\)'
git bisect log | grep "first bad" || git show -s --format='first bad commit: %s' refs/bisect/bad
git bisect reset >/dev/null 2>&1
echo "--- 7. merge keeps commit ids; rebase rewrites them"
git checkout -q main; git checkout -q -b feature; echo f > f.txt; git add f.txt; git commit -qm feature
git checkout -q main; echo m > m.txt; git add m.txt; git commit -qm mainline
git checkout -q feature; echo "feature commit before rebase: $(git rev-parse --short HEAD)"
git rebase -q main; echo "feature commit after rebase: $(git rev-parse --short HEAD) (same change, new parent, new id)"
SHA-1 and SHA-256 repositories side by side
A shell script that needs git 2.29 or newer, bc and Python 3. It creates throwaway repositories and removes them.
#!/bin/bash
# SHA-1 and SHA-256 repositories side by side: identifiers, size, speed, and what does not interoperate. Needs git 2.29 or newer.
set -e
W=$(mktemp -d); cd "$W"
export GIT_AUTHOR_DATE='2026-01-01T00:00:00Z' GIT_COMMITTER_DATE='2026-01-01T00:00:00Z' GIT_AUTHOR_NAME=t GIT_AUTHOR_EMAIL=t@t GIT_COMMITTER_NAME=t GIT_COMMITTER_EMAIL=t@t
for f in sha1 sha256; do
git init -q --object-format=$f r_$f; cd r_$f
printf 'hello\n' > a.txt; git add a.txt; git commit -q -m first
echo "== $f: blob $(git hash-object a.txt)"
echo " commit $(git rev-parse HEAD) ($(git rev-parse HEAD | tr -d '\n' | wc -c) hex characters), git log --oneline shows: $(git log --oneline)"
echo " config: repositoryformatversion=$(git config core.repositoryformatversion) extensions.objectformat=$(git config extensions.objectformat || echo '(not set: SHA-1)')"
cd ..
done
echo "== the naming rule by hand: sha256 of 'blob 6<NUL>hello<LF>' = $(printf 'blob 6\0hello\n' | sha256sum | cut -d' ' -f1)"
echo "== the same content gets a different commit id: the format is part of the identity, so ids from one repository mean nothing in the other"
echo "== fetch across formats:"; (cd r_sha256 && git fetch ../r_sha1 2>&1 | head -1) || true
# a history of 300 commits touching 20 files each, in both formats
for f in sha1 sha256; do
git init -q --object-format=$f big_$f; cd big_$f
python3 - <<'P'
import random, subprocess, os
r = random.Random(1)
for c in range(300):
for k in range(20):
open(f"f{(c*7+k) % 400}.txt", "w").write("".join(r.choice("abcdefgh \n") for _ in range(4000)))
subprocess.run(["git", "add", "-A"], check=True); subprocess.run(["git", "commit", "-q", "-m", f"c{c}"], check=True)
P
T0=$(date +%s.%N); git gc -q --aggressive; echo " $f: git gc --aggressive took $(echo "$(date +%s.%N) - $T0" | bc) s"
echo " $f: objects $(git count-objects -v | awk '/in-pack/ {print $2}'), pack size $(du -sk .git/objects/pack | cut -f1) KB, total .git $(du -sk .git | cut -f1) KB"
T0=$(date +%s.%N); git log --stat >/dev/null; echo " $f: git log --stat over 300 commits took $(echo "$(date +%s.%N) - $T0" | bc) s"
cd ..
done
rm -rf "$W"
Pinned dependencies, a reproducible build and signed provenance
Python 3 with the cryptography package for Ed25519 signatures. Local data stands in for package downloads and a build runner.
"""Supply-chain basics for a pipeline: pinned and hash-checked dependencies, a reproducible build, and signed provenance that the deploy step verifies.
Standard library plus `cryptography` (Ed25519). Local files stand in for downloads and for a build runner."""
import gzip, hashlib, io, json, os, tarfile, time
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
from cryptography.hazmat.primitives import serialization
from cryptography.exceptions import InvalidSignature
sha = lambda b: hashlib.sha256(b).hexdigest()
# ------------------------------------------------------------------ 1. pinning with hashes
def pin_demo():
print("1. Dependencies: a range changes under you, a pin with a hash cannot")
index = {"2026-01-01": ["2.30.0", "2.31.0"], "2026-06-01": ["2.30.0", "2.31.0", "2.32.0", "3.0.0"]} # versions available on two dates
pick = lambda day, rule: max((v for v in index[day] if rule(v)), key=lambda v: tuple(map(int, v.split("."))))
rule = lambda v: tuple(map(int, v.split("."))) >= (2, 0, 0)
print(f" 'lib>=2.0' installs {pick('2026-01-01', rule)} in January and {pick('2026-06-01', rule)} in June: the same commit builds with different code")
wheel = b"PK-not-really-a-wheel: lib 2.31.0 contents" # the file that was reviewed and locked
lock = {"lib": ("2.31.0", sha(wheel))}
def install(name, content):
ver, want = lock[name]; got = sha(content)
return f"installed {name}=={ver}" if got == want else f"REFUSED {name}=={ver}: hash {got[:12]}… does not match the lock file's {want[:12]}…"
print(" lock file, genuine download: ", install("lib", wheel))
print(" lock file, tampered download: ", install("lib", wheel + b" + a backdoor"))
# ------------------------------------------------------------------ 2. a reproducible build
def build_archive(files, deterministic, mtime_hint):
raw = io.BytesIO()
with tarfile.open(fileobj=raw, mode="w") as tar:
for name in (sorted(files) if deterministic else list(files)):
data = files[name]; info = tarfile.TarInfo(name); info.size = len(data)
info.mtime = 0 if deterministic else mtime_hint; info.uid = info.gid = 0 if deterministic else 1000; info.mode = 0o644
tar.addfile(info, io.BytesIO(data))
out = io.BytesIO()
with gzip.GzipFile(fileobj=out, mode="wb", mtime=0 if deterministic else int(mtime_hint)) as g: g.write(raw.getvalue())
return out.getvalue()
def repro_demo():
print("2. A build you can check: the same source twice")
files = {"app/main.py": b"print('hello')\n", "app/util.py": b"X = 1\n", "VERSION": b"1.4.2\n"}
a = build_archive(files, False, time.time()); time.sleep(1.1); b = build_archive(files, False, time.time())
print(f" default build, run twice: {sha(a)[:16]}… {sha(b)[:16]}… identical: {a == b}")
a = build_archive(files, True, 0); b = build_archive(dict(reversed(list(files.items()))), True, 0)
print(f" timestamps and order normalised: {sha(a)[:16]}… {sha(b)[:16]}… identical: {a == b}")
return a
# ------------------------------------------------------------------ 3. signed provenance, verified before deploy
def provenance_demo(artifact):
print("3. Provenance: who built these bytes, from which commit, and does the deploy step check?")
ci_key = Ed25519PrivateKey.generate(); ci_pub = ci_key.public_key() # in real life the key never leaves the build system (or is keyless, as in Sigstore)
other_key = Ed25519PrivateKey.generate()
def attest(art, commit, branch, builder="ci-runner-1", key=ci_key):
stmt = json.dumps({"subject": {"name": "app.tar.gz", "sha256": sha(art)}, "source": {"repo": "example/app", "commit": commit, "branch": branch},
"builder": builder, "materials": {"requirements.lock": "sha256:…"}}, sort_keys=True).encode()
return stmt, key.sign(stmt)
ALLOWED_BUILDERS, PROTECTED = {"ci-runner-1", "ci-runner-2"}, {"main"}
def verify(art, stmt, sig):
try: ci_pub.verify(sig, stmt)
except InvalidSignature: return "REJECT: signature does not verify"
p = json.loads(stmt)
if p["subject"]["sha256"] != sha(art): return "REJECT: the artifact is not the one that was built"
if p["builder"] not in ALLOWED_BUILDERS: return f"REJECT: builder {p['builder']} not allowed"
if p["source"]["branch"] not in PROTECTED: return f"REJECT: built from branch {p['source']['branch']}, not a protected branch"
return f"ACCEPT: built by {p['builder']} from {p['source']['commit'][:8]} on {p['source']['branch']}"
s, sig = attest(artifact, "9f1c2ab07d", "main")
print(" genuine build from main: ", verify(artifact, s, sig))
print(" artifact swapped after the build: ", verify(artifact + b"x", s, sig))
print(" statement edited to claim another commit: ", verify(artifact, s.replace(b"9f1c2ab07d", b"deadbeef00"), sig))
s2, sig2 = attest(artifact, "9f1c2ab07d", "main", key=other_key)
print(" signed with a key the deploy step does not trust:", verify(artifact, s2, sig2))
s3, sig3 = attest(artifact, "77aa00bb11", "feature/x")
print(" genuine build, but from a feature branch: ", verify(artifact, s3, sig3))
if __name__ == "__main__":
pin_demo(); art = repro_demo(); provenance_demo(art)
A minimal secret scanner over a throwaway repository
Python 3 and git. The 'secrets' are the AWS documentation's example values or made up. Real scanners (gitleaks, TruffleHog) have far larger rule sets.
"""A minimal secret scanner run over a throwaway git repository, to show what scanning the current files misses and what the history keeps.
All 'secrets' are the well-known example values from the AWS documentation or made up; the real tools (gitleaks, TruffleHog, platform push protection) have far larger rule sets."""
import math, os, re, shutil, subprocess, tempfile
RULES = {"AWS access key id": re.compile(r"\bAKIA[0-9A-Z]{16}\b"), "private key": re.compile(r"-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----"),
"generic secret assignment": re.compile(r"(?i)(?:secret|token|password|api[_-]?key)\w*\s*[:=]\s*['\"]?([A-Za-z0-9/+_\-]{16,})")}
def entropy(s): return -sum(s.count(c) / len(s) * math.log2(s.count(c) / len(s)) for c in set(s))
def scan_text(text, min_entropy=3.5):
hits = []
for line in text.splitlines():
for name, rx in RULES.items():
m = rx.search(line)
if m and (name != "generic secret assignment" or entropy(m.group(1)) >= min_entropy): hits.append((name, line.strip()[:70]))
return hits
def git(*a, cwd): return subprocess.run(["git", *a], cwd=cwd, capture_output=True, text=True, check=True).stdout
def main():
d = tempfile.mkdtemp(); g = lambda *a: git(*a, cwd=d)
g("init", "-q"); g("config", "user.email", "t@t"); g("config", "user.name", "t")
open(f"{d}/app.py", "w").write("print('hello')\n"); g("add", "."); g("commit", "-qm", "first")
open(f"{d}/.env", "w").write("DB_HOST=db\nAWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE\nAWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\n")
g("add", "-A"); g("commit", "-qm", "add config") # the mistake
os.remove(f"{d}/.env"); g("add", "-A"); g("commit", "-qm", "remove .env (oops)") # the 'fix'
open(f"{d}/notes.md", "w").write("build id: 3f786850e387550fdab836ed7e6dc881de23001b\napi_key = changeme-changeme-changeme\nrequest id 123e4567-e89b-12d3-a456-426614174000\n")
g("add", "-A"); g("commit", "-qm", "notes")
print("1. scanning only the files in the working tree now:")
found = [h for f in os.listdir(d) if os.path.isfile(f"{d}/{f}") for h in scan_text(open(f"{d}/{f}").read())]
print(f" {len(found)} findings" + "".join(f"\n {n}: {l}" for n, l in found))
print("2. scanning every commit's added lines (what a history scan does):")
seen = set()
for c in g("rev-list", "--reverse", "HEAD").split():
for n, l in scan_text("\n".join(x[1:] for x in g("show", "-U0", "--format=", c).splitlines() if x.startswith("+") and not x.startswith("+++"))):
if (n, l) not in seen: seen.add((n, l)); print(f" commit {c[:8]} ({g('log', '-1', '--format=%s', c).strip()}): {n}: {l}")
print("3. the file is gone from HEAD, but still in the repository:")
print(" git log --all --oneline -S AKIAIOSFODNN7EXAMPLE ->", g("log", "--all", "--oneline", "-S", "AKIAIOSFODNN7EXAMPLE").strip().replace("\n", " | "))
first = g("log", "--all", "--format=%h", "-S", "AKIAIOSFODNN7EXAMPLE").split()[-1]
print(f" git show {first}:.env contains the key: {'AKIAIOSFODNN7EXAMPLE' in g('show', f'{first}:.env')}")
print("4. a pre-commit check on the staged changes:")
open(f"{d}/deploy.sh", "w").write("export TOKEN=ghp_16C7e42F292c6912E7710c838347Ae178B4a\n"); g("add", "."); staged = g("diff", "--cached", "-U0")
hits = scan_text("\n".join(x[1:] for x in staged.splitlines() if x.startswith("+") and not x.startswith("+++")))
print(" staged diff:", f"BLOCK the commit: {hits[0][0]} in {hits[0][1]!r}" if hits else "clean")
shutil.rmtree(d)
if __name__ == "__main__": main()
Renaming a column across a rolling deploy
Python 3 with sqlite3.
"""Database migrations and rollback: which versions of the application work with which schema step. SQLite; the 'application' versions are two small classes.
Goal: rename a column (name -> full_name) while old and new code run together during a rolling deploy, and keep rollback possible."""
import sqlite3
class V1: # the version in production: knows only `name`
label = "v1 (reads/writes name)"
def write(self, db, i, n): db.execute("insert into users(id, name) values (?, ?)", (i, n))
def read(self, db, i): return db.execute("select name from users where id=?", (i,)).fetchone()[0]
class V2: # the new version: writes both columns when it can, reads full_name with a fallback to name
label = "v2 (reads/writes full_name, tolerates both)"
def write(self, db, i, n):
cols = {r[1] for r in db.execute("pragma table_info(users)")}
if "name" in cols and "full_name" in cols: db.execute("insert into users(id, name, full_name) values (?, ?, ?)", (i, n, n))
elif "full_name" in cols: db.execute("insert into users(id, full_name) values (?, ?)", (i, n))
else: db.execute("insert into users(id, full_name) values (?, ?)", (i, n)) # the column is not there yet: this will fail
def read(self, db, i):
cols = {r[1] for r in db.execute("pragma table_info(users)")}
sel = "coalesce(full_name, name)" if {"name", "full_name"} <= cols else "full_name"
return db.execute(f"select {sel} from users where id=?", (i,)).fetchone()[0]
STEPS = {
"S0 start: column name": lambda db: None,
"S1 NAIVE: rename name -> full_name": lambda db: db.execute("alter table users rename column name to full_name"),
"S2 expand: add full_name (nullable)": lambda db: db.execute("alter table users add column full_name text"),
"S3 backfill: copy name into full_name": lambda db: db.execute("update users set full_name = name where full_name is null"),
"S4 contract: drop name": lambda db: db.execute("alter table users drop column name"),
}
def fresh():
db = sqlite3.connect(":memory:"); db.execute("create table users(id integer primary key, name text not null)"); db.execute("insert into users values (1, 'Ada')"); return db
def works(app, steps):
db = fresh()
for s in steps: STEPS[s](db)
try:
app.write(db, 2, "Grace")
return "ok" if (app.read(db, 1), app.read(db, 2)) == ("Ada", "Grace") else "FAIL" # the old row and the new row must both read correctly
except (sqlite3.Error, TypeError): return "FAIL"
if __name__ == "__main__":
names = list(STEPS)
def run(path, title):
print(title)
print(f" {'schema after step':<44}{'v1':>6}{'v2':>6}")
applied = []
for s in path:
applied.append(s); print(f" {s:<44}{works(V1(), applied):>6}{works(V2(), applied):>6}")
run([names[0], names[1]], "Naive: rename in one step")
run([names[0], names[2], names[3], names[4]], "\nExpand, backfill, contract")
run([names[0], names[2], names[4]], "\nExpand and contract, but the backfill was forgotten")
print("\nReading it: v1 must keep working until the last v1 pod is gone AND the rollback window has closed, so 'contract' (S4) is its own later release.")
print("A row written by v2 at S4 has no `name`; after the column is dropped, rolling back to v1 cannot work: the data and the code must be rolled back together, or not at all.")
Sources
- Scott Chacon and Ben Straub, Pro Git, especially chapter 10, "Git Internals" (objects, refs, packfiles), and the chapters on branching, rebasing and
git bisect. - Git documentation: gitformat-pack, git-hash-object, git-bisect, git-rebase, git-reflog and Git and hash function transition.
- Marc Stevens et al., The first collision for full SHA-1 (the "SHAttered" attack, 2017), and the Git documentation on SHA-1 collision detection.
- Jez Humble and David Farley, Continuous Delivery (Addison-Wesley, 2010); Martin Fowler, Continuous Integration, Feature Toggles, BlueGreenDeployment and CanaryRelease.
- Nicole Forsgren, Jez Humble and Gene Kim, Accelerate (IT Revolution, 2018), and the DORA research on the four key delivery metrics.
- Paul Hammant, Trunk Based Development.
- Google, Site Reliability Engineering (O'Reilly, 2016), chapter 8 "Release Engineering", and The Site Reliability Workbook, chapter 16 "Canarying Releases".
- The Twelve-Factor App, factors V (build, release, run) and X (dev/prod parity); the SLSA framework for build provenance.
- Git documentation: hash-function-transition,
git init --object-formatand the release notes of Git 2.42; Marc Stevens et al., The first collision for full SHA-1 (2017). - SLSA (supply-chain levels and the provenance format), Sigstore (cosign), and the documentation of your CI platform on artifact attestations; pip's hash-checking mode.
- Martin Fowler, Parallel Change (expand and contract); PostgreSQL documentation,
CREATE INDEX CONCURRENTLY.
Practise this topic: DevOps & Cloud