CS Fundamentals Trainer · Guides
Git submodules, subtrees, worktrees, sparse and partial clones, and LFS
What each Git tool for sharing code and handling big repositories really does, with real command output: submodules (gitlinks, .gitmodules, detached HEAD, update, push order, removal, pitfalls), subtrees, worktrees, shallow and partial clones, sparse checkout, Git LFS, and a decision chart that includes the option of a package manager.
12 min read
Sooner or later one repository needs code that lives in another: a shared library, a vendored dependency, a design system used by five apps. Or the opposite problem appears: a repository grows so large that cloning it takes minutes. Git has several tools for these cases, and they are often confused: submodules, subtrees, worktrees, shallow and partial clones, sparse checkout and Git LFS. This guide explains each one by what it really does underneath, shows the commands with the output of real runs (Git 2.43), and ends with a table for choosing. It assumes you know commits and branches; the object model is in the guide "Git internals and CI/CD pipelines".
1. The problem and the options
| Need | Tool | Idea |
|---|---|---|
| Use another project's code at an exact version, keep it as a separate repo | Submodule | the outer repo records a commit id of the inner repo |
| Copy another project's code into yours, with its history, no special workflow for others | Subtree | the files become ordinary files in your repo |
| Several branches checked out at once | Worktree | one repository, many working directories |
| Clone faster, with less history | Shallow clone | only the last N commits |
| Clone faster, fewer files' contents | Partial clone | blobs fetched when needed |
| Work on a slice of a big tree | Sparse checkout | only some directories on disk |
| Big binary files | Git LFS | the repo stores small pointers; the content lives elsewhere |
| Shared libraries with versions | A package manager | not Git at all: publish to a registry (often the best answer) |
Before reaching for Git, ask whether a package (npm, Go modules, PyPI) would do. Submodules and subtrees are for the cases where it would not: code that is not published, that you change together with the main project, or that must be pinned to an exact commit.
2. Submodules
What a submodule really is
A submodule is a pointer from one repository to a specific commit of another. The outer repository (the superproject) does not contain the inner files. It contains two things:
- A tree entry of mode
160000(a "gitlink") that stores the inner commit id. - A file
.gitmodulesthat says where to clone it from.
This is what git ls-tree -r HEAD printed after adding one:
100644 blob 9b6e268... .gitmodules
100644 blob b80f0bd... app.txt
160000 commit 26f28f8... vendor/lib
and .gitmodules:
[submodule "vendor/lib"]
path = vendor/lib
url = ../lib
The inner repository is a normal Git repository; its data is kept under .git/modules/ of the superproject, and the working directory vendor/lib has a .git file pointing there.
Adding and cloning
$ git submodule add ../lib vendor/lib
$ git status --short
A .gitmodules
A vendor/lib
$ git commit -m "add lib"
When someone clones the superproject, the submodule directory is empty: Git fetched the pointer but not the files. In the test, the plain clone had 0 files in vendor/lib and git submodule status showed a leading - (not initialised). Two ways to get the content:
git clone --recurse-submodules <url> # all at once
# or, after a plain clone:
git submodule update --init --recursive
After this the submodule is on a detached HEAD at exactly the recorded commit ("HEAD detached at c30e138"). That is by design: the superproject says "I need this commit", not "I need this branch". Setting git config submodule.recurse true makes pull, checkout and other commands also update submodules; without it you must run git submodule update after switching branches or pulling, or the inner files stay at the old commit.
Changing the inner project
To move the pointer forward, go inside, update, and commit the new pointer in the outer repo:
$ cd vendor/lib && git pull origin main # the inner repo gets lib 3
$ cd ../..
$ git status --short
M vendor/lib
$ git diff --submodule
Submodule vendor/lib c30e138..26f28f8:
> lib 3
$ git add vendor/lib && git commit -m "bump lib"
Notice that the outer repo sees only "the pointer changed". git diff --submodule (or git config diff.submodule log) shows the inner commits. If you edit files inside the submodule, you must commit and push the inner repository first, then commit the pointer in the outer repository. If you commit the pointer and push only the outer repo, everyone else gets a pointer to a commit that does not exist on the server, and their update fails. git push --recurse-submodules=check refuses to push in that situation, and =on-demand pushes the inner ones for you. Set push.recurseSubmodules=check to make it a habit.
Daily commands
| Command | What it does |
|---|---|
git submodule status | the recorded commit for each submodule; - not initialised, + differs from the pointer, U conflict |
git submodule update --init --recursive | check out the recorded commits, including nested submodules |
git submodule update --remote | move to the latest commit of the branch named in .gitmodules (branch = main) |
git submodule foreach '<cmd>' | run a command in each one ($name, $sha1, $path are available) |
git submodule sync | copy changed URLs from .gitmodules to your local config |
Removing a submodule
There is no single command. The sequence that worked:
git submodule deinit -f vendor/lib # unregister it
git rm -f vendor/lib # remove the path and edit .gitmodules
rm -rf .git/modules/vendor/lib # optional: delete the stored repository
After git rm, the test showed M .gitmodules and D vendor/lib, and the inner repository was still under .git/modules/ until deleted by hand.
Pitfalls
- Forgetting to update. After
git pullthe pointer moves but the files do not, and the build runs old code. Usesubmodule.recurseor a post-merge habit. - Pointers to unpushed commits. See above; use
--recurse-submodules=check. - Detached HEAD surprises. Commits made in a detached submodule belong to no branch; create or switch to a branch before committing.
- CI. The checkout step must fetch submodules (
--recurse-submodules, orsubmodules: recursivein GitHub Actions), and the CI credentials must be able to read the inner repositories. - Relative URLs.
url = ../libfollows the superproject's own remote, so a fork and the original both resolve correctly; absolute URLs pin one server. - The file protocol. Since Git 2.38.1, submodules cannot be cloned over the local
file://transport by default, a security fix (CVE-2022-39253). Local experiments needgit -c protocol.file.allow=always; real projects use ssh or https. - Merging pointer conflicts. If two branches bump the same submodule to different commits, Git reports a conflict on the path; pick the commit you want with
git add vendor/libafter checking it out.
3. Subtrees
A subtree copies another repository's files into a directory of yours and records where they came from. There is no pointer file and no second repository on disk: to everyone else the files are ordinary files.
$ git subtree add --prefix=third_party/lib2 ../lib main --squash
Added dir 'third_party/lib2'
$ git log --oneline | head -3
739a047 Merge commit 'eb86450...' as 'third_party/lib2'
eb86450 Squashed 'third_party/lib2/' content from commit 26f28f8
0de6faf bump lib
$ git subtree pull --prefix=third_party/lib2 ../lib main --squash
Subtree is already at commit 26f28f8eb38ce39ffc30a227ab1288d645feaff1.
--squash collapses the other project's history into one commit; leave it out to import the full history. git subtree push sends your local changes in that directory back to the upstream repository, which is the feature that makes subtrees useful for code you both consume and contribute to.
| Submodule | Subtree | |
|---|---|---|
| Others must run extra commands | yes (update --init) | no |
| Pinned to an exact upstream commit | yes, by design | by the merge commit you made |
| Repo size | small (pointer only) | contains the code (and history unless squashed) |
| Edit the dependency in place | possible, but two repos to push | easy, one repo; subtree push to upstream |
| Easy to see "what version is this" | git submodule status | read the squash commit message |
| Removing it | several steps | git rm -r the directory |
| History noise | none | merge and squash commits |
Rule of thumb: submodules when the dependency is its own project and people should treat it as read-only at a pinned version; subtrees when you want a plain checkout for everyone, or you rarely update it.
4. Worktrees
A worktree is another working directory attached to the same repository (same objects and branches). It replaces the habit of cloning the repo twice, or stashing, just to look at a second branch.
$ git worktree add ../app-wt -b feature/x
$ git worktree list
/work/app 739a047 [main]
/work/app-wt 739a047 [feature/x]
$ git worktree remove ../app-wt
Each worktree has its own checked-out branch, index and untracked files, and Git refuses to check out the same branch in two worktrees. Use them to run a long build on one branch while you fix a bug on another, to review a pull request without touching your work, or to compare two versions side by side. They are also a good fit for several AI-agent or CI jobs working on one repo.
5. Making big repositories manageable
Shallow clone: git clone --depth 1 <url> downloads only the latest commit. The test repository reported one commit and git rev-parse --is-shallow-repository returned true. It is ideal for CI builds that only need the files. Limits: history-based commands (log, blame, bisect, describe) see only what you have; git fetch --unshallow gets the rest.
Partial clone: git clone --filter=blob:none <url> downloads commits and trees but not file contents; Git fetches a blob the first time it needs it (checkout, diff, blame). You keep full history, with a small first download, and the remote must support filters (git config remote.origin.partialclonefilter showed blob:none). --filter=tree:0 skips trees too, for build servers that want only recent commits.
Sparse checkout: after the clone, choose which directories appear on disk.
$ git sparse-checkout set --cone docs
$ ls
app.txt docs
$ git sparse-checkout list
docs
Cone mode (the default in recent versions) is fast and keeps the files in the repository root plus the directories you list. Combine it with a partial clone for a monorepo: git clone --filter=blob:none --sparse <url> and then sparse-checkout set <dir>, so you download neither the history of other teams' files nor their contents.
6. Git LFS (large file storage)
Git stores every version of every file forever, so a 200 MB binary edited ten times costs about 2 GB in every clone. LFS replaces large files with a small text pointer in the repository, and stores the real content on a separate server that is downloaded on demand.
git lfs install # once per machine
git lfs track "*.psd" # writes the pattern into .gitattributes
git add .gitattributes design.psd
What Git stores for a tracked file is a pointer like:
version https://git-lfs.github.com/spec/v1
oid sha256:4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
size 12345
Things to know: you must run git lfs track before adding the files (otherwise they go in as normal blobs, and moving them later means rewriting history with git lfs migrate); the hosting service needs LFS support and often has storage and bandwidth quotas; and a clone without LFS installed gets only the pointer files. Source code, text and small images do not need it.
7. Choosing
Diagram: Which Git tool for sharing or scaling
8. Common mistakes
- Cloning without
--recurse-submodulesand wondering why the folder is empty. - Committing a submodule pointer before pushing the inner commit.
- Editing inside a detached-HEAD submodule and losing the commits when switching away.
- Using a submodule for something a package manager would handle better.
- Using
git rmon a submodule and forgetting.git/modulesand.gitmodules. - Using shallow clones in CI and then running tools that need history (
git describe, changelog generators). - Adding large binaries to Git before turning on LFS, then trying to remove them with a normal commit (they stay in history).
- Checking out the same branch in two worktrees (Git refuses) or deleting a worktree directory by hand instead of
git worktree remove(rungit worktree prune).
9. Things to mention in an interview
- A submodule is a gitlink (mode 160000) plus
.gitmodules: a pointer to an exact commit of another repository, checked out as a detached HEAD. clone --recurse-submodules,submodule update --init --recursive, andpush --recurse-submodules=check.- Submodule versus subtree versus package manager, with the trade-offs (workflow cost, pinning, history, size).
- Worktrees for parallel branches in one repository.
- Shallow, partial and sparse clones solve different problems: history depth, file contents, and files on disk.
- LFS stores pointers in Git and content on a server; configure it before adding files.
Quick recap
Submodule: pinned pointer to another repo, extra commands for everybody. Subtree: copied files, simple for others, noisier history. Worktree: many branches checked out from one repository. Shallow, partial and sparse: download less history, less content, fewer files. LFS: big binaries as pointers. And often the right answer is a package.
Sources
- Scott Chacon and Ben Straub, Pro Git: Git Tools - Submodules.
- The Git project, documentation for git-submodule, gitsubmodules, git-worktree, git-sparse-checkout and partial clone.
- The Git project, git subtree (in
contrib/) and git-config forprotocol.file.allow(the CVE-2022-39253 change in Git 2.38.1). - Git LFS and its specification.
Practise this topic: DevOps & Cloud