CS Fundamentals Trainer · Guides

Git submodules, subtrees, worktrees, sparse and partial clones, and LFS

What each Git tool for sharing code and handling big repositories really does, with real command output: submodules (gitlinks, .gitmodules, detached HEAD, update, push order, removal, pitfalls), subtrees, worktrees, shallow and partial clones, sparse checkout, Git LFS, and a decision chart that includes the option of a package manager.

12 min read

Sooner or later one repository needs code that lives in another: a shared library, a vendored dependency, a design system used by five apps. Or the opposite problem appears: a repository grows so large that cloning it takes minutes. Git has several tools for these cases, and they are often confused: submodules, subtrees, worktrees, shallow and partial clones, sparse checkout and Git LFS. This guide explains each one by what it really does underneath, shows the commands with the output of real runs (Git 2.43), and ends with a table for choosing. It assumes you know commits and branches; the object model is in the guide "Git internals and CI/CD pipelines".

1. The problem and the options

NeedToolIdea
Use another project's code at an exact version, keep it as a separate repoSubmodulethe outer repo records a commit id of the inner repo
Copy another project's code into yours, with its history, no special workflow for othersSubtreethe files become ordinary files in your repo
Several branches checked out at onceWorktreeone repository, many working directories
Clone faster, with less historyShallow cloneonly the last N commits
Clone faster, fewer files' contentsPartial cloneblobs fetched when needed
Work on a slice of a big treeSparse checkoutonly some directories on disk
Big binary filesGit LFSthe repo stores small pointers; the content lives elsewhere
Shared libraries with versionsA package managernot Git at all: publish to a registry (often the best answer)

Before reaching for Git, ask whether a package (npm, Go modules, PyPI) would do. Submodules and subtrees are for the cases where it would not: code that is not published, that you change together with the main project, or that must be pinned to an exact commit.

2. Submodules

What a submodule really is

A submodule is a pointer from one repository to a specific commit of another. The outer repository (the superproject) does not contain the inner files. It contains two things:

  1. A tree entry of mode 160000 (a "gitlink") that stores the inner commit id.
  2. A file .gitmodules that says where to clone it from.

This is what git ls-tree -r HEAD printed after adding one:

100644 blob 9b6e268...  .gitmodules
100644 blob b80f0bd...  app.txt
160000 commit 26f28f8...  vendor/lib

and .gitmodules:

[submodule "vendor/lib"]
	path = vendor/lib
	url = ../lib

The inner repository is a normal Git repository; its data is kept under .git/modules/ of the superproject, and the working directory vendor/lib has a .git file pointing there.

Adding and cloning

$ git submodule add ../lib vendor/lib
$ git status --short
A  .gitmodules
A  vendor/lib
$ git commit -m "add lib"

When someone clones the superproject, the submodule directory is empty: Git fetched the pointer but not the files. In the test, the plain clone had 0 files in vendor/lib and git submodule status showed a leading - (not initialised). Two ways to get the content:

git clone --recurse-submodules <url>          # all at once
# or, after a plain clone:
git submodule update --init --recursive

After this the submodule is on a detached HEAD at exactly the recorded commit ("HEAD detached at c30e138"). That is by design: the superproject says "I need this commit", not "I need this branch". Setting git config submodule.recurse true makes pull, checkout and other commands also update submodules; without it you must run git submodule update after switching branches or pulling, or the inner files stay at the old commit.

Changing the inner project

To move the pointer forward, go inside, update, and commit the new pointer in the outer repo:

$ cd vendor/lib && git pull origin main     # the inner repo gets lib 3
$ cd ../..
$ git status --short
 M vendor/lib
$ git diff --submodule
Submodule vendor/lib c30e138..26f28f8:
  > lib 3
$ git add vendor/lib && git commit -m "bump lib"

Notice that the outer repo sees only "the pointer changed". git diff --submodule (or git config diff.submodule log) shows the inner commits. If you edit files inside the submodule, you must commit and push the inner repository first, then commit the pointer in the outer repository. If you commit the pointer and push only the outer repo, everyone else gets a pointer to a commit that does not exist on the server, and their update fails. git push --recurse-submodules=check refuses to push in that situation, and =on-demand pushes the inner ones for you. Set push.recurseSubmodules=check to make it a habit.

Daily commands

CommandWhat it does
git submodule statusthe recorded commit for each submodule; - not initialised, + differs from the pointer, U conflict
git submodule update --init --recursivecheck out the recorded commits, including nested submodules
git submodule update --remotemove to the latest commit of the branch named in .gitmodules (branch = main)
git submodule foreach '<cmd>'run a command in each one ($name, $sha1, $path are available)
git submodule synccopy changed URLs from .gitmodules to your local config

Removing a submodule

There is no single command. The sequence that worked:

git submodule deinit -f vendor/lib      # unregister it
git rm -f vendor/lib                    # remove the path and edit .gitmodules
rm -rf .git/modules/vendor/lib          # optional: delete the stored repository

After git rm, the test showed M .gitmodules and D vendor/lib, and the inner repository was still under .git/modules/ until deleted by hand.

Pitfalls

3. Subtrees

A subtree copies another repository's files into a directory of yours and records where they came from. There is no pointer file and no second repository on disk: to everyone else the files are ordinary files.

$ git subtree add --prefix=third_party/lib2 ../lib main --squash
Added dir 'third_party/lib2'
$ git log --oneline | head -3
739a047 Merge commit 'eb86450...' as 'third_party/lib2'
eb86450 Squashed 'third_party/lib2/' content from commit 26f28f8
0de6faf bump lib
$ git subtree pull --prefix=third_party/lib2 ../lib main --squash
Subtree is already at commit 26f28f8eb38ce39ffc30a227ab1288d645feaff1.

--squash collapses the other project's history into one commit; leave it out to import the full history. git subtree push sends your local changes in that directory back to the upstream repository, which is the feature that makes subtrees useful for code you both consume and contribute to.

SubmoduleSubtree
Others must run extra commandsyes (update --init)no
Pinned to an exact upstream commityes, by designby the merge commit you made
Repo sizesmall (pointer only)contains the code (and history unless squashed)
Edit the dependency in placepossible, but two repos to pusheasy, one repo; subtree push to upstream
Easy to see "what version is this"git submodule statusread the squash commit message
Removing itseveral stepsgit rm -r the directory
History noisenonemerge and squash commits

Rule of thumb: submodules when the dependency is its own project and people should treat it as read-only at a pinned version; subtrees when you want a plain checkout for everyone, or you rarely update it.

4. Worktrees

A worktree is another working directory attached to the same repository (same objects and branches). It replaces the habit of cloning the repo twice, or stashing, just to look at a second branch.

$ git worktree add ../app-wt -b feature/x
$ git worktree list
/work/app     739a047 [main]
/work/app-wt  739a047 [feature/x]
$ git worktree remove ../app-wt

Each worktree has its own checked-out branch, index and untracked files, and Git refuses to check out the same branch in two worktrees. Use them to run a long build on one branch while you fix a bug on another, to review a pull request without touching your work, or to compare two versions side by side. They are also a good fit for several AI-agent or CI jobs working on one repo.

5. Making big repositories manageable

Shallow clone: git clone --depth 1 <url> downloads only the latest commit. The test repository reported one commit and git rev-parse --is-shallow-repository returned true. It is ideal for CI builds that only need the files. Limits: history-based commands (log, blame, bisect, describe) see only what you have; git fetch --unshallow gets the rest.

Partial clone: git clone --filter=blob:none <url> downloads commits and trees but not file contents; Git fetches a blob the first time it needs it (checkout, diff, blame). You keep full history, with a small first download, and the remote must support filters (git config remote.origin.partialclonefilter showed blob:none). --filter=tree:0 skips trees too, for build servers that want only recent commits.

Sparse checkout: after the clone, choose which directories appear on disk.

$ git sparse-checkout set --cone docs
$ ls
app.txt  docs
$ git sparse-checkout list
docs

Cone mode (the default in recent versions) is fast and keeps the files in the repository root plus the directories you list. Combine it with a partial clone for a monorepo: git clone --filter=blob:none --sparse <url> and then sparse-checkout set <dir>, so you download neither the history of other teams' files nor their contents.

6. Git LFS (large file storage)

Git stores every version of every file forever, so a 200 MB binary edited ten times costs about 2 GB in every clone. LFS replaces large files with a small text pointer in the repository, and stores the real content on a separate server that is downloaded on demand.

git lfs install                  # once per machine
git lfs track "*.psd"            # writes the pattern into .gitattributes
git add .gitattributes design.psd

What Git stores for a tracked file is a pointer like:

version https://git-lfs.github.com/spec/v1
oid sha256:4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
size 12345

Things to know: you must run git lfs track before adding the files (otherwise they go in as normal blobs, and moving them later means rewriting history with git lfs migrate); the hosting service needs LFS support and often has storage and bandwidth quotas; and a clone without LFS installed gets only the pointer files. Source code, text and small images do not need it.

7. Choosing

Diagram: Which Git tool for sharing or scaling

8. Common mistakes

  1. Cloning without --recurse-submodules and wondering why the folder is empty.
  2. Committing a submodule pointer before pushing the inner commit.
  3. Editing inside a detached-HEAD submodule and losing the commits when switching away.
  4. Using a submodule for something a package manager would handle better.
  5. Using git rm on a submodule and forgetting .git/modules and .gitmodules.
  6. Using shallow clones in CI and then running tools that need history (git describe, changelog generators).
  7. Adding large binaries to Git before turning on LFS, then trying to remove them with a normal commit (they stay in history).
  8. Checking out the same branch in two worktrees (Git refuses) or deleting a worktree directory by hand instead of git worktree remove (run git worktree prune).

9. Things to mention in an interview

Quick recap

Submodule: pinned pointer to another repo, extra commands for everybody. Subtree: copied files, simple for others, noisier history. Worktree: many branches checked out from one repository. Shallow, partial and sparse: download less history, less content, fewer files. LFS: big binaries as pointers. And often the right answer is a package.

Sources

Practise this topic: DevOps & Cloud

More guides