User Tools

Site Tools


git_submodules

This is an old revision of the document!


Overview

The good and the bad of submodules, using Wind River's Pulsar Linux as a convenient example.

SYNOPSIS

git submodule [--quiet] add [<options>] [--] <repository> [<path>]
git submodule [--quiet] status [--cached] [--recursive] [--] [<path>...]
git submodule [--quiet] init [--] [<path>...]
git submodule [--quiet] deinit [-f|--force] (--all|[--] <path>...)
git submodule [--quiet] update [<options>] [--] [<path>...]
git submodule [--quiet] summary [<options>] [--] [<path>...]
git submodule [--quiet] foreach [--recursive] <command>
git submodule [--quiet] sync [--recursive] [--] [<path>...]
git submodule [--quiet] absorbgitdirs [--] [<path>...]

Cloning a project with submodules

The basic (non-recursive) clone

Example:

$ git clone -b pulsar-x -o wr-core \
    https://github.com/WindRiver-OpenSourceLabs/wr-core
$ cd wr-core
$ ls -F
docs/                       install_templates/  scripts/
init-intel-x86-env*         layers/             todo/
init-intel-x86-secure-env*  overc-installer/
init-raspberrypi-env*       README.TXT
$

At this point, the submodules are not checked out yet, so this is not taking up a lot of space:

$ du -ks .
1228
$

and there's nothing under the layers/ subdirectories:

$ tree -F layers
layers
├── bitbake/
├── external-binaries/
├── fsl-ls10xx/
├── intel-iot-refkit/
├── meta-browser/
├── meta-cloud-services/
├── meta-dpdk/
├── meta-flatpak/
├── meta-gateway/
├── meta-intel/
├── meta-iot-cloud/
├── meta-java/
├── meta-openembedded/
├── meta-overc/
├── meta-raspberrypi/
├── meta-secure-core/
├── meta-security/
├── meta-up-board/
├── meta-virtualization/
├── oe-core/
├── oe-meta-go/
├── qemuarm/
├── qemuarma9/
├── vendor-proprietary/
├── wrlabs-integration/
└── xilinx-zynq/

26 directories, 0 files
$

The .gitmodules file

The repository comes with a versioned .gitmodules file, identifying where to find the submodules (relative or absolute URLs) and where to clone them locally:

$ cat .gitmodules
[submodule "bitbake"]
        path = layers/bitbake
        url = ../bitbake
        branch = master
[submodule "oe-core"]
        path = layers/oe-core
        url = ../oe-core
        branch = master
[submodule "meta-intel"]
        path = layers/meta-intel
        url = ../meta-intel
        branch = master
... snip ...
[submodule "meta-raspberrypi"]
        path = layers/meta-raspberrypi
        url = git://git.yoctoproject.org/meta-raspberrypi
        branch = master 
... snip ...

The .git/config file

So far, there's nothing in the repository's config file about submodules:

$ cat .git/config 
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "wr-core"]
	url = https://github.com/WindRiver-OpenSourceLabs/wr-core
	fetch = +refs/heads/*:refs/remotes/wr-core/*
[branch "pulsar-x"]
	remote = wr-core
	merge = refs/heads/pulsar-x
$

git submodule init

Initialize the submodules by setting all of the submodule.$name.url lines in .git/config:

$ git submodule init
Submodule 'bitbake' (https://github.com/WindRiver-OpenSourceLabs/bitbake) registered for path 'layers/bitbake'
Submodule 'external-binaries' (https://github.com/WindRiver-OpenSourceLabs/external-binaries) registered for path 'layers/external-binaries'
Submodule 'fsl-ls10xx' (https://github.com/WindRiver-OpenSourceLabs/fsl-ls10xx) registered for path 'layers/fsl-ls10xx'

... etc etc ...

We're still not using any more space:

$ du -ks .
1228
$

but the repository's .git/config file now knows about the submodules:

$ cat .git/config

... snip existing earlier content ...

[submodule "bitbake"]
	active = true
	url = https://github.com/WindRiver-OpenSourceLabs/bitbake
[submodule "external-binaries"]
	active = true
	url = https://github.com/WindRiver-OpenSourceLabs/external-binaries
[submodule "fsl-ls10xx"]
	active = true
	url = https://github.com/WindRiver-OpenSourceLabs/fsl-ls10xx
... etc etc ...

git submodule update

Update all the initialized submodules:

$ du -ks layers/
108	layers
$
$ git submodule update
Cloning into '/home/rpjday/github/wr_open_source_labs/wr-core/layers/bitbake'...
Cloning into '/home/rpjday/github/wr_open_source_labs/wr-core/layers/external-binaries'...
Cloning into '/home/rpjday/github/wr_open_source_labs/wr-core/layers/meta-flatpak'...
... etc etc ...

Now we have the submodules:

$ du -ks layers/
153620	layers
$ du -ks .
449876
$

Abbreviating the above commands

One option:

$ git clone ...
$ cd ...
$ git submodule update --init --recursive

or simply, all in one step:

$ git clone --recurse-submodules ...

File structure of submodules

Using bitbake as an example, the working tree is under layers/bitbake:

$ cd layers/bitbake
$ ls -F
AUTHORS    conf/     HEADER       README
bin/       contrib/  lib/         toaster-requirements.txt
ChangeLog  COPYING   LICENSE      TODO
classes/   doc/      MANIFEST.in
$

However, here is the .git file:

$ cat .git
gitdir: ../../.git/modules/bitbake
$

The submodule Git directories themselvesare stored under .git/modules/:

$ ls -F .git/modules/
bitbake/              meta-intel/         meta-virtualization/
external-binaries/    meta-iot-cloud/     oe-core/
fsl-ls10xx/           meta-java/          oe-meta-go/
intel-iot-refkit/     meta-openembedded/  overc-installer/
layers/               meta-overc/         qemuarm/
meta-browser/         meta-raspberrypi/   qemuarma9/
meta-cloud-services/  meta-secure-core/   vendor-proprietary/
meta-dpdk/            meta-security/      wrlabs-integration/
meta-gateway/         meta-up-board/      xilinx-zynq/
$
$ ls -F .git/modules/bitbake/
branches/  description  hooks/  info/  objects/     refs/
config     HEAD         index   logs/  packed-refs
$

Checking status of submodules

$ git submodule status
 055865047c63b9c3b213b47a1884924ce0adeda0 layers/bitbake (1.38.0-10-g055865047)
 78d4168b29efe5e812c36502a1c300d3d4c45935 layers/external-binaries (78d4168)
 0a8c797caa70f517c327c67b15b0d55e902b7c44 layers/fsl-ls10xx (remotes/origin/pulsar-8)
 4ca2e7474b046fa0bd7813cd6da7cf455733d61c layers/intel-iot-refkit (2017.07-258-g4ca2e74)
 1cd38d701a49eade80a04140f70d3383117b9745 layers/meta-browser (remotes/origin/krogoth-266-g1cd38d7)
 31e3086409b5055b8ce5962a614f1284003c513d layers/meta-cloud-services (31e3086)
 8a0f847836671111915088d2838e3ff6f3c7642c layers/meta-dpdk (tmpfix-20180615)
 6975858540552012d8082c74b5c5d9db8e260865 layers/meta-flatpak (6975858)
 48dae0603506cff9bde674755d963e5cf3e75244 layers/meta-gateway (48dae06)
 fec5d45ec3978d4b7c9649bee45fe58f914bcee4 layers/meta-intel (9.0-sumo-2.5-8-gfec5d45e)
 6039a7a8a9a0cad4e32736be3a05a529b65e126a layers/meta-iot-cloud (remotes/origin/pulsar-x)
 aa276b00d8a6bbffffd59d1980153c99521ce6ea layers/meta-java (tmpfix-20180615)
 a4056a96ecc857b1d675b89dc35cf9ed8a876f80 layers/meta-openembedded (tmpfix-20180618)
 62f03f7d42f784923a8b02ce9d30d62d6552a22b layers/meta-overc (tmpfix-20180615)
 b2da4618b0ac2ad7dbc26387cad4c03796f8a06e layers/meta-raspberrypi (remotes/origin/krogoth-354-gb2da461)
 fb838242ad1b5c7caf5104cb3560dd6e655f1d92 layers/meta-secure-core (heads/master)
 f9c5e2022b54097474e46d9efb54080bd0e4d606 layers/meta-security (remotes/origin/pulsar-8-186-gf9c5e20)
 024d0f5e404cb29789c2bba90aa72efd00f2a0e6 layers/meta-up-board (remotes/origin/pulsar-x)
 8344dd9e9f55a5ea81f9c57c74bd9a8b9a04e8ad layers/meta-virtualization (tmpfix-20180615)
 b34e86b4ee13d53f09d558e613d5b66c845edde6 layers/oe-core (uninative-2.1-109-gb34e86b4ee)
 b29b384eae4a40933d07cc64c51e890c289d9621 layers/oe-meta-go (heads/ovp7-20150902)
 3998487c37639a484fde00fd66121f7660e06c56 layers/qemuarm (remotes/origin/LB13_8.0_RCPL0001)
 d902f0eeec53662e30cc1ab98c5f296496d29086 layers/qemuarma9 (remotes/origin/master)
 032d6a6f94297707acd228c96313a603c5702a38 layers/vendor-proprietary (032d6a6)
 02e0f675db192294044ffbce6146ec25f34ffdba layers/wrlabs-integration (remotes/origin/master-oci-fli-100-g02e0f67)
 cd816cf2bfcf0e917e02f4cbd48800ee5a583f33 layers/xilinx-zynq (remotes/origin/pulsar-8)
 aabcd6ab2b69383ba4b41b7bebfc600dd3b166dd overc-installer (remotes/origin/pulsar-7.0-224-gaabcd6a)
$

Adding submodules (IN PROGRESS)

Adding a submodule -- Part 1

Before adding

$ git status
On branch master
Your branch is up-to-date with 'origin/master'.

nothing to commit, working tree clean
$

Adding a submodule

$ git submodule add https://github.com/boostorg/coroutine2
Cloning into '/home/rpjday/ebooks/progit/pg2/coroutine2'...
remote: Counting objects: 1161, done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 1161 (delta 6), reused 16 (delta 6), pack-reused 1140
Receiving objects: 100% (1161/1161), 289.57 KiB | 969.00 KiB/s, done.
Resolving deltas: 100% (690/690), done.
$

The end result

Note the submodule is already staged, ready for commit:

$ git status
On branch master
Your branch is up-to-date with 'origin/master'.

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

	new file:   .gitmodules
	new file:   coroutine2       <-- file??

$

What's there?

$ tree coroutine2/
coroutine2/
├── doc
│   ├── acknowledgements.qbk
│   ├── architectures.qbk
│   ├── asymmetric.qbk
│   ├── coro.qbk
│   ├── coroutine.qbk
│   ├── images
│   │   ├── event_model.dia
... etc etc ...

Adding a submodule -- what just happened?

The new .gitmodules file

The new .gitmodules file, which is version-controlled just like your other files so other developers can get the submodules when they clone the “superproject”:

[submodule "coroutine2"]
	path = coroutine2
	url = https://github.com/boostorg/coroutine2

git diff

There is no current git diff since adding the submodule has already been staged:

$ git diff
$

but examine what's been staged:

$ git diff --staged
diff --git a/.gitmodules b/.gitmodules
new file mode 100644
index 0000000..d0564c5
--- /dev/null
+++ b/.gitmodules
@@ -0,0 +1,3 @@
+[submodule "coroutine2"]
+       path = coroutine2
+       url = https://github.com/boostorg/coroutine2
diff --git a/coroutine2 b/coroutine2
new file mode 160000
index 0000000..fc3cb67
--- /dev/null
+++ b/coroutine2
@@ -0,0 +1 @@
+Subproject commit fc3cb675279708a48eb31ad2d53461b0ed4c5540
$

You can also use the --submodule option:

$ git diff --cached --submodule
diff --git a/.gitmodules b/.gitmodules
new file mode 100644
index 0000000..d0564c5
--- /dev/null
+++ b/.gitmodules
@@ -0,0 +1,3 @@
+[submodule "coroutine2"]
+       path = coroutine2
+       url = https://github.com/boostorg/coroutine2
Submodule coroutine2 0000000...fc3cb67 (new submodule)
$

Committing the submodule

$ git commit -am "added coroutine2 submodule"
[master bec4c94] added coroutine2 submodule
 2 files changed, 4 insertions(+)
 create mode 100644 .gitmodules
 create mode 160000 coroutine2     <-- weird mode for submodule
$

at which point you can push the result:

$ git push origin master

Under the hood

The submodule directory has a file named .git, not a directory, which refers to submodule information in the superproject:

$ cat coroutine2/.git
gitdir: ../.git/modules/coroutine2
$

From the superproject, you can see the information for each submodule:

$ ls -l .git/modules/coroutine2/
total 48
drwxrwxr-x. 2 rpjday rpjday 4096 Feb 13 15:37 branches
-rw-rw-r--. 1 rpjday rpjday  297 Feb 13 15:37 config
-rw-rw-r--. 1 rpjday rpjday   73 Feb 13 15:37 description
-rw-rw-r--. 1 rpjday rpjday   24 Feb 13 15:37 HEAD
drwxrwxr-x. 2 rpjday rpjday 4096 Feb 13 15:37 hooks
-rw-rw-r--. 1 rpjday rpjday 7299 Feb 13 15:38 index
drwxrwxr-x. 2 rpjday rpjday 4096 Feb 13 15:37 info
drwxrwxr-x. 3 rpjday rpjday 4096 Feb 13 15:37 logs
drwxrwxr-x. 4 rpjday rpjday 4096 Feb 13 15:37 objects
-rw-rw-r--. 1 rpjday rpjday 1174 Feb 13 15:37 packed-refs
drwxrwxr-x. 5 rpjday rpjday 4096 Feb 13 15:37 refs
$

Cloning a project with submodules

If you don't want to populate the submodule directories:

$ git clone https://github.com/boostorg/boost

If you want to populate the submodule diredctories:

$ git clone --recurse-submodules https://github.com/boostorg/boost

Populate all submodules after the fact:

$ git submodule update --init --recursive
git_submodules.1551277217.txt.gz · Last modified: 2019/02/27 14:20 by rpjday