Doku Wiki

The good and the bad of submodules, using Wind River's Pulsar Linux as a convenient example.

Stuff to add:

• git config submodule.<what>.url PRIVATE_URL

• man page
• Pro Git -- Submodules
• GitHub blog post
• Atlassian -- Git submodules: core concept, workflows, and tips
• Mastering Git submodules (Porteneuve, 2015)
• Swap git submodule with own fork
• Why your company shouldn’t use Git submodules

git submodule [--quiet] add [<options>] [--] <repository> [<path>]
git submodule [--quiet] status [--cached] [--recursive] [--] [<path>...]
git submodule [--quiet] init [--] [<path>...]
git submodule [--quiet] deinit [-f|--force] (--all|[--] <path>...)
git submodule [--quiet] update [<options>] [--] [<path>...]
git submodule [--quiet] summary [<options>] [--] [<path>...]
git submodule [--quiet] foreach [--recursive] <command>
git submodule [--quiet] sync [--recursive] [--] [<path>...]
git submodule [--quiet] absorbgitdirs [--] [<path>...]

Example:

$ git clone -b pulsar-x -o wr-core \
    https://github.com/WindRiver-OpenSourceLabs/wr-core
$ cd wr-core
$ ls -F
docs/                       install_templates/  scripts/
init-intel-x86-env*         layers/             todo/
init-intel-x86-secure-env*  overc-installer/
init-raspberrypi-env*       README.TXT
$

At this point, the submodules are not checked out yet, so this is not taking up a lot of space:

$ du -ks .
1228
$

and there's nothing under the layers/ subdirectories:

$ tree -F layers
layers
├── bitbake/
├── external-binaries/
├── fsl-ls10xx/
├── intel-iot-refkit/
├── meta-browser/
├── meta-cloud-services/
├── meta-dpdk/
├── meta-flatpak/
├── meta-gateway/
├── meta-intel/
├── meta-iot-cloud/
├── meta-java/
├── meta-openembedded/
├── meta-overc/
├── meta-raspberrypi/
├── meta-secure-core/
├── meta-security/
├── meta-up-board/
├── meta-virtualization/
├── oe-core/
├── oe-meta-go/
├── qemuarm/
├── qemuarma9/
├── vendor-proprietary/
├── wrlabs-integration/
└── xilinx-zynq/

26 directories, 0 files
$

The repository comes with a versioned .gitmodules file, identifying where to find the submodules (relative or absolute URLs) and where to clone them locally:

$ cat .gitmodules
[submodule "bitbake"]
        path = layers/bitbake
        url = ../bitbake
        branch = master
[submodule "oe-core"]
        path = layers/oe-core
        url = ../oe-core
        branch = master
[submodule "meta-intel"]
        path = layers/meta-intel
        url = ../meta-intel
        branch = master
... snip ...
[submodule "meta-raspberrypi"]
        path = layers/meta-raspberrypi
        url = git://git.yoctoproject.org/meta-raspberrypi
        branch = master 
... snip ...

So far, there's nothing in the repository's config file about submodules:

$ cat .git/config 
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "wr-core"]
	url = https://github.com/WindRiver-OpenSourceLabs/wr-core
	fetch = +refs/heads/*:refs/remotes/wr-core/*
[branch "pulsar-x"]
	remote = wr-core
	merge = refs/heads/pulsar-x
$

Initialize the submodules by setting all of the submodule.$name.url lines in .git/config:

$ git submodule init
Submodule 'bitbake' (https://github.com/WindRiver-OpenSourceLabs/bitbake) registered for path 'layers/bitbake'
Submodule 'external-binaries' (https://github.com/WindRiver-OpenSourceLabs/external-binaries) registered for path 'layers/external-binaries'
Submodule 'fsl-ls10xx' (https://github.com/WindRiver-OpenSourceLabs/fsl-ls10xx) registered for path 'layers/fsl-ls10xx'

... etc etc ...

We're still not using any more space:

$ du -ks .
1228
$

but the repository's .git/config file now knows about the submodules:

$ cat .git/config

... snip existing earlier content ...

[submodule "bitbake"]
	active = true
	url = https://github.com/WindRiver-OpenSourceLabs/bitbake
[submodule "external-binaries"]
	active = true
	url = https://github.com/WindRiver-OpenSourceLabs/external-binaries
[submodule "fsl-ls10xx"]
	active = true
	url = https://github.com/WindRiver-OpenSourceLabs/fsl-ls10xx
... etc etc ...

Update all the initialized submodules:

$ du -ks layers/
108	layers
$

$ git submodule update
Cloning into '/home/rpjday/github/wr_open_source_labs/wr-core/layers/bitbake'...
Cloning into '/home/rpjday/github/wr_open_source_labs/wr-core/layers/external-binaries'...
Cloning into '/home/rpjday/github/wr_open_source_labs/wr-core/layers/meta-flatpak'...
... etc etc ...

Now we have the submodules:

$ du -ks layers/
153620	layers
$ du -ks .
449876
$

One option:

$ git clone ...
$ cd ...
$ git submodule update --init --recursive

or simply, all in one step:

$ git clone --recurse-submodules ...

Using bitbake as an example, the working tree is under layers/bitbake:

$ cd layers/bitbake
$ ls -F
AUTHORS    conf/     HEADER       README
bin/       contrib/  lib/         toaster-requirements.txt
ChangeLog  COPYING   LICENSE      TODO
classes/   doc/      MANIFEST.in
$

However, here is the .git file:

$ cat .git
gitdir: ../../.git/modules/bitbake
$

The submodule Git directories themselvesare stored under .git/modules/:

$ ls -F .git/modules/
bitbake/              meta-intel/         meta-virtualization/
external-binaries/    meta-iot-cloud/     oe-core/
fsl-ls10xx/           meta-java/          oe-meta-go/
intel-iot-refkit/     meta-openembedded/  overc-installer/
layers/               meta-overc/         qemuarm/
meta-browser/         meta-raspberrypi/   qemuarma9/
meta-cloud-services/  meta-secure-core/   vendor-proprietary/
meta-dpdk/            meta-security/      wrlabs-integration/
meta-gateway/         meta-up-board/      xilinx-zynq/
$

$ ls -F .git/modules/bitbake/
branches/  description  hooks/  info/  objects/     refs/
config     HEAD         index   logs/  packed-refs
$

$ git submodule status
 055865047c63b9c3b213b47a1884924ce0adeda0 layers/bitbake (1.38.0-10-g055865047)
 78d4168b29efe5e812c36502a1c300d3d4c45935 layers/external-binaries (78d4168)
 0a8c797caa70f517c327c67b15b0d55e902b7c44 layers/fsl-ls10xx (remotes/origin/pulsar-8)
 4ca2e7474b046fa0bd7813cd6da7cf455733d61c layers/intel-iot-refkit (2017.07-258-g4ca2e74)
 1cd38d701a49eade80a04140f70d3383117b9745 layers/meta-browser (remotes/origin/krogoth-266-g1cd38d7)
 31e3086409b5055b8ce5962a614f1284003c513d layers/meta-cloud-services (31e3086)
 8a0f847836671111915088d2838e3ff6f3c7642c layers/meta-dpdk (tmpfix-20180615)
 6975858540552012d8082c74b5c5d9db8e260865 layers/meta-flatpak (6975858)
 48dae0603506cff9bde674755d963e5cf3e75244 layers/meta-gateway (48dae06)
 fec5d45ec3978d4b7c9649bee45fe58f914bcee4 layers/meta-intel (9.0-sumo-2.5-8-gfec5d45e)
 6039a7a8a9a0cad4e32736be3a05a529b65e126a layers/meta-iot-cloud (remotes/origin/pulsar-x)
 aa276b00d8a6bbffffd59d1980153c99521ce6ea layers/meta-java (tmpfix-20180615)
 a4056a96ecc857b1d675b89dc35cf9ed8a876f80 layers/meta-openembedded (tmpfix-20180618)
 62f03f7d42f784923a8b02ce9d30d62d6552a22b layers/meta-overc (tmpfix-20180615)
 b2da4618b0ac2ad7dbc26387cad4c03796f8a06e layers/meta-raspberrypi (remotes/origin/krogoth-354-gb2da461)
 fb838242ad1b5c7caf5104cb3560dd6e655f1d92 layers/meta-secure-core (heads/master)
 f9c5e2022b54097474e46d9efb54080bd0e4d606 layers/meta-security (remotes/origin/pulsar-8-186-gf9c5e20)
 024d0f5e404cb29789c2bba90aa72efd00f2a0e6 layers/meta-up-board (remotes/origin/pulsar-x)
 8344dd9e9f55a5ea81f9c57c74bd9a8b9a04e8ad layers/meta-virtualization (tmpfix-20180615)
 b34e86b4ee13d53f09d558e613d5b66c845edde6 layers/oe-core (uninative-2.1-109-gb34e86b4ee)
 b29b384eae4a40933d07cc64c51e890c289d9621 layers/oe-meta-go (heads/ovp7-20150902)
 3998487c37639a484fde00fd66121f7660e06c56 layers/qemuarm (remotes/origin/LB13_8.0_RCPL0001)
 d902f0eeec53662e30cc1ab98c5f296496d29086 layers/qemuarma9 (remotes/origin/master)
 032d6a6f94297707acd228c96313a603c5702a38 layers/vendor-proprietary (032d6a6)
 02e0f675db192294044ffbce6146ec25f34ffdba layers/wrlabs-integration (remotes/origin/master-oci-fli-100-g02e0f67)
 cd816cf2bfcf0e917e02f4cbd48800ee5a583f33 layers/xilinx-zynq (remotes/origin/pulsar-8)
 aabcd6ab2b69383ba4b41b7bebfc600dd3b166dd overc-installer (remotes/origin/pulsar-7.0-224-gaabcd6a)
$

Add a submodule to the repo for the “Pro Git” book, which is clean and has no current submodules:

$ git status
On branch master
Your branch is up to date with 'rpjday/master'.
nothing to commit, working tree clean
$

$ git submodule status
$

$ git submodule add https://github.com/boostorg/coroutine2
Cloning into '/home/rpjday/ebooks/progit/progit2/coroutine2'...
remote: Enumerating objects: 2, done.
remote: Counting objects: 100% (2/2), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 1189 (delta 0), reused 1 (delta 0), pack-reused 1187
Receiving objects: 100% (1189/1189), 292.30 KiB | 1.82 MiB/s, done.
Resolving deltas: 100% (703/703), done.
$

Note that this has not committed the new submodule yet. So what has it done?

$ git status
On branch master
Your branch is up to date with 'rpjday/master'.

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

	new file:   .gitmodules
	new file:   coroutine2

$

$ git submodule status
 0233d35081de5b669c60ef8ec5a53854e1b2a577 coroutine2 (heads/develop)
$

The new .gitmodules file:

$ cat .gitmodules 
[submodule "coroutine2"]
	path = coroutine2
	url = https://github.com/boostorg/coroutine2

The content of the new submodule working tree:

$ tree coroutine2/
coroutine2/
├── doc
│   ├── acknowledgements.qbk
│   ├── architectures.qbk
│   ├── asymmetric.qbk
│   ├── coro.qbk
│   ├── coroutine.qbk
│   ├── images
│   │   ├── event_model.dia
│   │   ├── event_model.png
... etc etc ...

The content of the new submodule's Git directory:

$ ls -lF .git/modules/coroutine2/
total 48
drwxrwxr-x. 2 rpjday rpjday 4096 Feb 27 09:31 branches/
-rw-rw-r--. 1 rpjday rpjday  297 Feb 27 09:31 config
-rw-rw-r--. 1 rpjday rpjday   73 Feb 27 09:31 description
-rw-rw-r--. 1 rpjday rpjday   24 Feb 27 09:31 HEAD
drwxrwxr-x. 2 rpjday rpjday 4096 Feb 27 09:31 hooks/
-rw-rw-r--. 1 rpjday rpjday 7299 Feb 27 09:33 index
drwxrwxr-x. 2 rpjday rpjday 4096 Feb 27 09:31 info/
drwxrwxr-x. 3 rpjday rpjday 4096 Feb 27 09:31 logs/
drwxrwxr-x. 4 rpjday rpjday 4096 Feb 27 09:31 objects/
-rw-rw-r--. 1 rpjday rpjday 1161 Feb 27 09:31 packed-refs
drwxrwxr-x. 5 rpjday rpjday 4096 Feb 27 09:31 refs/
$

$ git diff --cached
diff --git a/.gitmodules b/.gitmodules
new file mode 100644
index 0000000..d0564c5
--- /dev/null
+++ b/.gitmodules
@@ -0,0 +1,3 @@
+[submodule "coroutine2"]
+       path = coroutine2
+       url = https://github.com/boostorg/coroutine2
diff --git a/coroutine2 b/coroutine2
new file mode 160000
index 0000000..0233d35
--- /dev/null
+++ b/coroutine2
@@ -0,0 +1 @@
+Subproject commit 0233d35081de5b669c60ef8ec5a53854e1b2a577
$

Abbreviated form:

$ git diff --cached --submodule
diff --git a/.gitmodules b/.gitmodules
new file mode 100644
index 0000000..d0564c5
--- /dev/null
+++ b/.gitmodules
@@ -0,0 +1,3 @@
+[submodule "coroutine2"]
+       path = coroutine2
+       url = https://github.com/boostorg/coroutine2
Submodule coroutine2 0000000...0233d35 (new submodule)
$

$ git commit -a -m "added coroutine2 submodule"
[master a7561cb] added coroutine2 submodule
 2 files changed, 4 insertions(+)
 create mode 100644 .gitmodules
 create mode 160000 coroutine2    <--- weird mode representing submodule
$

at which point you can push the result:

$ git push origin master