Lesson 6: Using git and working with custom kernels

Printer-friendly versionPrinter-friendly version

A few more issues regarding building your own modules

For the sake of tidying up some loose ends and pushing the envelope on what you can do in terms of building and loading your own modules, we're going to use one more lesson to cover the mechanics of compiling your own modules and show how that ties into the kernel source tree, your git checkout and how to tweak the kernel source itself (a topic we'll cover in far more detail in a later lesson).

For the rest of this lesson, we'll assume that you've covered all earlier lessons and that they worked for you. If you haven't (or they didn't), now is a good time to go back and work through it all until you get back here.

Oh, and this lesson could be fairly lengthy so make yourself comfortable.

What your working environment should look like so far

In order to work through the rest of this lesson, let's summarize what you should have on your system, to make sure we're all working with the same content.

First, all of these examples are being run on a fully-updated, 64-bit Ubuntu 10.04 system, but you're welcome to use some other Linux distro as long as you're prepared to translate things like the package names appropriately.

To build your module against the kernel tree for the current kernel, you should have installed the corresponding kernel headers package for that kernel.

In addition, to allow us to play out on the cutting edge of kernel development, you should also have used git to clone the main kernel source somewhere under your home directory, and keep it up to date with the occasional git pull command being run from the top of that source tree.

Next, you should have the standard development packages on your system. And if you managed to compile and load your sample module in an earlier lesson, it's a safe bet you have everything you need for the rest of this lesson.

And, finally, for consistency, you should be running the stock, default kernel for your Ubuntu (or other) system, not any custom kernel you might have built. So if necessary, reboot your system to ensure you're running the 2.6.32-22-generic and we'll take it from there.

The variety of ways to build your modules

Given that you have a choice of kernels you might run, and a variety of possible source trees, let's cover the possible combinations for building and loading your own modules, based on the crash1.c example from an earlier lesson, and this portion of the Makefile you used:


KERNELDIR ?= /lib/modules/$(shell uname -r)/build
PWD := $(shell pwd)

.PHONY: build clean

        $(MAKE) -C $(KERNELDIR) M=$(PWD) modules

In case you'd forgotten, the purpose of the snippet above is to determine which kernel tree to build against, and that's where your choices come in. And here are your choices.

Using the default tree under the default kernel

This is the simplest combination, and doesn't require you to create and reboot a new kernel. You simply write your module, at which point the Makefile above uses as the necessary kernel source tree the symlink that takes you to the source tree that came as an Ubuntu package. Everything matches, and everything works fine, and after you compile your module, you can see clearly that it was built against the kernel that is currently running:

$ modinfo crash1.ko
filename:       crash1.ko
... snip ...       
vermagic:       2.6.32-22-generic SMP mod_unload modversions 

Feel free to build that module again and make sure it loads and unloads properly. This variation of module building is ideal for someone who has root privilege, but doesn't have the freedom to build and boot to a new kernel. So what's next?

Building against your custom kernel

You don't need to verify this section, but it should be obvious that if you have the freedom to build your own kernel and install it as described in earlier sections, you can build and load your modules exactly the same way since, once you install that new kernel and reboot, the symlink above will refer to the source tree against which you built your module, so the versions will match and, again, everything will work.

You can verify this if you want but we covered this in an earlier lesson so you can just take my word for it. But let's talk about a slight different variation now.

Running a stock kernel but building against a custom kernel

This is possibly the most interesting variation. Assuming you're running the stock Ubuntu 2.6.32-22-generic, you can still build your module against a different version of the kernel source tree, let's say in preparation for booting to a new kernel shortly. By building your module against a different source tree, that means you won't be able to load it until you're running the matching kernel version, but it's still interesting to see how the process works because it contains a couple traps.

Building your module against your git tree

Let's spend some time on this variation since it will explain a few more issues related to modules and kernel source trees.

Let's assume you've done a git checkout of the latest kernel source (into, say, the directory ~/k/git) and that it's reasonably up to date. In short, the version of that source tree clearly doesn't match the version of the running kernel. What effect is that going to have on your module building?

First, go over to the git checkout and run make distclean to clean out any remnants of previous configurations or builds you might have run. And now, back in your module directory, examine this line from your Makefile:

KERNELDIR ?= /lib/modules/$(shell uname -r)/build

If you're familiar with Makefiles, you'll know that the above is a conditional assignment, in that it will assign the standard kernel source location to the KERNELDIR variable only if it hasn't been set already. And, normally, that's just fine because that default value is what you want.

But if you're running the default kernel and yet want to build your module against the cutting edge kernel source, that's exactly the variable that you'll want to override to do that. In short, what you would do is clean out your module directory with:

$ make clean

then rebuild the module while overriding the kernel source tree location with:

$ make KERNELDIR=~/k/git

Theoretically, that will build a module against the newer kernel and loadable only when running a kernel with the same version number. It seems simple, but there's still a catch.

So what's the catch?

This is the catch:

$ make KERNELDIR=~/k/git
make -C /home/rpjday/k/git M=/home/rpjday/courses/crash/lkp/crash1   modules  
make[1]: Entering directory `/home/rpjday/k/git'

  ERROR: Kernel configuration is invalid.
         include/generated/autoconf.h or include/config/auto.conf are missing.
         Run 'make oldconfig && make prepare' on kernel src to fix it.
... snip ...

Oops, what just happened there? This is what happened.

You can't build a module against an absolutely sterile, pristine, make distcleaned kernel tree. At the very least, the kernel tree being used must be configured because there are some versioning files that are created by the configuration process that are essential to the module build process. But that's not enough.

You also need to perform at least the first part of the kernel build process since that will generate a few more files that the module build process needs. However, unlike what some documentation tells you, you don't need to perform an entire build. All that's required is, in the kernel source tree, to run the subsequent command:

$ make modules_prepare

Without getting into massive detail, that make invocation will run just enough of the kernel build process that the tree is now ready to let modules build against it. And once you do that in your kernel source tree, your module build should now work:

$ make KERNELDIR=~/k/git
make -C /home/rpjday/k/git M=/home/rpjday/courses/crash/lkp/crash1   modules  
make[1]: Entering directory `/home/rpjday/k/git'

  WARNING: Symbol version dump /home/rpjday/k/git/Module.symvers
           is missing; modules will have no dependencies and modversions.

Building with KERNELRELEASE = 2.6.35-rc3
  CC [M]  /home/rpjday/courses/crash/lkp/crash1/crash1.o
  Building modules, stage 2.
Building with KERNELRELEASE = 2.6.35-rc3
  MODPOST 1 modules
  CC      /home/rpjday/courses/crash/lkp/crash1/crash1.mod.o
  LD [M]  /home/rpjday/courses/crash/lkp/crash1/crash1.ko
make[1]: Leaving directory `/home/rpjday/k/git'

and I can verify the kernel version of the created module file with:

$ modinfo crash1.ko
filename:       crash1.ko
... snip ...       
vermagic:       2.6.35-rc3 SMP mod_unload 

and there we go. However, as I've already warned you, since you built that module against a different kernel version, you can't load it under the current kernel:

$ sudo insmod crash1.ko
insmod: error inserting 'crash1.ko': -1 Invalid module format

And all of the above should let you conclude one more obvious fact -- that the distro-specific kernel headers package installed on your system that lets you compile modules against it clearly represents at least that much of a configured kernel source tree; it can't just be pure source or you would have run into exactly the same failure as you just saw.

So what does all this have to do with git?

I'm glad you asked. As we've already established, some people will not have the freedom to build and run their own custom kernels, as much as they would like to, so they're restricted to simply building modules against the default kernel source and loading and unloading their modules. And for a lot of people, that's sufficient.

If, on the other hand, you have the flexibility to check out the git repository, then configure, build and reboot a custom, state-of-the-art kernel, not only can you build and load your modules against that new kernel source, but you can tweak the kernel source itself, and git will always let you revert all your changes back to the repository contents after you've finished playing.

To see how this works, move to the top directory of your git kernel source tree, and verify that nothing's different from the repository contents with:

$git diff

See? No output, no difference. Now let's pretend we wanted to tweak the source, and rebuild a kernel to see the difference. If you're feeling ambitious, examine this snippet of code from around line 396 of the kernel source file init/main.c:

  printk(KERN_INFO "Brought up %ld CPUs\n", (long)num_online_cpus());

If you've ever watched your system boot, you might recognize that line of output embedded in the midst of all the other boot messages. And if you have root privilege, you can see the record of all those boot messages in the log file /var/log/messages:

$ sudo grep "Brought up" /var/log/messages
... snip ...
Jun 19 06:36:52 lynx kernel: [    0.320009] Brought up 2 CPUs
Jun 19 08:00:09 lynx kernel: [    0.330014] Brought up 2 CPUs
Jun 19 11:22:49 lynx kernel: [    0.330015] Brought up 2 CPUs

And why do you care?

Because if you have the freedom to build your own kernel, you certainly have the freedom to start messing with the source, as in changing that boot-time diagnostic to something more entertaining, as in:

printk(KERN_INFO "Oh, my, there are %ld CPUs\n", (long)num_online_cpus());

And if you make that simple editing change and save it, git will be happy to tell you how your current checked-out source differs from the HEAD revision, as in:

$ git diff
diff --git a/init/main.c b/init/main.c
index 3bdb152..cd13d5b 100644
--- a/init/main.c
+++ b/init/main.c
@@ -393,7 +393,7 @@ static void __init smp_init(void)
        /* Any cleanup work */
-       printk(KERN_INFO "Brought up %ld CPUs\n", (long)num_online_cpus());
+       printk(KERN_INFO "Oh, my, there are %ld CPUs\n", (long)num_online_cpus());

If you're feeling ambitious, you can make that kind of innocuous change, save it, rebuild your kernel, install it, reboot and verify that, yes, your boot-time log message is now different. Which is something we'll be doing later in the course when we start modifying the kernel source in more meaningful ways.

And when you're tired of your local checkout changes, you can always simply revert everything in the entire tree back to the original source with:

$ git reset --hard HEAD
HEAD is now at 7e27d6e Linux 2.6.35-rc3
$ git diff

at which point, as you can see, git diff now verifies that you no longer have any local changes.

How much of the above you want to test is entirely up to you. And in the next lesson, we get back to writing and extending our loadable modules.

As they used to say on the TV series Sports Night, we'll be back so stick around. And by all means, leave questions and other feedback in the comments section. That's what it's there for.


Possible to have kernel version min modules?

So, is it possible to create a module that will work, say on every kernel from say, 2.6.35-rc3 and higher?

Or every kernel up to 2.6.35-rc3? for a kernel range?

Or say, that will be loaded by all 2.6.35 kernels, regardless whether its rc1, rc2 or rc3?

Or must it be an exact match? If so I assume then you could write kernel modules that can only be run against your own custom built kernel, such as 2.6.35-mipa+...

Inquiring minds.

Loading mismatched modules

As I understand it at the moment (and I'll check this later to make sure I'm right), your only option is to build a kernel that allows the loading of mismatched modules.

In other words, you can't control this from the module side, you can allow this only from the kernel side, so you might want to pop into your kernel configuration menu and, under "Enable loadable module support", you'll see a choice for "Module versioning support." If you *select* that, you are making the kernel you build more forgiving in terms of what modules it's prepared to accept.

Beyond that, I don't know of any way to be as specific as you describe above, but I'll look into it to make sure.


Ok, that makes more sense actually. It is different from user space programming, when you compile against modules or libs, but for this its obvious now that you say it...it should be up to the kernel to decide what it can and cannot load.

I wonder if there are any security implications to allowing more forgiving module loading in your kernel?

Not that I know of.

Typically, you'd only take advantage of mismatched module loading within a *very* narrow range of version numbers because you're testing a slightly newer kernel and you have all these loadable modules lying around that you'd rather not have to rebuild.

Beyond that, the worst that would normally happen is that your module and your kernel are so far out of sync that your module refers to kernel symbols that don't even exist anymore, so the module will fail.

But that's about as bad as things might get.

Some other gotchas with kernels and modules

Unfortunately, it isn't even possible to guarantee that a module will work with the same version of kernel.

For example, there are 3 different versions of kmalloc, depending on how your kernel has been configured. These 3 different versions are controlled by CONFIG_SLAB, CONFIG_SLUB, and CONFIG_SLUB (you can only pick one of these).

The kmalloc function is declared as inline (see include/linux/slab.h, slob_def.h and slub_def.h), and if your module uses it, your module can only be loaded against a kernel which has the same kmalloc allocator defined.

tried but could not make it work

Ok I have pulled the git tree to my local machine a month back and after I was able to reach this page I did try the change to line 396 as you mentioned.
But my problem is now the git diff command
pwd shows

and I made change to line 396 as you suggested and tried following

git diff diff --git ./main.c ./main.c
fatal: git diff [--no-index] takes two paths

My understanding is where you said in your example as
a/init/main.c is where is my problem is.
But I am not clear as what I am doing let me know what do you think?

git diff


If you're typing this:

git diff diff --git ./main.c ./main.c

then that's the problem.

All you need to type is "git diff" .

Git works on the whole tree, not individual files, and all the info it's working with is at the top level of the tree you cloned, under a directory called ".git" .

No matter how deep down in your tree you are, "git diff" goes back up to the top, and looks through the whole tree to report what you've changed in it.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <p> <br> <pre> <h1> <h2> <h3> <h4>
  • Lines and paragraphs break automatically.

More information about formatting options

By submitting this form, you accept the Mollom privacy policy.