Lesson 13: Proc files and sequence files -- Part 3

Printer-friendly versionPrinter-friendly version

So what's this about sequence files again?

Having taken the past two lessons to demonstrate some admittedly trivial (but still useful) examples of how to use proc files to allow users to dump/debug information from kernel space, we're going to complete the picture by explaining why the particular proc file implementation of sequence files was invented. And this lesson is going to get fairly involved so bear with me -- it'll be worth it in the end.

If you want a more complete coverage of the history of proc files, I've already mentioned that there's a good writeup in the online version of LDD3 here, which is why we'll try to keep things brief and not duplicate what's already in that book.

And, finally, there are a number of proc file issues we're simply not going to cover. Among other things, you can create writable proc files, but since we're concentrating on using proc files for informational and/or debugging purposes, we'll leave writable proc files as an exercise for the student.

The sample program

Since the sample program for sequence files is going to be somewhat involved, I'm going to attach both the module source file and its Makefile at the bottom of this page. So take a minute, grab them and save them into a directory of your choice, then we can talk about them.

At this point, I will explain nothing about sequence files except to explain what that sample program does and how you should build it and test it. The explanation will follow shortly.

The purpose of the sample program crash_evens.c is to create a proc file /proc/evens such that, when you list that proc file, it (verbosely) generates as output a certain number of even numbers, starting at zero. By default, the limit on the number of even values is 10 so that, if you build and load this module, you can then:

$ cat /proc/evens
The current value of the even number is 0
The current value of the even number is 2
The current value of the even number is 4
The current value of the even number is 6
The current value of the even number is 8
The current value of the even number is 10
The current value of the even number is 12
The current value of the even number is 14
The current value of the even number is 16
The current value of the even number is 18
$

As long as that module is loaded, if you run that same command, you should get the same output. (Note that I haven't even vaguely explained how that sample program works, I'm just explaining how you should build it, load it and test it.)

In addition, while you're testing the basic usage of that module in one display, you can also be following the debugging information generated by that module in another by tailing your /var/log/messages file thusly:

$ sudo tail -f /var/log/messages
... Entering start(), pos = 0.
... In start(), even_ptr = ffff880126662c98.
... In show(), even = 0.
... In next(), v = ffff880126662c98, pos = 0.
... In show(), even = 2.
... snip ...
... Entering stop().
... v is null.
... In stop(), even_ptr = ffff880126662c98.
... Freeing and clearing even_ptr.
... Entering start(), pos = 10.
... Apparently, we're done.
... Entering stop().
... v is null.
... In stop(), even_ptr = (null).
... even_ptr is already null.

Finally, sharp-eyed students will notice that the default limit of 10 even numbers to be printed is actually a module parameter, so you have the right to ask for more at module load time with something like:

$ sudo insmod crash_evens.ko limit=25

In any event, we don't expect you to understand the code yet, but if you could verify all of the above before you go any further and that the module works the way it's supposed to, that would be great. Then we'll dive into the explanation.

Exercise for the student: Note that while the limit of even numbers to print is a module parameter so that you can modify it at module load time, because it's a read-only parameter, it can't be changed once the module is loaded.

If you're feeling ambitious, change that to a writable parameter so that you can modify it while the module is loaded and running, and test that.

So what was the point of sequence files again?

As I've already mentioned, early proc files had an annoying limitation in that it was difficult (but not impossible) to generate more than a page of output in a single kernel space print operation. Because of this, a newer implementation of proc files -- called "sequence files" -- was invented that allowed the underlying kernel space code to produce the proc file's output in smaller, bite-size pieces, one print operation at a time, to beat the page size limit. But there's an important point to be made here.

While the kernel code will be producing the sequence file output a bit at a time, this is entirely transparent to user space. A user who wants to print the contents of a sequence file will still invoke a single command such as:

$ cat /proc/evens

and will get potentially page after page of what appears to be consecutive output lines. The fact that the kernel space code is generating all that output in bits and pieces is of no concern to user space, who doesn't need to know the underlying mechanics. All of that detail is hidden in the implementation of the sequence file.

So when would I use a sequence file?

Obviously, you'd want to use a sequence file if you have a massive amount of output to "print" to a proc file. But there's more to it than that.

Typically, you'd use a sequence file to print lengthy output that consists of a list or set of related items of some kind, such as all of the entries in an array, or the objects in a linked list, that sort of thing. That's because the way a sequence file works is that each invocation of its "show" routine is normally responsible for printing a single one of those "items," then remembering where it is in the list of items. The next invocation of the show routine will check to see which item it printed last, bump up its pointer or index or whatever, then print the next item in the list.

In short, a single user space listing of a sequence file will translate to multiple calls to the sequence file's show routine, each invocation printing the next item in the collection and keeping track of where it is, until there are no more items to be printed, and all of that happening completely transparently to the user. The only limitation is (you guessed it) that each of those items to be printed can't be larger than a single page but, for the most part, they rarely are depending on what you're trying to print.

Finally, you might remember that we've used sequence files to represent even proc files that generate very litle output, such as fs/proc/version.c:

$ cat /proc/version

That's only because sequence files have such a simple implementation that a lot of code uses them even when it's, strictly speaking, not necessary.

We're almost ready to get into the source of our sample program but there's one more critically important issue we need to cover.

The show() routine of a sequence file

Recall that all of our generic, readable proc files were responsible for implementing a "show()" routine that was responsible for generating the alleged contents of that proc file when someone tried to list in. In the case of the /proc/version file, the entire show routine was:

static int version_proc_show(struct seq_file *m, void *v)
{
        seq_printf(m, linux_proc_banner,
                utsname()->sysname,
                utsname()->release,
                utsname()->version);
        return 0;
}

In other words, in a single invocation, that file's show routine was responsible for generating the entirety of the proc file's output in a single print operation. However, when dealing with sequence files, how a show routine works is noticeably different.

With a sequence file, your "show" routine will be invoked once per item to be printed, so that the code in your show routine is not responsible for printing all of the output at once, only the output corresponding for one of the items that make up the entire output. This sounds simple, except that it raises the obvious question:

How does the show routine know which item to print?

The solution is fairly obvious -- while your show routine is printing each item, it's also keeping track of "where" in the list or set of items it is. In other words, somehow, your show routine is quietly storing off to the side a pointer or address or index of whatever item it last printed so that when it's called again, it will know which item has already been printed, it can "move" to the next item (however it does that) and, finally, it will clearly have to update its current pointer or address or index to move to the next one. Eventually, of course, the show routine will reach the end of the list of items to print, and it will return a special value of some kind to make this clear. All in all, a fairly simple idea. So what is it that your sequence file routine will be storing?

Curiously, it will be storing two seemingly equivalent pieces of information. It will store an "index" (normally representing the number of the item in the set), plus it will be storing an actual kernel space address of the current item. And every time that show routine is called, it will use one or both of these pieces of information to determine where to get the next item to print.

It might be that you're sequencing your way through an array, so it's easy to determine the next item to "print." Or you could be iterating your way through a linked list, which is still fairly simple to process. In any event, you'll see shortly that, no matter what kind of "items" you're working with, it's your responsibilily to know how to move from one to the next as your show routine is called over and over.

Now, let's talk about the routines you'll need to implement, and what they need to do.

The seq_operations structure

To have a proper sequence file that does actual sequencing, you'll need to define four routines as defined in the following structure that's declared in the seq_file.h header file:

struct seq_operations {
        void * (*start) (struct seq_file *m, loff_t *pos);
        void (*stop) (struct seq_file *m, void *v);
        void * (*next) (struct seq_file *m, void *v, loff_t *pos);
        int (*show) (struct seq_file *m, void *v);
};

Predictably, given their names, each of those routines will define what it means to start/initialize the printing, show the current item, move to the next item and any cleanup you need to do when you finally stop printing. So let's tackle those routines one at a time.

start()

When you try to print a sequence file, the first routine invoked is its start() routine:

void * (*start) (struct seq_file *m, loff_t *pos);

This routine does no actual printing -- all it does is initialize everything in preparation for printing. It may be that you need to, perhaps, initialize a device driver, or allocate some space, or what have you. Regardless, the start routine's job is to do all preliminary processing before any output is generated.

More specifically, the start routine is called with an initial offset of zero (that's the loff_t *pos argument you see up there), so that's where you can store your constantly increasing offset or index for the items as you print them one at a time.

In addition, the void* pointer that you return is the address in kernel space of the very first item to be printed in this sequence. In other words, part of the responsibility of your start routine is to figure out the address of the first item to be printed, and to return it. And you'll see why right away.

So far, so good?

show()

Your show routine will have the following prototype:

int (*show) (struct seq_file *m, void *v);

and its purpose is delightfully simple -- it will take the void* address of an "item," print it in whatever format you want, then return zero to signify success. In short, this is the code that "prints" a single item, nothing more. So far, so good? What's next, then?

next()

Here's the prototype of the next routine:

void * (*next) (struct seq_file *m, void *v, loff_t *pos);

and its job is actually fairly self-evident, but only after someone explains it to you, :-)

Given both the current item pointer and the "offset" of that item, the job of the next routine is:

  • Doing whatever it takes, calculate the address of the next item to be printed.
  • Increment the offset value based on whatever criteria you're using to define the offset of an item (perhaps as simple as its position in a list).
  • If there is a next item, return its address. Otherwise, if you're out of items and you're therefore done, return NULL to signify that.

All in all, the next routine is not that complicated -- its sole job is to simply figure out where the next item is, and return that. And finally ...

stop()

Predictably, you need to define a closing stop routine, whose job it is clean up any device initialization or dynamic allocation or what have you that was done in your start routine. In most cases, this is an incredibly trivial routine, but you need to define it anyway.

And that's it -- the general structure of an actual sequence file.

Generating even numbers -- a trivial example

As you can see, the example I chose to demonstrate sequence files was to generate a finite sequence of even numbers, where each printed number was treated as an "item" that was printed individually, whereupon the sequence file routines then determined the next item to go on to, and so on. But because this is such a massively simple example, I was able to cut corners in the code.

Recall that the purpose of a sequence file's "next" routine was to, given the address of an item, somehow calculate the address of the next item in the sequence -- suggesting that the next item would exist at a different address, and that perhaps all items would exist in kernel space at the same time.and have different kernel space addresses.

In the case of printing even numbers, I was able to simplify the code and allocate a single value to store the "current" even number, and when the "next" was called with the address of the current even number, rather than calculate where the next number would be, I quietly change the value being stored at that address, whereupon I return the same address I was passed.

Note well that this isn't what you'd normally do -- normally, you'd really have different addresses for all the items you're trying to print. But in this case, I'm going to dynamically allocate a single integer in the start routine, and just keep reusing it over and over for subsequent even numbers. In short, the "address" of succeeding items is never going to change, I'll just keep updating the value at that address. But that only works given the triviality of this example.

Now, let's see some output...

A sample run

If you build the code and load it, you can list the "contents" of the sequence file with the single command:

$ cat /proc/evens

and you'll get precisely the output you expect. But it's what's going on in /var/log/messages that shows you how the sequence file routines are being invoked in kernel space, where you'll see something like this:

... Entering start(), pos = 0.
... In start(), even_ptr = ffff880124546110.
... In show(), even = 0.
... In next(), v = ffff880124546110, pos = 0.
... In show(), even = 2.
... In next(), v = ffff880124546110, pos = 1.
... In show(), even = 4.
... In next(), v = ffff880124546110, pos = 2.
... In show(), even = 6.
... In next(), v = ffff880124546110, pos = 3.
... In show(), even = 8.
... In next(), v = ffff880124546110, pos = 4.
... In show(), even = 10.
... In next(), v = ffff880124546110, pos = 5.
... In show(), even = 12.
... In next(), v = ffff880124546110, pos = 6.
... In show(), even = 14.
... In next(), v = ffff880124546110, pos = 7.
... In show(), even = 16.
... In next(), v = ffff880124546110, pos = 8.
... In show(), even = 18.
... In next(), v = ffff880124546110, pos = 9.
... Entering stop().
... v is null.
... In stop(), even_ptr = ffff880124546110.
... Freeing and clearing even_ptr.

and if you look carefully, that output should make a certain amount of sense.

The first routine that's called is the start routine, which does any necessary initialization, followed by a call to the show routine which "prints" the value of the current item, then a call to the next routine to calculate the address of the next item to print, culminating in a call to the stop routine when the next routine signifies that there are no more items to print.

Note particularly how the address v of the item doesn't change from item to item, for precisely the reason I explained earlier -- that I'm being lazy and simply "reusing" the same item over and over.

But just when you think you understand sequence files, there's one more ugly detail that makes them even more complicated.

What happens you have lots of output?

To see what I mean, unload the module, then reload it with a limit of, say, 1000, to print that many even numbers. List the file, then examine the log file /var/log/messages to notice something strange in all that output:

... Entering start(), pos = 0.
... In start(), even_ptr = ffff880126662b88.
... In show(), even = 0.
... snip ...
... In show(), even = 186.
... In next(), v = ffff880126662b88, pos = 93.
... In show(), even = 188.
... Entering stop().
... v is ffff880126662b88.
... In stop(), even_ptr = ffff880126662b88.
... Freeing and clearing even_ptr.
... Entering start(), pos = 94.
... In start(), even_ptr = ffff880126662b88.
... In show(), even = 188.
...

Hang on, what happened there? You were nowhere near finished printing the first 1000 even numbers, but out of nowhere, the code called the stop routine, cleaned up after itself, then promptly called start to, apparently, pick up where it left off. Actually, that's exactly what it did.

Recall that limitation of early proc files -- an inability to print more than a single page of output. In fact, sequence files have a similar limitation in that, as they're printing item after item, if the next print would exceed a page size, the sequence files closes off, prints that output and calls the stop routine. Except that it immediately calls the start routine again with the last known offset and gives the start routine to pick things up where they left off and keep going! And the only way the start routine knows that it's being called in the middle of a sequence is based on that offset value that it's passed.

So, having explained all that, let's walk through the sample code routine by routine and convince ourselves that it all makes sense.

The code

The item data value

Note this near the top of the source file:

static int* even_ptr;

That's the pointer to where my even numbers will eventually reside -- the data item that I will cheat and keep reusing for every single even number.

The start routine

static void *
ct_seq_start(struct seq_file *s, loff_t *pos)
{
    printk(KERN_INFO "Entering start(), pos = %Ld.\n", *pos);

    if ((*pos) >= limit) {     // are we done?
        printk(KERN_INFO "Apparently, we're done.\n");
        return NULL;
    }

    //  Allocate an integer to hold our increasing even value.

    even_ptr = kmalloc(sizeof(int), GFP_KERNEL);

    if (!even_ptr)     // fatal kernel allocation error
        return NULL;

    printk(KERN_INFO "In start(), even_ptr = %pX.\n", even_ptr);
    *even_ptr = (*pos) * 2;
    return even_ptr;
}

What to notice about the above:

  • Every time I enter, I'll print my current "offset". The very first time, this is guaranteed to be zero, which is how the start routine is notified that we're actually starting. If that value is greater than zero, that denotes that we're being called again in the middle of a sequence, and we have to behave accordingly.
  • If our current position is larger than the limit, we've obviously printed all the even numbers and we're done and we return the value of NULL to tell the kernel that, yes, we're truly finished.
  • Each time the start routine is called, dynamically allocate a single integer to hold the even numbers. That's the address we're going to return as the address of every item.

Most importantly, you can see that, whenever that start routine is called, the initial even number is not just initialized to zero. Rather, it's initialized to twice the offset value passed in as an argument, to take into account that we're being re-invoked in the middle of a sequence, as I explained above.

So far, so good?

The show routine

This one's fairly simple:

static int
ct_seq_show(struct seq_file *s, void *v)
{
    printk(KERN_INFO "In show(), even = %d.\n", *((int*)v));
    seq_printf(s, "The current value of the even number is %d\n",
        *((int*)v));
    return 0;
}

You've been given the pointer to the "item" to print -- cast it to an integer pointer and print it as an integer, both to the sequence file and to the log file for debugging purposes.

Moving on ...

The next routine

This one should also be fairly self-evident:

static void *
ct_seq_next(struct seq_file *s, void *v, loff_t *pos)
{
    int* val_ptr;

    printk(KERN_INFO "In next(), v = %pX, pos = %Ld.\n", v, *pos);

    (*pos)++;                // increase my position counter
    if ((*pos) >= limit)     // are we done?
         return NULL;

    val_ptr = (int *) v;     // address of current even value
    (*val_ptr) += 2;         // increase it by two

    return v;
}

In short, you've been given two values to increment -- you should increment the offset to the next value depending on how you're defining that offset, and you need to take the current item address, and do whatever it takes to calculate where the next one is and return that.

Again, you'll notice that because our example is so simple, I just keep reusing the same data object, so I'm always returning the same address I'm receiving.

Also, if I can tell that I've run out of items, I return NULL.

The stop routine

Finally, closing out the picture, the stop routine is responsible for cleaning up but, as we explained above, this routine might actually be invoked in the middle of a sequence being printed, so it's important to know that the very end of your processing might involve two extra calls to both start and stop, which immediately recognize that there's really no more to be printed. This means that your stop routine has to be extremely careful not to try to deallocate space twice, as you can see it doing above.

Exercise for the student: To really appreciate what I wrote here, invoke the module and print 1000 numbers, whereupon you'll see something like this at the end of the messages in the log file:

... In next(), v = ffff88011405fa88, pos = 999.
... Entering stop().
... v is null.
... In stop(), even_ptr = ffff88011405fa88.
... Freeing and clearing even_ptr.
... Entering start(), pos = 1000.
... Apparently, we're done.
... Entering stop().
... v is null.
... In stop(), even_ptr = (null).
... even_ptr is already null.

Note well that, even after the last even number is printed and your stop routine is called and the space deallocated, your start routine will always be called again just in case, and it's your responsibility to make sure that your code can handle that boundary case -- that there really are no more items to print.

As another exercise, run the module again for just 10 even numbers, and if you check the log file, you'll still see those extraneous calls to start and stop at the end. This is just emphasizing that you have to code your start and stop routines to always be able to handle being called multiple times, including once even after all the items have been printed.

And in conclusion ...

I'll probably give this lesson a chance to digest, then come back later and tweak it as necessary. But, essentially, that's the end of proc files.

Exercise for the student: If you want to see an actual example of all of the above in action:

$ cat /proc/devices

then examine the kernel source file fs/proc/devices.c to see how that file is implemented.

Afterthoughts

I underestimated just how much there was to say about even simple examples of sequence files, but let me throw out a few more observations and reminders, just in case you're still following along and want even more of a challenge.

It's all invisible to the user

I know I've mentioned this already but it's worth driving home that all of the above intricacy involving sequence files and iterating and offsets and pointers is totally transparent to the user, who simply runs the command:

$ cat /proc/evens

and expects to see the correct output, regardless of what's happening underneath.

Do you really need a sequence file?

That's a good question. The rationale behind using a sequence file is that you're going to have lots of output from your proc file and, in addition, the output seems to fit the pattern of iterating through some kind of set or collection of objects.

But in some cases, even if you have loads of output, sometimes a trivial proc file will still do nicely. Take a look at the output from this command:

$ cat /proc/meminfo

While that output looks like it might match the sequence file model nicely, take a look at the kernel source file fs/proc/meminfo.c -- it's generated by one massive, single seq_printf() call, which works just fine.

The extra calls to start() and stop() at the end

As you'll have noticed earlier, even if you test this module with a very small number of even numbers to be printed, if you check the log file, there will always be an extra invocation of the start and stop routines after all your output has been printed. Sample tail end of the log file, as you may recall:

...
... Entering stop().
... v is null.
... In stop(), even_ptr = ffff880124546eb8.
... Freeing and clearing even_ptr.
... Entering start(), pos = 10.
... Apparently, we're done.
... Entering stop().
... v is null.
... In stop(), even_ptr = (null).
... even_ptr is already null.

The point I'm making here is that, even in trivial cases like this, you have to design your start and stop routines to handle those final calls and recognize immediately that there is no more output, and behave accordingly.

Why there are two different position values

Observant readers will have noticed that, as you step through the items to be printed, there are actually two values that are allegedly keeping track of your current "position." There's the kernel space pointer void* v that you keep passing around (whose purpose is fairly obvious), but there's also that offset value that seems not so clearly defined.

The first time people see the offset value, they're not quite sure what they're supposed to do with it since it would appear that the v address does everything they need.

I could be wrong but, as I've read it, the reason for the offset value is to allow you to pick up where you left off after intermediate calls to your start and stop routines. Unless I'm mistaken, when your start routine is called to resume printing partway through the list of items, you no longer have access to the kernel space item address, so you can't just start there. What you are passed is the offset value as it was last set and, from that value alone, you need to be able to figure out where you left off. The actual value you choose to store in that offset data object is entirely up to you, as long as you know how to map its value back to where in the list of items you need to resume printing.

In fact, if you look at the code for the /proc/devices file in fs/proc/devices.c, you'll notice something odd about how that sequence file moves from one item to the next:

static void *devinfo_next(struct seq_file *f, void *v, loff_t *pos)
{
        (*pos)++;
        if (*pos >= (BLKDEV_MAJOR_HASH_SIZE + CHRDEV_MAJOR_HASH_SIZE))
                return NULL;
        return pos;
}

If you look closely, the item address pointer v and the offset value are being used for the same purpose; that is, the offset value is what's being returned as the void kernel space address pointer. That's because sequencing through the kernel space devices is so simple that there's no reason to treat those two values differently. So they're just lumped together. If you can get away with this, it's perfectly acceptable.

What's that seq_file structure, anyway?

You'll notice that, in your sequencing routines, you're continually being passed a pointer to a structure of type seq_file. You didn't need to create it yourself, it's created for you and, in most cases, you can ignore it since you have no need to understand what it looks like -- you just use it, such as during a call to seq_printf().

But if you're curious, its declaration can be found in the kernel header file include/linux/seq_file.h:

struct seq_file {
        char *buf;
        size_t size;
        size_t from;
        size_t count;
        loff_t index;
        loff_t read_pos;
        u64 version;
        struct mutex lock;
        const struct seq_operations *op;
        void *private;
};

It's actually not a complicated structure and you'll notice that it appears to keep track of a number of obvious values, like size and offset. This suggests that, every time one of your sequencing routines is called, you have the ability to take a peek into that structure and, say, print some of its current values, just for the fun of it. And if you're feeling ambitious, why don't you do that, just to see how those values change as you're printing your even numbers.

Final exercise for the student: Can you find any other simple examples of sequence files being used anywhere in the kernel source tree?

AttachmentSize
Makefile391 bytes
crash_evens.c2.73 KB

Comments

Interesting

To be honest, this was the one that cause the light-bulb to come on regarding sequence files.

I'm going to try those exercises and mess with this on my own later

Very well-explained and clear to follow!

I thought the way Robert covered this and illustrated it with such a good code example made sequence files really easy to understand. Which is good because it's surprising how many examples I spotted which, like meminfo, obfuscate the sequence file mechanism by not using it 'fully'.
Great work, I thoroughly enjoyed this lesson.

Another simple example is in fs/proc/interrupts.c (for /proc/interrupts).
I thought this was interesting because the show() function - show_interrupts() - is not defined in this file along with the other three routines, but prototyped in include/linux/interrupt.h (definition example in arch/x86/kernel/irq.c).

Very Neat

I was really wondering with the sudden invocation of stop () and start () routines ? when i played with seq file initially ... so thanks a lot for explaining this .

Another simple example goes here .. [ cat /proc/net/vlan/config ]

Of course this we can see after adding the vlan interface :-)

This is how vlan module is careful in the start routine

/* start read of /proc/net/vlan/config */
static void *vlan_seq_start(struct seq_file *seq, loff_t *pos)
__acquires(rcu)
{
struct net_device *dev;
struct net *net = seq_file_net(seq);
loff_t i = 1;

rcu_read_lock();
if (*pos == 0)
return SEQ_START_TOKEN; [ Check for the first invocation ]

for_each_netdev_rcu(net, dev) {
if (!is_vlan_dev(dev))
continue;

if (i++ == *pos)
return dev;[Intermittent invocation ..returning new dev]
}

return NULL;
}

Very Neat

I was really wondering with the sudden invocation of stop () and start () routines ? when i played with seq file initially ... so thanks a lot for explaining this .

Another simple example goes here .. [ cat /proc/net/vlan/config ]

Of course this we can see after adding the vlan interface :-)

This is how vlan module is careful in the start routine

/* start read of /proc/net/vlan/config */
static void *vlan_seq_start(struct seq_file *seq, loff_t *pos)
__acquires(rcu)
{
struct net_device *dev;
struct net *net = seq_file_net(seq);
loff_t i = 1;

rcu_read_lock();
if (*pos == 0)
return SEQ_START_TOKEN; [ Check for the first invocation ]

for_each_netdev_rcu(net, dev) {
if (!is_vlan_dev(dev))
continue;

if (i++ == *pos)
return dev;[Intermittent invocation ..returning new dev]
}

return NULL;
}

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <p> <br> <pre> <h1> <h2> <h3> <h4>
  • Lines and paragraphs break automatically.

More information about formatting options

By submitting this form, you accept the Mollom privacy policy.