Truth, Computing and Fail

Warning: Long post

This year is almost at its end. It’s been quite an eventful one for me. I graduated in May and in the meanwhile got a chance to work on Planga in my last semester thanks to Swaroop. Anyway, I’ll put off the details of my experience in those 5 months for a later post. Back to the actual topic.

I got a job around July in Bangalore. Being passionate about martial arts, it was kind of a plan to be based in Bangalore so I could get a chance to train at Independent Shootfighters Inc. It has been 5 months since I joined the classes. I have been training quite regularly and there have been remarkable improvements in all the physiological aspects. I’m stronger, fitter and faster than ever before. But these are just the changes on the surface. I believe the bigger changes have come internally, which is what this post is about. I have been been able to reduce my ego by a huge extent. I can’t claim it to be gone completely. I’m not that good yet. With the diminishing ego, I gained the ability to reflect on the opinions, biases I held and how I interacted with different people. Earlier the same effort would be thwarted by the ego that grows like cancer unless kept in check. In plain words, I realized I have been an asshole to a lot of people just because of my ego; and those were just the incidents I could recall. I have no idea about the occasions I might have done so without even having a clue about it.

While in college, I was part of the LUG and with a close group of other guys there, formed a well knit group that’d occasionally geek out. Individually the only things I’d care about were: coding, working out and insane partying (not in any order). It’s the first one that was at the root of all problems. Unknowingly, it developed a false sense of superiority, an air of smugness about the less technically accomplished people and also those not conforming to the ideology of hardcore geekdom. The aim of this post is to publicly apologize to whomever it concerns regarding such retarded indiscretions of mine when it came to getting my point across. Some examples follow.

• I published a rant of epic proportions against the generic “tech blogs”. Whether they serve any purpose or not is a different question. It’s certainly none of my concern if they are useless. If that’s really the case, people will stop visiting them and they’ll die out. Regarding this incident, I’m sorry about the insane douchebaggery I unleashed upon them and the persons in charge.




• I used to be full of contempt towards those who didn’t have a clue about open source or were willfully ignorant about it. I’d pitch in with particularly caustic and scathing comments right on their face. Behind their back, I’d bitch and make fun of them. I now see it was misplaced. It’s certainly none of my business if they don’t care. People are free to choose what they want, believe what they believe is the truth. I’d like to apologize to all such persons who have come across such behavior from me. It’s totally uncalled for.




• ..and a lot more I don’t recall.

Apart from these specific instances, I’m genuinely sorry for all the hateful speech and contempt directed towards individuals and groups just because their line of thought didn’t align with mine. (This does not cover the posts against the university administration though. That was some serious nonsense going on there. I’m glad to have got out of it all.) I also plan to personally apologize to the people I can recall having done the same to. There’s nothing better to start the new year than to shed the bad blood.

PS: This is for all the geeks/hackers/whatever-you-want-to-call-yourself and not quite related to this post. You might be really good with computers and be able to make them bend at your will. But please don’t treat that as a license to treat others like shit or get your nose up in the air. Realize that you might be completely useless for something that the other person might be a wizard in. Having a strong analytic bent of mind that is capable of following logic and reason is an enviable trait but don’t give yourself too much credit for it. When you do that, you start to feed your ego. You start to believe you are eligible to comment on anything and everything just because you are a “hacker”. Case in point, Hacker News. It’s beyond doubt that a lot of the users who hang around there are awesome hackers who’d rock the socks off most of the novices like me. But when you look at the comments on the submissions completely unrelated to programming or technology, one thing becomes amply clear. There’s a lot of nonsense hidden in verbiage from people who think they are awesome in the concerned field and under the implicit assumption that since such stuff is of interest to them, it automatically makes them experts. “Field of interest” and “expertise” are two very different things. The only thing it leads to is even more verbiage in ensuing threads of conversations which go round and round without contributing anything of real value. Except for feeding the ego of said persons. The internet makes it convenient to have a debate stretch endlessly on nothing but semantic bullshittery without either party conceding anything. So, in the end nothing of value is created. I know this might touch a few raw nerves and I might get flamed too. Still, think about it. The mental energy and space you get to play with when you let go of the ego is immense .

I have been recently looking into the sysenter/sysexit way of implementing system calls on Linux. It’s then that I came to know about the concept of VDSO (Virtual Dynamic Shared Object). It may look hacky to some but IMO, it’s quite an elegant and practical solution to overcome the incompatibilities that might be introduced if it was left to the userland libraries like libc to use the software interrupt or sysenter/sysexit mechanism. You will get more information about the VDSO here than I could ever dig into.

Even though the post linked above is very informative, it suffers from the same problems that plague most of the resources on linux kernel on the web. A couple of things are outdated there which could seriously put you off if you like to get your hands dirty along with reading such stuff.

• It states that the VDSO is

  a shared object exposed by the kernel at a fixed address in every process’ memory

  Unfortunately, this isn’t the case anymore. It might have been true for the <2.6.15 kernels but it certainly isn't that way on my 2.6.32 kernel. To get an idea, try this command a few times:
  cat /proc/self/maps | fgrep vdso
  This will give you the memory map of the process running `cat` itself. You will see the memory address to which the vdso is mapped is different each time rather than the fixed mapping to 0xffffe000 as the post claims, which brings us to our second problem.

• Assuming the fixed mapping at 0xffffe000, the post tells you to use dd to extract the relevant information by accessing the process' pages through /proc/self/mem.
  dd if=/proc/self/mem of=linux-gate.dso bs=4096 skip=1048574 count=1
  But things aren't the same now. You will never know the pages to skip over because the VDSO is always mapped at a different location everytime you run the `dd` command. If you don't believe me, try it out yourself.

To overcome this problem, I created this small script in python which will extracts the VDSO from its own mapping into a file and then you can use `objdump` to examine it.

"""
This script writes the VDSO to the file linux-gate.dso.1 .
Use `objdump -d linux-gate.dso.1` to examine it.
You might also want to play around more with the other objdump options and
the readelf tool  

LICENSE: MIT License ( http://www.opensource.org/licenses/mit-license.php )
"""
from __future__ import with_statement
import os
import re

## regex pattern for finding out the memory address range from the output line
pattern = re.compile(r'[\w\d]+-[\w\d]+')
with open('/proc/self/maps', 'r') as file:
    for line in file:
        line = line.rstrip()
        if '[vdso]' in line:
            addr_range = pattern.findall(line)[0]
            start_addr, end_addr = [int(addr, 16)
                                    for addr in addr_range.split('-')]

fd = os.open('/proc/self/mem', os.O_RDONLY)
os.lseek(fd, start_addr, os.SEEK_SET)
buf = os.read(fd, (end_addr-start_addr))

with open('linux-gate.dso.1', 'w') as file:
    file.write(buf)
    file.close()
os.close(fd)

I also created a github gist in case you need to track any further corrections to problems that might arise later on or maybe fork it.

The only documentation you have got for libfs is the code itself, which says a lot about the hoops I had to jump through to at least get this thing working

I still don’t claim that I know all the innards of libfs regarding this functionality. It took me around 3 days, poking around the code and quite a few kernel freezes and oops to figure this thing out. Also to be noticed is the fact is that this is for a non disk-based file system like /proc. I guess things will be different when it’s done on file systems like ext3.

WARNING: This is quite a long drawn post as I haven’t presented the solution in a straight forward manner. I have rather chosen to ramble on about my personal experience. For a no-nonsense, tldr-free answer, visit this

To begin with, I was absolutely clueless. All this time I had been using `ln -s` or symlink() without worrying about what really goes on under the hood. Imagine my frustration when ln -s didn’t work on the file system. I had completely neglected this part and thought that it’d be somehow taken care of automagically.

Going through a few online resources, I managed to deduce that the symlink operation takes place on inode level (whatever you make of that). Looking through the fields of struct inode , I found struct i_op for inode operations. Referring to both the book Understanding Linux Kernel and the source code for the fields in i_op, I found out a few function pointers that had something to do with links, namely symlink(), readlink(), follow_link().

Initially, I thought of implementing getattr() so that it’d return a S_IFLNK for the symlink but the idea of having to handle attribute generation for the rest of the dentry objects was too much for my puny brain and this plan was discarded.

Reading through man 2 symlink, I came across this and instantly everything was clear:

Symbolic links are interpreted at run time as if the contents of the link had been substituted into the path being followed to find a file or directory.

First, I changed the file creation function a bit for the symlink so that the proper st_mode is set and put the target location string in the i_private field of the inode structure.

Then I created a inode_operations structure and put in the following function definitions:

static void *sample_follow_link (struct dentry *dentry, struct nameidata *nd)
{
    nd->depth = 0;
    nd_set_link(nd, (char *)dentry->d_inode->i_private);
    return NULL;
}

static struct inode_operations sample_inode_ops = {
    .readlink = generic_readlink,
    .follow_link = sample_follow_link,
};

....
//in the function for the dentry and inode creation
inode->i_op = sample_inode_ops

But it was the second step that actually took up the most of my time. I read the man page of readlink and naively put in my own readlink implementation.

int sample_readlink(struct dentry *dentry, char __user *buffer, int buflen)
{
    unsigned long ret = copy_to_user(buffer, dentry->d_inode->i_private, buflen);
    if (ret == 0)
        return buflen;
    else
        return -EFAULT;
}

Kernel freezes happened and wailing of wolves were heard in the distance. Putting in just generic_readlink didn’t solve the problem either with kernel oops happening every time ls was run. Looking into the function definition of generic_readlink in the kernel source, I found it’s mentioned in the corresponding comment block that follow_link needs to be implemented for it to work. Looking into the code for ext2 gave me some idea about dealing with struct nameidata. Problem solved

I generally don’t rant on such well established open source projects simply because I’m not even remotely qualified to do so. But I’ll be making an exception this time. The bad cold I got more than a week back simply refuses to go away and poorly thought out documentation rules really raised my hackles this time.

Sample this from the struct fuse_operations documentation page:

int(* fuse_operations::write)(const char *, const char *, size_t, off_t, struct fuse_file_info *) 

Read the accompanying description. How is someone who is just starting off with FUSE supposed to know what the two char *’s are for? So off you go looking into source code provided by some tutorial and thereby waste at least 10 minutes in the process.

As it can be seen the docs have been generated by Doxygen. I know it’s “experimental” and all but is it really that difficult to write a bit more detailed comment on the function?

Disclaimer: I have only started out with developing kernel modules and even novice would be an overstatement to describe my current skills. What follows is stuff I gathered from different sources while trying to debug a kernel oops due to a module: some googled, some from the LDD3 book which finally put together gives more or less a basic strategy to start with debugging a kernel module.

As I figured out from reading LDD3, you can use one of these tools to debug a module

• plain ol’ gdb
• kgdb
• kdb

kgdb doesn’t really strike me as something I will be needing in the near future. I’m quite sure I won’t be taking the trouble to find another system to set up a debug session. But for all I know it might be invaluable to those involved in serious work.

kdb requires you to patch the kernel. I’ll admit I didn’t try this out of sheer laziness.

gdb should be a part of the arsenal of even a half-serious programmer and in my case, it was. There are just a few things that need to be in place before you start using it. First, you need the uncompressed kernel image, vmlinux (not vmlinuz). Second, you need to compile the kernel with some extra options to help you with debugging. This one is again from the LDD3 book, Chapter 4.

CONFIG_DEBUG_KERNEL* 

CONFIG_DEBUG_SLAB

CONFIG_DEBUG_PAGEALLOC

CONFIG_DEBUG_SPINLOCK

CONFIG_DEBUG_SPINLOCK_SLEEP

CONFIG_INIT_DEBUG*

CONFIG_DEBUG_INFO*

CONFIG_MAGIC_SYSRQ

CONFIG_DEBUG_STACKOVERFLOW

CONFIG_DEBUG_STACK_USAGE

CONFIG_KALLSYMS*

CONFIG_IKCONFIG*

CONFIG_IKCONFIG_PROC*

CONFIG_ACPI_DEBUG

CONFIG_DEBUG_DRIVER

CONFIG_SCSI_CONSTANTS

CONFIG_INPUT_EVBUG

CONFIG_PROFILING*

It’s not that all of these are absolutely necessary to get any kind of debugging work done but you never know what kind of oops/kernel panic you might be facing. Still I have starred the ones that I feel *must* be enabled. But don’t go by my words, compile and recompile to find out the truth

With all the yak mowing out of the way, you can finally start debugging the module with your freshly recompiled kernel.

Start the debugger with
#gdb /usr/src/linux/vmlinux /proc/kcore

But gdb doesn’t yet know where to find the module’s code and data sections. You can either do it manually by going into /sys/module/module_name/sections, cat-ing the values of .text, .data and .bss and then this command at the gdb prompt

(gdb)add-symbol-file /path/to/module 0xd081d000 \  # .text

 		-s .data 0xd08232c0 \

		-s .bss  0xd0823e20

or this shell script will output the whole command for you:

#!/bin/bash
#
# gdbline module image
#
# Outputs an add-symbol-file line suitable for pasting into gdb to examine
# a loaded module.
#
cd /sys/module/$1/sections
echo -n add-symbol-file $2 `/bin/cat .text`

for section in .[a-z]* *; do
    if [ $section != ".text" ]; then
	echo  " \\"
	echo -n "	-s" $section `/bin/cat $section`
    fi
done
echo

This information is again thanks to the LDD3 author Corbet, from this article. What would I have done without his book and articles?!

The module I was trying to debug was causing an oops due to null pointer dereferencing, which actually has been the source of quite a few vulnerabilities in the mainline kernel source. The following is what it looked like (got it from dmesg)

[27570.020736] BUG: unable to handle kernel NULL pointer dereference at 00000018
[27570.020747] IP: [<e07b3c31>] :plan9_net:socknet_connect+0xd1/0x110
[27570.020760] *pde = 00000000
[27570.020767] Oops: 0000 [#1] SMP
[snip]
[27570.020939]
[27570.020945] Pid: 8622, comm: bash Tainted: P          (2.6.27-14-generic #1)
[27570.020951] EIP: 0060:[<e07b3c31>] EFLAGS: 00010296 CPU: 0
[27570.020960] EIP is at socknet_connect+0xd1/0x110 [plan9_net]
[27570.020966] EAX: 00000000 EBX: 00000000 ECX: 00000000 EDX: 00f60000
[27570.020971] ESI: de4182a8 EDI: 00000002 EBP: dddedf20 ESP: dddedef4
[27570.020977]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[27570.020983] Process bash (pid: 8622, ti=dddec000 task=c198f110 task.ti=dddec000)
[27570.020988] Stack: 00000000 de4189f0 de418308 17000002 0101a8c0 d0f85494 d0f85493 de4ed200
[27570.021004]        dda077c0 d0f85494 d0f85493 dddedf54 e07b37d5 00000000 dd4fd340 dda077c0
[27570.021019]        0000000e dd4fd340 de4182a8 d0f85493 dd4fd340 d0f85480 09582808 caaa4540
[27570.021035] Call Trace:
[27570.021041]  [<e07b37d5>] ? tcp_n_ctl_process+0x145/0x170 [plan9_net]
[27570.021053]  [<e07b3505>] ? slashnet_write_file+0x185/0x190 [plan9_net]
[27570.021070]  [<c01b2c70>] ? vfs_write+0xa0/0x110
[27570.021081]  [<e07b3380>] ? slashnet_write_file+0x0/0x190 [plan9_net]
[27570.021092]  [<c01b2db2>] ? sys_write+0x42/0x70
[27570.021101]  [<c0103f7b>] ? sysenter_do_call+0x12/0x2f
[27570.021110]  [<c0380000>] ? __down_killable+0x60/0xd0
[27570.021121]  =======================
[27570.021124] Code: 7b e0 bb ff ff ff ff e8 cc ab bc df eb 8f 8b 58 0c 8d 55 e0 b9 10 00 00 00 c7 04 24 00 00 00 00 ff 53 10 89 c3 8b 45 f0 8b 40 14 <8b> 40 18 c7 04 24 5c 3e 7b e0 89 44 24 04 e8 9a ab bc df 85 db
[27570.021211] EIP: [<e07b3c31>] socknet_connect+0xd1/0x110 [plan9_net] SS:ESP 0068:dddedef4
[27570.021235] ---[ end trace 1d54537d6fc8b3bc ]---

Phew that’s a lot of information! You get a dump of all the register values, the stacktrace, codetrace etc in an oops message. I’ve given a couple of links at the end that deal with all the information present. Refer to them for more details.

For now, we can see that something was executed in the socknet_connect section at an offset of 0xd1 which caused the null pointer dereference. We’re very close to finding out the errant piece of code now. Just do the following in the gdb prompt to home in right on the culprit statement :
(gdb)list *socknet_connect+0xd1

..and we are done! Pretty simple and basic, wasn’t it?

These two links are really good for pointers on how to look for the necessary information in an oops message

• Re: what’s an OOPS by John Bradford from LKML
• A very detailed oops report analysis that’ll really help you with ‘how to get from bug report to the source of bug’

I’ve been also trying to use the offset information with the disassembled module to figure out which part of the source code it might actually correspond to. I haven’t met with much success though.

You can find a detailed tutorial here. This will only help you get started real soon with cscope.

First of all, you need to have cscope support compiled with your vim installation. Packages for vim offered by Ubuntu and Fedora already have it. In Gentoo, I had to add app-editors/vim cscope in /etc/portage/package.use for the cscope USE flag to be enabled.

With all the yak shaving out of the way, get down straight to business. Move over the cursor to any symbol or identifier in your source code and try out these:

• Ctrl-\-s: Search results are loaded in the same window
• Ctrl-Space-s: Search results are loaded in a horizontally split window
• Ctrl-Space(twice)-s: Search results are loaded in a vertically split window

You should also check out a link within that tutorial that will help you to use cscope efficiently with a large codebase such as the Linux kernel

PS: I forgot to mention that you need to install the plugin for these keymappings from the sourceforge project whose tutorial link I have already posted.

That is something which has been playing at the back of my mind for the past couple of days.

Google Code Jam Round 2 is on 26th Sept, 21:30 IST, right on the day PyCon India starts. In theory I can attend the 1st day of PyCon and compete in GCJ as well at night. But there are a few small things that have been bugging me like the possibility of being dead tired at the end of the day, lack of a decent internet connection etc.

Advancing to round 3 of GCJ would require me to be placed within the top 500 of the 3000 competitors. Effectively I have about 5-6 days in total to prepare for it, excluding the useless exams in between and a ~500 rank isn’t asking for too much. This is one of the reasons that I’m inclined to stay back instead of taking on a 10 hour long overnight bus journey coupled with running around the city for a whole day.

If you are reading this, put in a few words of wisdom (considering you have them at your disposal).

TechTatva is the technical fest held at my college and this year it starts from 8th September, that is a few days from now. No use of rambling on about its greatness and the value it brings to an otherwise morbid campus life.

Param is the category under TechTatva broadly relating to the computer science and IT oriented events. I’m in charge of handling it this time. Since last year, we have been trying to infuse some freshness into this category with the help of new events. Last year it was Mobivision which was about application development on two mobile platforms: Android and Symbian S60. The event was a huge hit in the college with 200+ people turning up for the workshops on PyS60 and Android development.

This time we have introduced an algorithm intensive event, Algosm which will be hosted and evaluated online by Codechef. This is the first time such an event will be a part of Param like Mobivision was last year. We hope this will help a lot of students in our college get acquainted with the overall concept of online programming contests.

Outstation participants are also welcome (obviously because it’s an online event . Do check out the Algosm page on the TechTatva website for more details.

This would be my first major open source contribution out in the wild. If you don’t know what an online judge means here, consider paying a visit to SPOJ or UVa Online Judge to get an idea.

I wrote a mammoth README file that covers all the issues and features of this system and won’t waste much space here copying the same thing. It’s hosted at github (Rohit, happy now? ). Head over to the main tree of the project.. Right now the source needs ugly hackery to get it running on another computer but I promise to correct that in the next few releases. Till then I believe the README you will help you out.

Oh, the code is licensed GPLv2 by the way. But you already knew it, didn’t you?