Friday, March 16, 2012

Introduction to Creating DKMS Packages for Ubuntu, Part 2

The DKMS framework makes it easy to create and distribute an out-of-tree kernel module, as demonstrated in part 1. However, the process is rather manual (read: error prone) for packages that will be updated regularly. This post is going to look at a way to set up the debian packaging components to make updating easier. A basic familiarity with debian packaging is assumed.

As an example, we'll use an updated version of the apple-gmux-dkms source tree from part 1, which can be donwloaded here. This time it's not necessary to extracting it to /usr/src; just extract it to wherever you like in your home directory.

The layout of the files has changed a little bit. The source file and makefile are still in the root of the tree, but dkms.conf has been moved into the debian directory and renamed to We'll talk more about this file later. Most of the remaining files are the typical debian packaging boilerplate; examining these files is left as an exercise for the reader. The one file we will take a closer look at is debian/rules.

The first item of note in the rules file is that it grabs the version number for the package from the changelog with the following.

 version := $(shell dpkg-parsechangelog | grep '^Version:' | cut -d' ' -f2 | cut -d- -f1)  

This eliminates the error-prone process of making sure the version gets updated throughout the package by making the changelog the canonical location for the version number. Towards that end, the rules file also has a rule to auto-generate the DKMS configuration file, debian/apple-gmux-dkms.dkms, from This file is identical to the dkms.conf file used in part 1, except that the version number has been replaced by the string @VERSION@. A sed command is used to generate the config file with @VERSION@ replaced by the actual version number.

 debian/apple-gmux-dkms.dkms: debian/  
     sed s/@VERSION@/$(version)/g $< > $@  

The other item worth noting is that there's a DKMS debhelper, dh_dkms, which makes setting up the rules file a breeze when used along with the dh helper (for more information about the dh helper consult the man page). We just need a rule to let dh do its thing:

     dh $@  

Then we need to override a few of the debhelper commands to install files for the DKMS package and to invoke dh_dkms. We also need to clean up the auto-generated DKMS configuration file, and since we're not actually building anything to generate the package we override the build command to do nothing. This leaves us with the following override rules.

        dh_clean debian/apple-gmux-dkms.dkms


override_dh_auto_install: debian/apple-gmux-dkms.dkms
        dh_install apple-gmux.c Makefile /usr/src/apple-gmux-$(version)/

override_dh_auto_install_indep: debian/apple-gmux-dkms.dkms  

That's it! Now you can build the binary and source packages with the usual debian packaging commands. When it comes time to update the package, you only need to update the version number in the changelog to get it updated throughout the package.

Introduction to Creating DKMS Packages for Ubuntu, Part 1

Dynamic Kernel Module Support (DKMS) is a really useful framework that allows kernel modules to be built dynamically for each kernel present on a system. The modules can also be automatically rebuilt any time a new kernel is installed. This is obviously a great tool if you want to distribute a driver that isn't included in the Linux kernel. I've used it a number of times to distribute test versions of drivers that are still under development.

It turns out that using DKMS is actually pretty easy, most of the time anyway. In this post I'm going to cover the basics of setting up a DKMS build on your local machine and the quick-and-dirty method of creating a debian package so that others can easily install the driver on their systems. Then in the next post I'll go over a slightly more sophisticated technique for using DKMS with debian packaging.

For this explanation I'm going to go through an example setup with a driver I've been working on recently to support the gmux device on some Apple laptops. This device is responsible for switching between the two GPUs found in these systems as well as backlight control. The files used in this example can be downloaded here. This archive should be extracted to the /usr/src directory. You'll also need to install the DKMS tools, which in Ubuntu is done by simply installing the package named "dkms."

DKMS expects the source for packages to be located under /usr/src in a directory named according to the format <module>-<version>. In this case, my module is apple-gmux and I'm giving it the version 0.1, so the source files are located in /usr/src/apple-gmux-0.1. Within this directory is configuration file named dkms.conf. In the case of apple-gmux this file is very simple. Let's go through it line by line.

1:  PACKAGE_NAME="apple-gmux"  
4:  BUILT_MODULE_NAME="apple-gmux"  
5:  DEST_MODULE_LOCATION="/updates"  
6:  AUTOINSTALL="yes"  

  • Line #1: PACKAGE_NAME unsurprisingly defines the name of the package. This needs to match the module name used in naming the directory.
  • Line #2: PACKAGE_VERSION tells DKMS the current version number, which once again must match the one used in the path.
  • Line #4: BUILT_MODULE_NAME tells DKMS about the filename of the module(s) that will be built. In this case the module is named apple-gmux.ko. This variable is actually an array, so if your package supplies multiple modules these can be specified as BUILT_MODULE_NAME[0], etc.
  • Line #5: DEST_MODULE_LOCATION tells DKMS where to install the module(s) relative to the kernel's module install path (typically /lib/modules/$(uname -r)). This is also an array, with each entry specifying the install path of the corresponding entry in BUILT_MODULE_NAME. For Ubuntu at least it's a good idea to install to /updates, as modules here automatically override the ones that shipped with your kernel.
  • Line #6: AUTOINSTALL, when set to yes, directs DKMS to automatically install the module(s) for any new kernels you install.
There are a number of other options, all documented in the DKMS man page.

Besides dkms.conf we also have the source file for the module as well as a makefile. The makefile is in the kernel format and has access to the kernel configuration variables. In this example the source is a single file, but it can be more complicated. You can have a source tree with multiple directories and multiple makefiles just like in the kernel. DKMS will use kbuild to invoke the top-level makefile, which can descend into subdirectories in the usual fashion. But since apple-gmux is so simple, I've opted to place everything in the top-level directory.

Once the DKMS module's source tree is in place, it's time to build the module. Working with DKMS packages is done using the dkms program by invoking a series of commands with the following syntax.

 dkms <command> <module>/<version>  

To build and install apple-gmux, we must invoke the add command followed by the build and install commands. Note that this only builds and installs for the current kernel; the build and install commands must be invoked with the -k option for other kernels you have installed.

 $ sudo dkms add apple-gmux/0.1  
 $ sudo dkms build apple-gmux/0.1  
 $ sudo dkms install apple-gmux/0.1  

Now we're able to use modprobe to load the apple-gmux module just like any other module. If we want to distribute the module for others to use, DKMS even has mkdeb and mkrpm commands do automatically create a Debian or RPM package.

This all works great for one-off DKMS packages, but it's a little cumbersome for something that will be updated regularly. In part 2 we'll cover a technique to simplify matters for frequently updated packages.

Thursday, December 15, 2011

Using ftrace to Identify the Process Calling a Kernel Function

Sometimes it's useful to know the context from which a kernel function is being called. For instance, I recently wanted to know what process was responsible for adjusting the keyboard backlight in response to hotkey presses on the MacBook Air I've been working with. Based on the way the LED class driver works, I knew that the callback into the hardware-specific driver (in this case applesmc_brightness_set() from drivers/hwmon/applesmc.c) would be called from the context of the process performing the backlight adjustment.

The easiest way that I know of to do this is using the ftrace function tracer, which will display the name and process ID for the current task each kernel function that is executed. Even better, if CONFIG_DYNAMIC_FTRACE is enabled it the kernel build configuration (as it is in the stock Ubuntu kernels), ftrace can be limited to tracing only a single function. Here's how to do it.

The files used for working with ftrace are located in the /sys/kernel/debug/tracing directory. All commands that follow assume that this is the current working directory, and all must be executed as root.

In order to limit the trace to only applesmc_brightness_set(), I used the set_ftrace_filter file.

$ echo applesmc_brightness_set > set_ftrace_filter

ftrace has a number of filters for different propose. As I mentioned earlier, the one needed to trace function calls is predictably named the function tracer.

$ echo function > current_tracer

With those command ftrace was set up. The next step was to enable tracing, adjust the keyboard backlight using the hotkey, and disable tracing.

$ echo 1 > tracing_on
# adjust backlight...
$ echo 0 > tracing on

The results of the trace a read from the trace file.

$ cat trace
# tracer: function
#              | |       |          |         |
         upowerd-1264  [003]  5875.189679: applesmc_brightness_set <-led_brightness_store

Using this technique, I was quickly able to determine that the upowerd processes is responsible for changing the keyboard backlight whenever I press the hotkey.

Thursday, November 17, 2011

Touchpad Protocol Reverse Engineering

Recently I was working on adding Linux support for some undocumented touchpad protocols, and in the process I developed some useful reverse engineering techniques for touchpads. I'm documenting them here in hopes that they may be useful to others, or at least to myself at some point in the future.

First some background information about touchpads. After a reset, most touchpads use the old, reliable PS/2 mouse protocol. This allows the touchpad to function at a basic level without any special drivers, but most of the features that laptop owners have come to expect from touchpads are missing. Enabling those feature requires sending the touchpad a vendor-specific sequence of commands that switches the format of the data reported by the touchpad from the PS/2 mouse protocol to a proprietary data format that generally encodes much more information.

So to get a touchpad working in Linux as more than just a mouse we need two pieces of information: the magic command sequence to switch the touchpad to the proprietary data format and the layout of the proprietary data packets. Figuring out how to decode the data packets is tedious and time consuming, but it really isn't all that difficult. Mostly it involves doing different actions on the touchpad and seeing which bits change as a result, repeating until you've worked out the meaning of all (or at least most) of the data. The reverse engineering of the magic command sequence, however, is trickier.

In the absence of a specification, probably the only reasonable way to work out the initialization sequence for the touchpad is to observe how the Windows driver does it. The technique I settled on for doing this was to modify a virtual machine to replace the usual PS/2 mouse emulation with support for passing the raw PS/2 data between the guest OS and the touchpad, logging each byte of data as it passes through. Linux provides a driver named serio_raw that makes this pretty easy by providing a character device node that can be used to interact directly with the PS/2 port from userland.

At the end of this post you will find links to patches that enable this functionality in both qemu and VirtualBox. These are little more than quick-and-dirty hacks to get the functionality I needed, and the VirtualBox version is quite a bit more complex than the qemu version (and even still includes some debugging code). In order to use the patches you must first put the touchpad in serio mode (I have also included a script below that searches for a PS/2 mouse and changes the first one it finds to serio mode). When in serio mode the touchpad will be unusable to Linux, so having an external mouse is a must.

The patched virtual machine will look for two environment variables to enable passthrough to the touchpad and control data logging. PSMOUSE_SERIO_DEV_PATH must be set to the path of the serio_raw device node, usually /dev/serio_raw0. If it is not set or the device node cannot be opened the VM will fall back to the standard PS/2 mouse emulation. PSMOUSE_SERIO_LOG_PATH is used to specify the path to file used to log the protocol data. Without this variable the passthrough will still work, but no data will be logged.

The log file is structured as plain text with one line for each byte of data passed between the touchpad and the guest OS. Data is in hexadecimal, and bytes sent to the touchpad are prefixed with S while lines received from the touchpad are prefixed with R.

The method I would recommend is to first install the guest OS without the PSMOUSE_SERIO_DEV_PATH set. After successful installation, restart the VM with the environment variables set and verify that the guest can interact with the touchpad as a standard PS/2 mouse. Finally, install the driver and verify that the touchpad is fully functional. If it is, shut down the VM and have fun diving into the log!

I also identified one potential alternate technique for capturing touchpad protocol data under Windows that could be done in a bare-metal installation. If this technique worked it would likely be more convenient, as installing Windows under qemu and VirtualBox doesn't always go so smoothly (which is the reason I have patches for both virtual machines!).

Windows supports something called a filter driver which, as I understand it, can sit transparently above or below another driver, filtering the data going between the driver and the adjacent layer in the stack. It may be possible to write such a driver that would be inserted below the vendor-supplied touchpad driver, logging all the PS/2 protocol data passing between the driver and the hardware.

I'm not at all familiar with the Windows driver architecture and haven't researched this in great detail to know whether or not it's actually possible. I thought I'd include it here in case someone knows of an existing tool that can do this or is up to the task of writing such a tool. If so, please drop me a line; I'd love to hear about it!

As promised, here are the files to make it all happen.

  • qemu-psmouse-serio-passthrough.patch: Patch to enable PS/2 mouse passthrough in qemu. Based on qemu-kvm_0.14.1+noroms-0ubuntu6 from Ubuntu Oneiric.
  • vbox-psmouse-serio-passthrough.patch: Patch to enable PS/2 mouse passthrough in VirtualBox. Based on virtualbox_4.1.2-dfsg-1ubuntu1 from Ubuntu Oneiric.
  • Script to change your touchpad to/from serio_raw mode. Run ' 1' to place the mouse in serio_raw mode and ' 0' to place it back into psmouse mode. Must be run as root.