Biboroku

Polymorphism with Single Dispatch in Python

Sat, 31 Aug 2024 00:00:00 +0000

Python does not natively support function overloading, a feature commonly used in languages like C++ to create different behaviors based on function signatures. However, in Python, there is a way to achieve a level of polymorphism although in an extremely limited scope.

For functions that accept a single argument, functools.singledispatch() offers a straightforward means of registering dispatch functions according to the type of the single argument. The following code snippet illustrates a basic example of this feature:

import functools
from typing import *


@functools.singledispatch
def f(x: Any) -> str:
    return f"type: {type(x)}, value: {x}"


@f.register
def _(x: int) -> str:
    return f"int: {x}"


@f.register
def _(x: list) -> str:
    return f"list: {x}"


print(f(3))                     # int: 3
print(f([4, 5]))                # list: [4, 5]
print(f("foo"))                 # type: <class 'str'>, value: foo

Here, the original function f(x) is old-school Python, accepting an object of any type. Traditionally, branching based on the type of x might appear as:

def f(x: Any) -> str:
    if isinstance(x, int):
        ...
    elif isinstance(x, list):
        ...
    else:  # Any
        ...

This approach lacks elegance, concealing the different behavior for each input type within the implementation; we have to look into the code to know it. In contrast, using the functools.singledispatch decorator clearly exposes the distinct behaviors based on input types.

Needless to say, functools.singledispatch() does not provide full polymorphism in Python. However, for utility library functions that handle a single object at a time, it offers an elegant approach to streamline our code.

Creating a Debian Boot Partition on a USB Flash Drive

Mon, 08 Jul 2024 00:00:00 +0000

Having a boot partition on a USB flash drive provides two major benefits. One is to have the USB flash drive work as a “security key” without which the computer cannot boot into an OS; effectively, the boot USB flash drive becomes a house key. Another is to provide the consolidated storage for an encryption key to encrypt the main SSD drive.

Bootstrapping a Debian system to boot in this particular way is done at an installation time. This document outlines that process. We will provide the detailed steps only for partitioning storage devices (i.e., the main SSD drive and the USB flash drive), but not for the entire Debian installation. This is to keep the document succinct, but also because they are the only part that is different from the standard Debian installation steps.

What You Need

A Debian install medium (likely a USB flash drive)
A computer on which Debian is being installed
One unused USB flash drive on which to put the /boot partition

The unused USB flash drive must be different from the one with the Debian install media.

Here is a note about the /boot partition size: I would recommend that the USB flash drive to have at least 500 MB of storage space. The ideal size depends on usage, but the very minimal size should be determined by how many kernel-related files you would want to keep in the boot partition at a time. The very minimum number of kernel versions to keep is two, one for the main use and another for fallback or backup. As of writing, each kernel would need about 80 MB (for config, initrd.img, and vmlinuz files), which might be larger or smaller depending on how much you customize the kernel. The idea is to have enough space to keep all the kernel versions that you need available on boot. In reality, it is increasingly rare to be found in retail USB flash drives with less than 8 GB of storage, and there is no harm in using that much space for your boot partition.

I now assume that you are in the process of Debian installation, and right before the step for partitioning the hard drive(s).

Creating a Boot Partition on a USB Flash Drive

Before getting to the “Detect disks” step of the Debian installer, make sure that the unused USB flash drive is already plugged into the computer.

When you get to the “Partition disks” step, choose “Manual” as “Partitioning method.”

Identify the device name for the unused USB flash drive, e.g., “SCSI (0,0,0) (sdb).”

Select that entire device to create a new empty partition table on it.

Create a new EFI System partition on its free space:

Partition size: 128 MB
Location for the new partition: Beginning
Partition settings:
- Use as: EFI System Partition
- Bootable flag: on

Create a new boot partition on its remaining free space:

Partition size: (use all the remaining free space)
Partition settings:
- Use as: Ext2 file system
- Mount point: /boot
- Mount options: defaults
- Label: none
- Reserved blocks: 5%
- Typical usage: standard
- Bootable flag: off

The newly partitioned USB flash drive is now set up to be and hold the entire /boot partition.

Creating an Encrypted Partition on an SSD Drive

You will use the entirety of the SSD drive as an encrypted volume. Inside, the logical volumes for the / (root) partition and the SWAP partition will be created.

Throughout the current process, you may be asked a question “Write changes to disk and configure (something).” Be ready to answer “Yes” to this question. All the actions going forward are destructive, and all data on the SSD drive will be cleared.

Creating the Encrypted Volume

Select that entire SSD drive to create a new empty partition table on it.

Create a new partition on its free space with the following configuration:

Partition size: (use all the remaining free space)
Partition settings:
- Use as: physical volume for encryption
- Encryption method: Device-mapper (dm-crypt)
- Encryption: aes
- Key size: 256
- IV algorithm: xts-plain64
- Encryption key: Passphrase
- Erase data: yes
- Bootable flag: off

Choose “Configure encrypted volumes”:

Encryption configuration actions: Finish
Really erase the data on …: Yes
Type encryption passphrase: (choose wisely)

Configuring the Logical Volume Manager

Choose “Configure the Logical Volume Manager.”

Creating a volume group

Under “LVM configuration action,” choose “Create volume group”:

Volume group name: vg
Devices for the new volume group: /dev/mapper/* (choose encrypted device)

Creating logical volumes

Under “LVM configuration action,” choose “Create logical volume” for the swap space:

Volume group: vg
Logical volume name: swap
Logical volume size: (roughly the same as RAM)

Under “LVM configuration action,” choose “create logical volume” for the root partition:

Volume group: vg
Logical volume name: root
Logical volume size: (the remaining free space)

Under “LVM configuration action,” choose “Finish.”

Creating the Root and SWAP Partitions in Logical Volumes

Select the logical volume named “root” and configure the partition:

Partition settings:
- Use as: Ext4 journaling file system
- Mount point: /
- Mount options: defaults
- Label: none
- Reserved blocks: 5%
- Typical usage: standard

Select the logical volume named “swap” and configure the partition:

Partition settings:
- Use as: swap area

Finally, choose “Finish partitioning and write changes to disk.”

Write changes to disks: Yes

Follow and complete the rest of the Debian installation. Before the very first reboot, keep the boot USB flash drive plugged in, but remove the USB flash drive with the Debian install media.

Upon reboot, make sure that you can boot into the Debian login screen using the encryption passphrase that you set up earlier.

Move on to the next section once you confirm that you may log in on the newly bootstrapped machine as root. The shortcut Alt + Ctrl + F2 may be useful for console login as root.

Storing SSD Drive Encryption Key File in the Boot USB Flash Drive

In this section, we replace the encryption passphrase with a key file and store it in the boot USB stick. The key file will be encrypted with a (possibly different) passphrase using GnuPG.

Boot off the boot USB flash drive, and use Alt + Ctrl + F2 to log in as root via console. All the shell commands below are meant to be executed as root.

Generate a random key file:

dd if=/dev/urandom of=/keyfile bs=512 count=16

Look in /etc/crypttab for a line that looks like this:

nvme0n1p1_crypt UUID=<uuid> none luks,discard

where <uuid> depends on your hardware. Take note of the device name here (nvme0n1p1), as it may be different for your machine. It should refer to the encrypted volume of the SSD. Change that line to:

nvme0n1p1_crypt UUID=<uuid> /boot/keyfile.gpg luks,keyscript=/lib/cryptsetup/scripts/decrypt_gnupg

Add the key file to the LUKS keyring:

/sbin/cryptsetup luksAddKey /dev/nvme0n1p1 /keyfile

When asked, type in the existing passphrase created for the encrypted volume during the Debian installation process.

Encrypt the key file with gpg:

gpg -c --cipher-algo AES256 /keyfile

When asked, enter a (new) passphrase. Keep in mind that it is for encrypting the key file, not the volume this time. Thus, it can be the same as the one for the encrypted volume used earlier, as eventually, we will only keep the passphrase created in the current step.

Move the GPG file just created to the boot USB flash drive at /boot/keyfile.gpg:

mv /keyfile.gpg /boot/keyfile.gpg

Update the initramfs:

/sbin/update-initramfs -u

Reboot.

Once the reboot is successful, notice that you will get prompted for a passphrase of the encrypted key file, not the encrypted volume this time. Log in as root again, and remove the raw encryption key file created for temporary use:

cryptsetup luksKillSlot /dev/nvme0n1p1 0 --key-file /keyfile
shred -n 30 -uvz /keyfile

At this point, the boot USB flash drive becomes the “key” and will be the only way to boot this machine.

Once the computer boots, you may unmount the boot partition

Removing the Boot USB Flash Drive

After the successful boot, you may want to remove the boot USB flash drive. (You don’t want your key left inserted!)

Unmount the boot partition as follows:

sudo umount /boot/efi
sudo umount /boot

Then, physically remove the boot USB flash drive.

Waking up from Hibernation

When hibernation is active, you will need the boot USB flash drive to wake up the computer to boot back into the hibernated session.

With regular sleep (suspend-to-RAM), you will not need the boot USB flash drive.

Don’t forget to bring along your boot USB flash drive when traveling with a computer in hibernation.

Creating Spare USB Flash Drives

Like any other (physical) keys, you might want to keep a spare in case you lose it. It is easy to create one as follows.

First, obtain the device paths pointing to the master USB flash drive and the spare USB flash drive:

fdisk -l

Say, you have the master at /dev/sda and the spare at /dev/sdb. Then use dd to create an image dump of the master, and use it to create a spare:

dd if=/dev/sda of=/tmp/boot-key.img bs=16M  # generate an image of the master
dd if=/tmp/boot-key.img of=/dev/sdb bs=16M  # create the spare from the image

Of course, both USB flash drives need not be present at the same time. Just be very, very careful with the device paths when you insert/remove the USB devices, as it is easy to misremember the device reference and destroy important data.

Handling Kernel Updates

A maintenance release of Debian sometimes includes a kernel upgrade. Since Linux kernel-related files are stored in the /boot directory, such an update must be done while the boot USB flash drive is mounted on the computer.

Check if a pending Debian update includes a kernel update:

sudo apt update
sudo apt list --upgradable

If the list includes packages named like linux-image-*, that kernel will be installed upon the next APT upgrade, and the relevant kernel files will be created under /boot. For each kernel version, the following files would exist:

System.map-<x.x.x-x>-amd64
config-<x.x.x-x>-amd64
initrd.img-<x.x.x-x>-amd64
vmlinuz-<x.x.x-x>-amd64

where <x.x.x-x> is the kernel version and its revision number.

Insert the boot USB flash drive, and mount the boot partition if not done so already:

sudo mount /boot
sudo mount /boot/efi

Then, perform APT upgrade:

sudo apt upgrade

Especially if you manage more than one boot USB flash drive, a nice thing to remember is how to reinstall a package. With a kernel image, the command would look like this:

sudo apt install --reinstall linux-image-<x.x.x-x>-amd64

The kernel reinstall would become necessary when you have more than one boot USB flash drive to keep in sync with each other. Essentially, you need to repeat the kernel upgrade process for each boot USB flash drive, and the apt install --reinstall command above is all that is necessary.

Finally, purging the kernel package will remove the kernel-related files from the /boot directory:

dpkg -l | grep linux-image-*  # list installed kernel versions
sudo dpkg -P linux-image-<x.x.x-x>-amd64  # perge specific kernel version

You may need to perform this cleanup occasionally if your boot partition is small.

Snipping Transcripts with Snipd

Sat, 24 Feb 2024 00:00:00 +0000

For the past few years, podcasts have served as both a valuable source of learning and inspiration for me. Then, I quickly realized that integrating audio into a personal knowledge management (PKM) workflow is not a simple task. Fortunately, the emergence of new podcast apps is changing this situation. I have been using Snipd, an app that has become an indispensable companion in my PKM workflow over the past year or so.

A major draw of Snipd is its AI-powered text summarization feature, known as “snips.” The app transcribes a relatively short conversation window around each key point, somewhat similar to bookmarks, and then uses AI to summarize and aggregate these notes for syncing with PKM apps like Readwise, Notion, Obsidian, and others. My second brain is in Emacs, so I manually export and import these notes as markdown using purpose-written scripts.

While I am generally not a big fan of AI-powered summarization, I appreciate that Snipd effectively adds structures, as in chapters and sections, to otherwise unstructured audio content. Given that I am often interested in particular segments rather than entire podcast episodes, being able to get a quick overview before diving in saves me time. Apps like Snipd address the issue of inefficient content consumption and the lack of information discoverability inherent in audio formats.

At the same time, I often encounter situations where I simply want to extract short transcripts for personal reference. They might include interesting tangents, memorable quotes, or tidbits unrelated to the main theme. As these are often brief and not always crucial in a broader context, the AI tends to summarize away, which can be frustrating when I only want to capture the exact conversation. Snipd’s tendency to summarize with a wider context can be a drawback in these specific cases, and this limitation has been a significant pain point for me when using the app.

As it turns out, there is already a method available to achieve precisely what I want.

Instead of using the primary Snip button to generate a snip, I navigate to the transcript, highlight the specific text snippet that I wish to retain, and then choose “Create snip” from the context menu, as seen in the screenshot:

That’s all there is to it. It is straightforward.

This method still produces a snip with an AI summary, which might be unnecessary and overkill for a very brief text excerpts. Nonetheless, I get to keep the exact transcription that I need.

What’s described above is something I learned directly from Kevin Smith, the CEO of Snipd. It took place at a small meetup, in a cozy bar in Mission a few weeks ago.

Looking back, I should’ve been able to discover this feature earlier on my own by poking around the app in an exploratory manner. Nevertheless, it was enjoyable to learn this tip directly from the CEO. Hearing that the app and the entire business are still being managed by fewer than five people was indeed impressive .

It was also interesting to learn about other podcast enthusiasts and how they are building their PKM systems around podcasts. Overall, it was a pleasant gathering for a product of which I have been a fan and for which I anticipate a promising future.

Attribute Access with Dict

Sat, 09 May 2020 00:00:00 +0000

Python dict is useful. The access to a nested item can be tedious, however. For example,

data = {
  "hosts": {
    "name": "localhost",
    "cidr": "127.0.0.1/8",
  }
}

Here, data["hosts"]["cidir"] would get you "127.0.0.1/8", but all those quotes and brackets can be annoying to type and read. A singly nested dict like the case above is not bad, but this can quickly become tedium as the nesting deepens. It would be so much more succinct to be able to access an item at a key through attributes, as in data.hosts.cidir.

Such attribute access is fairly simple to implement using the __dict__ magic attribute Python uses to define object attributes:

from collections import UserDict

class AttrDict(UserDict):
    pass

def attrdict(d: dict) -> AttrDict:
    def addattrs(d):
        if not isinstance(d, dict):
            return d
        obj = AttrDict()
        for k in d:
            obj[k] = obj.__dict__[k] = addattrs(d[k])
        return obj

    obj = AttrDict()
    obj.update(d)
    obj.__dict__.update(addattrs(d))
    return obj

There are caveats. If a key in the input dict contains invalid characters for an attribute name (e.g., space), the item for the key will not be accessible as an attribute (though it remains accessible through an index). Keys also need to be str, whereas dict keys in general can be any hashable Python object. More importantly, if any key in the input dict conflicts with the method or attribute names of the dict interface, the object may not function properly as dict anymore.

So long as the limitations are understood, a solution like this can be good enough in some cases, simple JSON objects with well-defined structure, for example.

About Biboroku

Mon, 20 Apr 2020 10:55:26 -0700

Biboroku is a blog collecting my occasional thoughts and memorandum.

Conventions Used in This Blog

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function mames, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

For the shell prompt, $ denotes one for a normal, unprivileded user and # for the root user:

$ whoami
johndoe
# sudo su - root
# whoami
root

On Lazy Logging Evaluation

Tue, 14 Apr 2020 00:00:00 +0000

The stdlib logging package in Python encourages the C-style message format string and passing variables as arguments to its log method. For example,

logging.debug("Result x = %d, y = %d" % (x, y))  # Bad
logging.debug("Result x = %d, y = %d", x, y)     # Good

logging.debug("Result x = %(x)d, y = %(y)d" % {"x": x, "y": y})  # Bad
logging.debug("Result x = %(x)d, y = %(y)d", {"x": x, "y": y})   # Good

As the message does not need to be formatted when the logging level is set higher than the value set for the logger, we would waste time formatting the string in the “bad” cases above, when the message does not get logged at all. The “good” cases avoid this by deferring formatting until the logger finds it necessary.

Sometimes, though, we may want to log debug messages with values that are expensive to compute. For example, in the above example, what if y depends on x, and the function that does the computation, e.g.,

def get_y(x):
    ... expensive computation ...
    return result

is rather expensive? If this is done only for the purpose of the debug message, that would be a wasteful computation when the logging level is higher than the value at which the result is logged.

Lazy Function Evaluation with `logging`

Fortunately, it is easy to add lazy evaluation of function using a custom dict and functools.partial.

import logging
from collections import UserDict
from functools import partial


class Args(UserDict):
    def __getitem__(self, key):
        value = super().__getitem__(key)
        if callable(value):
            value = value()
        return value


def get_y(x):
    # ... expensive computation ...
    return result


logging.debug("Result x = %(x)d, y = %(y)d", Args(x=x, y=partial(get_y, x)))

The idea is to use partial to defer the execution of the function,¹ and store it in a dict-like object that is aware of the need to call the function on access through the key.

This is not a typical use of partial, but is a perfectly fine way of using the function. ↩︎

Installing Debian/Jessie on ThinkPad T460s

Sat, 30 Apr 2016 00:00:00 +0000

This is a Debian/Jessie installation instruction on ThinkPad T460s (product ID: 20F9CTO1WW).

First, update BIOS on Windows if necessary, since that kind of task is unfortunately typically much easier on it. Also make sure to create a recovery media for Windows 10.

Disable Secure Boot in BIOS.

Boot from the “netinst” (with firmware) install media. Use wired network while installing. Wireless does not work out of the box.

Boot from the USB install media. Follow the standard Debian installer instruction:

Language: English
Country: United States
Keymap to use: American English
Primary network interface: wlan0
Wireless network: (your choice)
Hostname: (your choice)
Domain name: (your choice)
Root password: (your choice)
Full name for the new user: (your choice)
Username for your account: (your choice)
Choose a password for the new user:
Re-enter password to verify: (your choice)
Select your time zone: (your choice)
Partitioning method: Manual

Here is my partitioning scheme:

1 /boot/efi  300 MB EFI System Partition (bootable, EFI System partition)
2 /boot      250 MB ext4
3 /          100 GB ext4
4 /usr/local  50 GB ext4
6 /home      338 GB ext4
5 SWAP        24 GB

Write changes to disks: yes
Debian archive mirror country: (your choice)
Debian archive mirror: (your choice)
HTTP proxy information: (your choice)
Participate in the package usage survey: (your choice)
Choose software to install: (your choice; Windows manager with WiFi support is recommended)

If asked during installation, force UEFI installation: Yes (on the SSD, not USB device from which the installer is running).

After the first reboot, log in as root and do the following:

It would be convenient to add your regular user account to sudo group:

# apt-get install sudo
# adduser <username> sudo

Intel Wireless 8260

Wireless does not work out of box. Upgrade kernel and install firmware. Add contrib and non-free, as well as backports to /etc/apt/sources.list, which should have lines similar to this:

deb http://ftp.us.debian.org/debian/ jessie main contrib non-free
deb-src http://ftp.us.debian.org/debian/ jessie main contrib non-free

deb http://security.debian.org/ jessie/updates main contrib non-free
deb-src http://security.debian.org/ jessie/updates main contrib non-free

deb http://ftp.debian.org/debian jessie-backports main contrib non-free

Then upgrade the kernel and video driver and install updated firmware:

# apt-get update
# apt-get install -t jessie-backports linux-image-4.5.0-0.bpo.1-amd64
# apt-get install -t jessie-backports firmware-linux-nonfree
# apt-get install -t jessie-backports firmware-iwlwifi
# apt-get install -t jessie-backports xserver-xorg-video-intel

Reboot and wireless should be working.

TrackPoint scrolling and disabling TrackPad

Install xinput:

$ sudo apt-get install xinput

Create ~.xsessionrc/ file with the following content:

#!/bin/bash
#
# ThinkPad T460s TrackPad/TrackPoint configuration
#
# Required package: xinput
#


# extract device id from xinput output
dev_id=""
function get_dev_id() {
    device_name="$1"
    s=$(xinput | grep "$device_name")
    s="${s#*id=}"
    dev_id="${s%%[!0-9]*}"
}


# TrackPoint
get_dev_id "TPPS/2 IBM TrackPoint"

xinput --set-prop --type=int --format=8 "$dev_id" "Evdev Wheel Emulation" 1
xinput --set-prop --type=int --format=8 "$dev_id" "Evdev Wheel Emulation Button" 2
xinput --set-prop --type=int --format=8 "$dev_id" "Evdev Wheel Emulation Axes" 6 7 4 5
xinput --set-prop --type=float "$dev_id" "Device Accel Constant Deceleration" 0.25
xinput --set-prop --type=float "$dev_id" "Device Accel Adaptive Deceleration" 1.0
xinput --set-prop --type=float "$dev_id" "Device Accel Velocity Scaling" 12.5


# TrackPad
get_dev_id "PS/2 Synaptics TouchPad"

# disable TrackPad if desired
xinput --set-prop --type=int "$dev_id" "Device Enabled" 0

Source this file explicitly

$ . ~/.xsessionrc

or upon the next boot, TrackPoint should be fully working.

Creating a Debian Bootable USB Stick with Non-Free Firmware

Sat, 14 Mar 2015 00:00:00 +0000

Debian installation on new hardware can be a hassle when it depends on non-free firmware support. A typical workaround is to use a Debian install image that includes non-free drivers, which is available here: Unofficial non-free images including firmware packages. Choose the right image for the kind of USB media you wish to create.

For this post, I chose (amd64 Jessie 8.4.0 netinst + non-free):

$ wget http://cdimage.debian.org/cdimage/unofficial/non-free/cd-including-firmware/8.4.0/amd64/iso-cd/firmware-8.4.0-amd64-netinst.iso

Find a USB stick (which you do not mind formatting entirely; all existing contents will be wiped out with this method), connect it to a working Linux box, and find the device pointer to the stick:

$ sudo fdisk -l
... (other devices) ...

Disk /dev/sde: 14.9 GiB, 16008609792 bytes, 31266816 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x156252ec

Device     Boot Start     End Sectors  Size Id Type
/dev/sde1  *        0 7800959 7800960  3.7G  0 Empty
/dev/sde2       20368   21199     832  416K ef EFI (FAT-12/16/32)

... (other devices) ...

We see that the USB stick is attached at /dev/sde in this case (IMPORTANT: this will certainly be different for you; do make sure the device is the USB stick you would not mind being wiped out).

Then create a bootable installation stick:

$ sudo dd if=firmware-8.4.0-amd64-netinst.iso of=/dev/sde bs=4M
$ sudo sync

TODO

There appears to be an issue writing to USB from isohybrid .iso image, from which I have not been able to create a proper bootable image. Check on this and update the post.

Reference

http://askubuntu.com/questions/372607/how-to-create-a-bootable-ubuntu-usb-flash-drive-from-terminal

Install Google Chrome on Debian Jessie

Thu, 27 Nov 2014 00:00:00 +0000

Download a DEB file for the version of Chrome you want from here.

Then

$ sudo aptitude install gconf-service libgconf-2-4 libnspr4 libnss3 libpango1.0-0 libappindicator1 libcurl3
$ sudo dpkg -i google-chrome-stable_current_amd64.deb

Installing Adobe Reader on Debian/Jessie

Fri, 10 Oct 2014 00:00:00 +0000

Once in a while you need to deal with fancier PDF files which may allow you to type in using forms, but Linux applications like Okular might not be fully capable of handling Adobe’s proprietary features. In such an unfortunate event, you might need to use Adobe Reader.

Go to the FTP download site on Adobe and download the version you wish to install. Here the version 9.x is assumed:

$ wget ftp://ftp.adobe.com/pub/adobe/reader/unix/9.x/9.5.5/enu/AdbeRdr9.5.5-1_i386linux_enu.deb
$ sudo dpkg --add-architecture i386
$ sudo apt-get update
$ sudo apt-get install libgtk2.0-0:i386 libxml2:i386 libstdc++6:i386
$ sudo dpkg -i AdbeRdr9*.deb

If the installation doesn’t fully complete, you might need to do

$ sudo apt-get -f install

to install the rest of missing packages.

Interpreting A/B Test using Python

Mon, 12 May 2014 00:00:00 +0000

Suppose we ran an A/B test with two different versions of a web page, $a$ and $b$, for which we count the number of visitors and whether they convert or not. We can summarize this in a contingency table showing the frequency distribution of the events:

	Not converted ($f$)	Converted ($t$)
$a$	$4,514$	$486$
$b$	$4,473$	$527$

It is trivial to compute the conversion rate of each version, $486 / (486 + 4514 ) = 9.72 \%$ for $a$ and $10.5 \%$ for $b$. With such a relatively small difference, however, can we convincingly say that the version $b$ converts better? To test the statistical significance of a result like this, hypothesis testing can be used.

Background

An appropriate hypothesis test here is Pearson’s chi-squared test. There are two types of the chi-squared test, the goodness of fit and the test of independence, but it is the latter that is useful for the case in question. The reason why a test of “independence” is applicable becomes clear by converting the contingency table into a probability matrix by dividing each element by the grand total of frequencies:

	Not converted ($f$)	Converted ($t$)
$a$	$P(V=a, C=f) = 0.4514$	$P(V=a, C=t) = 0.0486$
$b$	$P(V=b, C=f) = 0.4473$	$P(V=b, C=t) = 0.0527$

A table like this is sometimes called a correspondence matrix. Here, the table consists of joint probabilities where $V$ is the version of the web page ($a$ or $b$) and $C$ is the conversion result ($f$ or $t$).

Now, our interest is whether the conversion $C$ depends on the page version $V$, and if it does, to learn which version converts better. In probability theory, the events $C$ and $V$ are said to be independent if the joint probability can be computed by $P(V, C) = P(V)P( C)$, where $P(V)$ and $P( C)$ are marginal probabilities of $V$ and $C$, respectively. It is straightforward to compute the marginal probabilities from row and column marginals:

\begin{eqnarray} P(V=a) &=& \frac{4514 + 486}{10000} \nonumber \\ P(V=b) &=& \frac{4473 + 527}{10000} \nonumber \\ P(C=f) &=& \frac{4514 + 4473}{10000} \nonumber \\ P(C=t) &=& \frac{486 + 527}{10000} \nonumber \end{eqnarray}

where $10,000$ is the grand total of all the elements. The null hypothesis (i.e., a neutral hypothesis in which the effect we seek is absent) is that $V$ and $C$ are independent, in which case the elements of the matrix are equivalent to

	Not converted ($f$)	Converted ($t$)
$a$	$P(V=a) P(C=f)$	$P(V=a) P(C=t)$
$b$	$P(V=b) P(C=f)$	$P(V=b) P(C=t)$

The conversion $C$ is said to be dependent on the version $V$ of the website if this null hypothesis is rejected. Hence rejecting the null hypothesis means that one version is better at converting than the other. This is the reason why the test is on independence.

The chi-squared test compares an observed distribution $O_{ij}$ to an expected distribution $E_{ij}$

\begin{equation*} \chi^2 = \sum_{i, j} \frac{(O_{ij} - E_{ij})^2}{E_{ij}} \ , \end{equation*}

where $i$ and $j$ are the row and column indices of the matrix.¹ The values of $E_{ij}$ are computed from $P(V=i)$ and $P(C=j)$. The $\chi^2$ statistics thus obtained is now compared to the distribution assumed in the null hypothesis, and to do this we need to find the degree of freedom (dof) which is the shape parameter of the chi-squared distribution. For the test of independence using a $r \times c$ contingency matrix, the dof is computed from the total number of matrix entries ($r \times c$) minus the reduction in dof, which is given by $r + c - 1$. The reductions come from the row and column sum constraints, but decrease by one because the last entry in the matrix is determined by either the row or column sum on that row/column and therefore degenerate. Hence the dof for the test of independence comes out to be $(r - 1)(c - 1)$.

Python Implementation

Fortunately, it is very straightforward to carry out this hypothesis testing using scipy.stats.chi2_contingency. All we need is to supply the function with a contingency matrix and it will return the $\chi^2$ statistics and the corresponding $p$-value:

#!/usr/bin/env python2.7
# -*- coding: utf-8 -*-
"""An example of A/B test using the chi-squared test for independence."""
import numpy as np
import pandas as pd
from scipy.stats import chi2_contingency


def main():
    data = pd.io.parsers.read_csv('n10000.csv')
    data = data.set_index('version')
    observed = data.values
    print observed

    result = chi2_contingency(observed)
    chisq, p = result[:2]
    print 'chisq = {}, p = {}'.format(chisq, p)

    print

    data = pd.io.parsers.read_csv('n40000.csv')
    data = data.set_index('version')
    observed = data.values
    print observed

    result = chi2_contingency(observed)
    chisq, p = result[:2]
    print 'chisq = {}, p = {}'.format(chisq, p)


if __name__ == '__main__':
    main()

n10000.csv:

version	not converted	converted
A	4514	486
B	4473	527

n40000.csv:

version	not converted	converted
A	17998	2002
B	17742	2258

(The code and data are available in Gist.)

The result for the original table (of $n = 10,000$) is $\chi^2 = 1.76$ and $p = 0.185$. Since the $p$-value is greater than the standard threshold $0.05$, we cannot reject the null hypothesis that the page version and the conversion are independent. Therefore the difference in the conversion rates cited in the beginning of this article is not statistically significant.

What if we keep running the same A/B test a bit longer until we accumulate $n = 40,000$ visitors? Using example data (n40000.csv), we have the conversion rates of $2002 / 20000 = 10.0 \%$ for version $a$ and $2258 / 20000 = 11.3 \%$ for version $b$. Running the same test on the new data yields $\chi^2 = 17.1$ and $p = 3.58 \times 10^{-5}$. Since $p \ll 0.05$, the difference we see in the conversion rates is statistically significant this time. This is a demonstration of how a bigger sample helps to see a tiny difference. (The example data used in this article are generated assuming the true conversion rates of $10 \%$ for $a$ and $11 \%$ for $b$.)

For a $2 \times 2$ contingency table, Yate’s chi-squared test is commonly used. This applies a correction of the form

\begin{equation*} \chi^2_{\rm Yate’s} = \sum_{ij} \frac{(|O_{ij} - E_{ij}| - 0.5)^2}{E_{ij}} \end{equation*}

to account for an error between the observed discrete distribution and the continuous chi-squared distribution. ↩︎

Brand Positioning by Correspondence Analysis

Wed, 07 May 2014 00:00:00 +0000

I was reading an article about visualization techniques using multidimensional scaling (MDS), the correspondence analysis in particular. The example used R, but as usual, I want to find ways to do it with Python, so here goes.

The correspondence analysis is useful when you have a two-way contingency table for which relative values of ratio-scaled data are of interest. For example, I here use a table where the rows are fashion brands (Chanel, Louis Vuitton, etc.) and the columns are the number of people who answered that the particular brand has the particular attribute expressed by the adjective (luxurious, youthful, energetic, etc.). (I borrowed the data from this article.)

The correspondence analysis (or MDS in general) is a method of reducing dimensions to make the data more sensible for interpretation. In this case, I get a scatter plot of brands and adjectives in two-dimensional space, in which brands/adjectives more closely associated with each other are placed near each other.

Figure 1: Brand Positioning

As you see, brands like GAP, H&M, and Uniqlo are associated with youth, friendliness, and energy, while old-school brands like Chanel and Tiffany are associated with luxury and brilliance. This way of visualization is useful because the high-dimensional information (11 brands and 9 attributes) is reduced into a two-dimensional plane, and the distance on that plane is meaningful.

Here’s the code and data (available also in Gist):

#!/usr/bin/env python2.7
# -*- coding: utf-8 -*-
import matplotlib.pyplot as plt
import numpy as np
from numpy.linalg import svd


class CA(object):
    """Simple corresondence analysis.

    Inputs
    ------
    ct : array_like

      Two-way contingency table. If `ct` is a pandas DataFrame object,
      the index and column values are used for plotting.

    Notes
    -----
    The implementation follows that presented in 'Correspondence
    Analysis in R, with Two- and Three-dimensional Graphics: The ca
    Package,' Journal of Statistical Software, May 2007, Volume 20,
    Issue 3.

    """

    def __init__(self, ct):
        self.rows = ct.index.values if hasattr(ct, 'index') else None
        self.cols = ct.columns.values if hasattr(ct, 'columns') else None

        # contingency table
        N = np.matrix(ct, dtype=float)

        # correspondence matrix from contingency table
        P = N / N.sum()

        # row and column marginal totals of P as vectors
        r = P.sum(axis=1)
        c = P.sum(axis=0).T

        # diagonal matrices of row/column sums
        D_r_rsq = np.diag(1. / np.sqrt(r.A1))
        D_c_rsq = np.diag(1. / np.sqrt(c.A1))

        # the matrix of standarized residuals
        S = D_r_rsq * (P - r * c.T) * D_c_rsq

        # compute the SVD
        U, D_a, V = svd(S, full_matrices=False)
        D_a = np.asmatrix(np.diag(D_a))
        V = V.T

        # principal coordinates of rows
        F = D_r_rsq * U * D_a

        # principal coordinates of columns
        G = D_c_rsq * V * D_a

        # standard coordinates of rows
        X = D_r_rsq * U

        # standard coordinates of columns
        Y = D_c_rsq * V

        # the total variance of the data matrix
        inertia = sum([(P[i,j] - r[i,0] * c[j,0])**2 / (r[i,0] * c[j,0])
                       for i in range(N.shape[0])
                       for j in range(N.shape[1])])

        self.F = F.A
        self.G = G.A
        self.X = X.A
        self.Y = Y.A
        self.inertia = inertia
        self.eigenvals = np.diag(D_a)**2

    def plot(self):
        """Plot the first and second dimensions."""
        xmin, xmax = None, None
        ymin, ymax = None, None
        if self.rows is not None:
            for i, t in enumerate(self.rows):
                x, y = self.F[i,0], self.F[i,1]
                plt.text(x, y, t, va='center', ha='center', color='r')
                xmin = min(x, xmin if xmin else x)
                xmax = max(x, xmax if xmax else x)
                ymin = min(y, ymin if ymin else y)
                ymax = max(y, ymax if ymax else y)
        else:
            plt.plot(self.F[:, 0], self.F[:, 1], 'ro')

        if self.cols is not None:
            for i, t in enumerate(self.cols):
                x, y = self.G[i,0], self.G[i,1]
                plt.text(x, y, t, va='center', ha='center', color='b')
                xmin = min(x, xmin if xmin else x)
                xmax = max(x, xmax if xmax else x)
                ymin = min(y, ymin if ymin else y)
                ymax = max(y, ymax if ymax else y)
        else:
            plt.plot(self.G[:, 0], self.G[:, 1], 'bs')

        if xmin and xmax:
            pad = (xmax - xmin) * 0.1
            plt.xlim(xmin - pad, xmax + pad)
        if ymin and ymax:
            pad = (ymax - ymin) * 0.1
            plt.ylim(ymin - pad, ymax + pad)

        plt.grid()
        plt.xlabel('Dim 1')
        plt.ylabel('Dim 2')

    def scree_diagram(self, perc=True, *args, **kwargs):
        """Plot the scree diagram."""
        eigenvals = self.eigenvals
        xs = np.arange(1, eigenvals.size + 1, 1)
        ys = 100. * eigenvals / eigenvals.sum() if perc else eigenvals
        plt.plot(xs, ys, *args, **kwargs)
        plt.xlabel('Dimension')
        plt.ylabel('Eigenvalue' + (' [%]' if perc else ''))


def _test():
    import pandas as pd

    df = pd.io.parsers.read_csv('data/fashion_brands.csv')
    df = df.set_index('brand')

    print df.describe()
    print df.head()

    ca = CA(df)

    plt.figure(100)
    ca.plot()

    plt.figure(101)
    ca.scree_diagram()

    plt.show()


if __name__ == '__main__':
    _test()

The data in the CSV format used for the analysis:

brand,luxurious,traditional,intellectual,brilliant,calm,youthful,friendly,simple,energetic
Chanel,449,252,106,236,61,13,8,29,16
Louis Vuitton,410,286,83,142,80,18,20,48,31
Christian Dior,356,200,95,206,67,19,18,27,9
Tiffany,362,219,103,187,59,55,36,35,10
Rolex,442,248,114,89,109,4,9,52,12
Burberry,287,287,143,42,199,29,67,124,9
Ralph Lauren,198,191,101,39,147,61,70,100,9
Benetton,86,62,31,88,35,216,97,65,21
Uniqlo,6,7,10,8,23,260,331,199,291
H&M,8,5,10,2,10,272,132,91,223
GAP,10,10,10,9,24,275,203,137,84

PCA and Biplot using Python

Thu, 24 Apr 2014 00:00:00 +0000

There are several ways to run principal component analysis (PCA) using various packages (scikit-learn, statsmodels, etc.) or even just rolling out your own through singular-value decomposition and such. Visualizing the PCA result can be done through a biplot. I was looking at an example of using prcomp and biplot in R, but it does not seem like there is a comparable plug-and-play way of generating a biplot with Python.

As it turns out, generating a biplot from the result of PCA by pcasvd of StatsModels is fairly straightforward from the rotation matrix supplied by the function. Here is a code snippet (which is also available in Gist):

#!/usr/bin/env python2.7
# -*- coding: utf-8 -*-
"""Biplot example using pcasvd from statsmodels and matplotlib.

This is an example of how a biplot (like that in R) can be produced
using pcasvd and matplotlib. Additionally, this example does k-means
clustering and color observations by which cluster they belong to.

"""
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from scipy.cluster.vq import kmeans, vq
from statsmodels.sandbox.tools.tools_pca import pcasvd


def biplot(plt, pca, labels=None, colors=None,
           xpc=1, ypc=2, scale=1):
    """Generate biplot from the result of pcasvd of statsmodels.

    Parameters
    ----------
    plt : object
        An existing pyplot module reference.

    pca : tuple
        The result from statsmodels.sandbox.tools.tools_pca.pcasvd.

    labels : array_like, optional
        Labels for each observation.

    colors : array_like, optional
        Colors for each observation.

    xpc, ypc : int, optional
        The principal component number for x- and y-axis. Defaults to
        (xpc, ypc) = (1, 2).

    scale : float
        The variables are scaled by lambda ** scale, where lambda =
        singular value = sqrt(eigenvalue), and the observations are
        scaled by lambda ** (1 - scale). Must be in [0, 1].

    Returns
    -------
    None.

    """
    xpc, ypc = (xpc - 1, ypc - 1)
    xreduced, factors, evals, evecs = pca
    singvals = np.sqrt(evals)

    # data
    xs = factors[:, xpc] * singvals[xpc]**(1. - scale)
    ys = factors[:, ypc] * singvals[ypc]**(1. - scale)

    if labels is not None:
        for i, (t, x, y) in enumerate(zip(labels, xs, ys)):
            c = 'k' if colors is None else colors[i]
            plt.text(x, y, t, color=c, ha='center', va='center')
        xmin, xmax = xs.min(), xs.max()
        ymin, ymax = ys.min(), ys.max()
        xpad = (xmax - xmin) * 0.1
        ypad = (ymax - ymin) * 0.1
        plt.xlim(xmin - xpad, xmax + xpad)
        plt.ylim(ymin - ypad, ymax + ypad)
    else:
        colors = 'k' if colors is None else colors
        plt.scatter(xs, ys, c=colors, marker='.')

    # variables
    tvars = np.dot(np.eye(factors.shape[0], factors.shape[1]),
                   evecs) * singvals**scale

    for i, col in enumerate(xreduced.columns.values):
        x, y = tvars[i][xpc], tvars[i][ypc]
        plt.arrow(0, 0, x, y, color='r',
                  width=0.002, head_width=0.05)
        plt.text(x* 1.4, y * 1.4, col, color='r', ha='center', va='center')

    plt.xlabel('PC{}'.format(xpc + 1))
    plt.ylabel('PC{}'.format(ypc + 1))


def main():
    """Run a PCA on state.x77 from R and generate its biplot. Color
    observations by k-means clustering."""
    df = pd.io.parsers.read_csv('data/state.x77')
    print df.describe()
    print df.head()

    columns = ['Population', 'Income', 'Illiteracy',
               'Life Exp', 'Murder', 'HS Grad']

    data = df[columns]
    data = (data - data.mean()) / data.std()
    pca = pcasvd(data, keepdim=0, demean=False)

    values = data.values
    centroids, _ = kmeans(values, 3)
    idx, _ = vq(values, centroids)

    colors = ['gby'[i] for i in idx]

    plt.figure(1)
    biplot(plt, pca, labels=data.index, colors=colors,
           xpc=1, ypc=2)
    plt.show()


if __name__ == '__main__':
    main()

In addition to PCA, $k$-means clustering (three clusters) was run on the data to color the observations by how they cluster. The resulting biplot for states.x77 (which I exported and borrowed from R) looks like this:

Figure 1: Biplot

Near-Duplicate Detection using MinHash: Background

Sat, 12 Apr 2014 00:00:00 +0000

There are numerous pieces of duplicate information served by multiple sources on the web. Many news stories that we receive from the media tend to originate from the same source, such as the Associated Press. When such contents are scraped off the web for archiving, a need may arise to categorize documents by their similarity (not in the sense of the meaning of the text but the character-level or lexical matching).

Here, we build a prototype for a near-duplicate document detection system. This article presents the background material on an algorithm called MinHash and a method for probabilistic dimension reduction through locality-sensitive hashing. A future article presents their implementation with Python and CouchDB.

(Note that all the numbers generated for the tables in this article are made up for illustration purposes; they may not be mathematically consistent with any hashing algorithm.)

Jaccard Similarity Index

A similarity is represented by the Jaccard index:

\begin{equation*} J(D_i, D_j) = \frac{|D_i \cap D_j|}{|D_i \cup D_j|} \end{equation*}

where $D_i$ and $D_j$ are sets representing the two documents in our context.

Shingling

A useful way to construct a set representing a document is by shingling. To construct a set of $k$-singles from a text, a sliding window of $k$ characters is applied over the text. For example, if the text is “abcdabd,” the resulting set of 2-shingles is {ab, bc, cd, da, bd} (note that “ab” appears only once and is not repeated in the set).

The value of $k$ is arbitrary, but it should be large enough that the probability of any given shingle appearing in any random document is low. That is, if the available number of characters is $c$ and the character length of typical documents is $l$, we should at least ensure $c^k \gg l-k+1$. Since each character has a different frequency of appearance in a typical text, a suitable value for $k$ depends on the nature of documents and should be tuned accordingly. A good rule of thumb for an order of magnitude estimate is to assume $c = 20$ for English texts.

Instead of using individual characters, shingles can also be constructed from words. For example, in a math textbook we may often see a sentence beginning with a terse expression “it is trivial to show,” whose 3-shingle set is {“it is trivial”, “is trivial to”, “trivial to show”}. This has an advantage in that shingles built this way are more sensitive to the styles of writing. The style sensitivity may aid in identifying similarities between domain-specific texts buried in other types of documents.

Hashing Shingles

Typically, shingles are all hashed and grouped into buckets represented by an integer. The use of an integer is a huge advantage in terms of data compression. For example, a 4-shingle (of characters) typically uses 4 bytes, each byte used for a character, and this is good for representing $160,000$ 4-shingles (i.e., $20^4$). With a 4-byte, however, about 4 million ($2^{32}$) integers and therefore shingles could be represented, which is a good enough size for $k = 7$ (i.e., $20^7$). If a tiny probability of collision into the same bucket can be tolerated, $k$ can be chosen even larger. From here on, we assume a random hash function does not produce any collision between any pair of randomly chosen shingles, i.e., the mapping $s_i \rightarrow h(s_i)$ yields a unique integer.

Characteristic Matrix

Suppose we have a random hash function $h(s)$ and all possible singles $s_1$, $s_2$, $\dots$, $s_m$ from $D_1$, $D_2$, $\dots$, $D_l$ for a total of $l$ documents. We can summarize this in a characteristic matrix:

	$D_1$	$D_2$	$\dots$	$D_l$
$h(s_1)$	$1$	$0$	$\dots$	$1$
$h(s_2)$	$1$	$1$	$\dots$	$0$
$\vdots$	$\dots$	$\dots$	$\dots$	$\dots$
$h(s_m)$	$1$	$1$	$\dots$	$0$

where the entry of 1 indicates that the document $D_j$ contains a shingle $s_i$ for which a hash value $h(s_i)$ exists. (The entry of 0 means the shingle itself does not appear in that document.) It is trivial to compute Jaccard indices using any pair of documents from this matrix. In practice, however, the requirement for comparing all the hash values for a large number of documents makes the process prohibitive.

MinHash as a Jaccard Index Estimator

Let us focus on a pair of documents, $D_1$ and $D_2$, for which the shingles $s_1$, $s_2$, $\dots$, $s_7$ have been hashed by a function $h$. The relevant entries from the characteristic matrix look as follows:

	$D_1$	$D_2$
$h(s_1)$	$0$	$0$
$h(s_2)$	$1$	$0$
$h(s_3)$	$1$	$1$
$h(s_4)$	$0$	$1$
$h(s_5)$	$1$	$0$
$h(s_6)$	$0$	$0$
$h(s_7)$	$1$	$1$

There are three types of rows: (a) both columns have 1, (b) one of the columns has 1, and (c\) both columns have 0. We let $X$, $Y$, and $Z$ denote the numbers of rows categorized this way, respectively. For $D_1 = \left\{h(s_2), h(s_3), h(s_5), h(s_7)\right\}$ and $D_2 = \left\{ h(s_3), h(s_4), h(s_7)\right\}$, $X$ is the cardinality of their joint set and $Y$ is that for their union. Hence the Jaccard index is $X / (X + Y) = 2/5 = 0.4$.

Now, consider an experiment in which the rows in the matrix are randomly permutated. Remove the rows of type (c\), since they do not contribute at all to the union of two sets. We look at the first row of the matrix thus constructed and note its type defined above, either (a) or (b). Repeat the process n times. What is the chance that the first row found this way to be of type (a) above? The probability is given by $X / (X + Y)$, which is similar to the way Jaccard index is computed. This is the property that we use as a Jaccard index estimator.

In practice, randomly permuting a huge number of rows is very inefficient. Instead, we prepare a set of random hash functions $h_i(s)$ (for $i = \left\{1, 2, \dots, n\right\}$ for a set of $n$ measurements) that effectively permute the row order given the same set of shingles and sort rows in ascending order by hash values. (For this to be true, the hash functions need to be well-chosen and produce few collisions.) The row of the minimum hash value corresponds to picking the first row in the example above.

What we have shown is that, for estimating Jaccard indices, we only need to keep the minimum hash values generated from $n$ different hash functions. Therefore the very sparse characteristic matrix can be condensed into a signature matrix of minimum hash values with entries given by

\begin{equation*} h_i \equiv \min \left\{h_i(s_1), h_i(s_2), \dots, h_i(s_m)\right\} \ , \end{equation*}

where

\begin{equation*} D_j = \left\{ s_1, s_2, \dots, s_m \right\} \end{equation*}

is the set of shingles from the document $D_j$.

	$D_1$	$\dots$	$D_j$	$D_{j+1}$	$\dots$	$D_l$
$h_1$	$98,273$	$\dots$	$23$	$23$	$\dots$	$63,483$
$h_2$	$2,763$	$\dots$	$524$	$524$	$\dots$	$2,345$
$\vdots$	$\vdots$	$\dots$	$\vdots$	$\vdots$	$\dots$	$\vdots$
$h_{n-1}$	$325$	$\dots$	$567,849$	$567,849$	$\dots$	$987$
$h_n$	$876$	$\dots$	$7,849$	$32$	$\dots$	$897,347$

For supposedly near-duplicate documents such as $D\_j$ and $D_{j+1}$ in the table, most MinHash values are similar, and the fraction of similar values is an estimate of the Jaccard index. This is the gist of the MinHash algorithm. In other words, the probability $p$ that a pair of MinHash values from two documents $D_i$ and $D_j$ match is equivalent to their Jaccard index:

\begin{equation} p = J(D_i, D_j) \ . \label{eq:equality} \end{equation}

Locality-Sensitive Hashing

While the information necessary to compute document similarity has been compressed quite nicely into a signature matrix, examining all documents would take $l(l-1)/2$ pairs, each involving $n$ comparisons from signature entries. The vast majority of documents may not be near-duplicate, however, and in such a case we do not need every pair to be compared. Locality-sensitive hashing (LSH) offers a method of reducing the number of dimensions in high-dimensional MinHash signatures.

The idea is to partition a MinHash signature matrix into $b$ bands, each with $r$ rows (such that $n = b r$ is the total number of rows), and hashing MinHash integer sequences grouped by band. For example, if we have $n = 12$ MinHash values, we could partition them into $b = 3$ bands of $r = 4$ rows:

Band		$D_1$	$\dots$	$D_j$	$D_{j + 1}$	$\dots$	$D_l$
$1$	$h_1$	$98,273$	$\dots$	$23$	$23$	$\dots$	$63,483$
$1$	$h_2$	$2,763$	$\dots$	$524$	$524$	$\dots$	$2,345$
$1$	$h_3$	$49,368$	$\dots$	$7,207$	$7,207$	$\dots$	$59,542$
$1$	$h_4$	$9,559$	$\dots$	$34,784$	$34,784$	$\dots$	$6,095$
$2$	$h_5$	$37,153$	$\dots$	$14,927$	$14,927$	$\dots$	$581$
$2$	$h_6$	$8,671$	$\dots$	$17,492$	$17,492$	$\dots$	$6,664$
$2$	$h_7$	$2,763$	$\dots$	$43,306$	$43,306$	$\dots$	$10,916$
$2$	$h_8$	$2,600$	$\dots$	$38,712$	$38,712$	$\dots$	$45,472$
$3$	$h_9$	$14,352$	$\dots$	$25,862$	$25,862$	$\dots$	$14,812$
$3$	$h_{10}$	$263$	$\dots$	$52$	$52$	$\dots$	$11,973$
$3$	$h_{11}$	$325$	$\dots$	$567,849$	$567,849$	$\dots$	$987$
$3$	$h_{12}$	$876$	$\dots$	$7,849$	$32$	$\dots$	$897,347$

Then we use some good hash function $H$ which takes $r$ MinHash values and summarizes into integers, $H(h_1, h_2, h_3, h_4) \equiv H_1$ for band 1, $H(h_5, h_6, h_7, h_8) \equiv H_2$ for band 2, and so on. This reduces the $n \times l$ signature matrix into $b \times l$ matrix:

	$D_1$	$\dots$	$D_j$	$D_{j+1}$	$\dots$	$D_l$
$H_1$	$390$	$\dots$	$57,232$	$57,232$	$\dots$	$33,719$
$H_2$	$62,509$	$\dots$	$453$	$453$	$\dots$	$51,954$
$H_3$	$453$	$\dots$	$13,009$	$23,905$	$\dots$	$12,174$

Near-duplicate documents will be hashed into the same bucket within each band. In this example, $D\_j$ and $D\_{j+1}$ are in the same bucket for bands 1 and 2. (Note that $D\_1$ in band 3 has the same hash value as $D\_j$ and $D_{j+1}$ in band 2, but they are not considered to be in the same bucket since the bands are different.) The documents that share a bucket within a band are considered candidates for further examination. The advantage is that, since $b \ll n$ in general, the number of required comparisons is much smaller. The LSH thus provides a way to select candidates for near-duplicate detection, before full signature comparisons are carried out.

The assumption is that a pair of documents, if near-duplicate, has a total of $b$ chances of being hashed into a common bucket in at least one of the available bands. Recall from Eq. $\eqref{eq:equality}$ that the probability that a pair of MinHash values from two documents match is equivalent to their Jaccard index. The probability that a pair of documents share a bucket in a band of $r$ rows is given by $p^r$. Its complement, $1 - p^r$, is the probability that a document pair does not get hashed into the same bucket for a band. Then the probability that two documents become candidates in at least one band is given by $1 - (1 - p^r)^b$. Plotting for varying $b$ and $r$, the function forms a series of S-curves¹:

Figure 1: Probability of Being Chosen as Candidates vs. Jaccard Index

The figure provides an intuition as to how the values of $b$ and $r$ should be chosen for a target Jaccard similarity threshold $t$ (above which two documents are considered near-duplicate). Let $f(p) = 1 - (1-p^r)^b$. The value of $p$ for the steepest slope is obtained from the second derivative, $d^2f/dp^2 |_{p=p_t} = 0$, which is

\begin{equation*} p_t = \left(\frac{r-1}{br-1}\right)^{1/r} \approx \left(\frac{1}{b}\right)^{1/r} \end{equation*}

for $b$, $r \gg 1$. As a rule of thumb, we want $p_t \approx t$, but the exact value of $p_t$ can be adjusted based on rejection criteria. Choosing $p_t > t$ reduces false positives, whereas $p_t < t$ reduces false negatives at the candidate selection step.

Reference

Anand Rajaraman and Jeffrey David Ullman (2011). Mining of Massive Datasets. Cambridge University Press. ISBN 978-1-107-01535-7

The figure is generated by the following Python script:

import matplotlib.pyplot as plt
import numpy as np

ps = np.arange(0, 1, 0.01)

for r in [5, 10]:
    for b in [2, 4, 8]:
        ys = 1 - (1 - ps**r) ** b
        plt.plot(ps, ys, label=f"r={r}, b={b}")

plt.xlabel("Jaccard index")
plt.ylabel("Probability of being chosen as candidates")
plt.legend(loc="upper left")
plt.show()

↩︎

A Trick for Computing the Sum of Geometric Series

Fri, 04 Apr 2014 00:00:00 +0000

Say if I need to compute the sum of a series like this one:

\begin{equation} y = 1 + 2 x + 3 x^2 + 4 x^3 + \dots \ , \label{eq:a} \end{equation}

where $|x| < 1$. This series looks like a geometric series in which case the sum can be computed from

\begin{equation*} \sum_{k = 0}^{\infty} a x^k = \frac{a}{1 - x} \ . \end{equation*}

The coefficients vary in Eq. $\eqref{eq:a}$, so the relation cannot be directly used.

There is a trick to transform a series like above into a geometric series. Multiplying Eq. $\eqref{eq:a}$ by $x$,

\begin{equation} xy = x + 2x^2 + 3x^3 + \dots \, \label{eq:b} \end{equation}

Subtracting Eq. $\eqref{eq:b}$ from Eq. $\eqref{eq:a}$,

\begin{equation*} y(1-x) = 1+x+x^2+x^3+\dots \end{equation*}

where the right-hand side is a geometric series with $a = 1$. Hence

\begin{equation*} y = \frac{1}{(1-x)^2} \ . \end{equation*}

Creating Debian Wheezy Installation USB Stick

Thu, 20 Feb 2014 00:00:00 +0000

The simplest way to create a bootable USB stick for Debian installation is to use command dd to copy an ISO image to an empty USB stick.

First, connect an empty USB stick to a working Linux box, and see which device represents the stick:

$ sudo fdisk -l
...
Disk /dev/sdb: 16.0 GB, 16008609792 bytes
255 heads, 63 sectors/track, 1946 cylinders, total 31266816 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xb745c02d

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              32    31266815    15633392    c  W95 FAT32 (LBA)

So, on my computer, /dev/sdb is the USB stick. Whatever is on it will be deleted and overwritten in this procedure, so make sure that I use a stick with large enough disk capacity specifically for this purpose.

For the netinst installer, about 300 MB would be enough. When the access to the network is not guaranteed during the installation (e.g., the network adapter is not properly recognized by the Debian Installer), it is desired to at least use the full CD/DVD media, which would take about 4 GB (just the first medium should be enough for bare minimal installation, after which more full configuration can be resumed afterward).

Download an ISO image for the Debian installer:

$ wget http://cdimage.debian.org/debian-cd/7.4.0/amd64/iso-cd/debian-7.4.0-amd64-netinst.iso

Then copy the ISO image to the USB stick:

$ sudo dd if=debian-7.4.0-amd64-netinst.iso of=/dev/sdb bs=4M
$ sudo sync

Customizing & Installing Linux Kernel on Debian Wheezy

Sun, 30 Sep 2012 00:00:00 +0000

Here is a quickie for customizing and install Linux kernel 3.5.x on Wheezy.

Add yourself (with account username) to sudoer group:

# adduser username sudo

You need to logout and login for this change to take effect. You also need to be able to use sudo or su to install the new kernel in the end.

Install some packages and kernel source:

$ sudo aptitude update
$ sudo aptitude install kernel-package pkg-config bzip2 g++ libncurses5-dev fakeroot bc
$ sudo aptitude install libqt4-dev

(libqt4-dev is only necessary if you wish to use make xconfig instead of make menuconfig below.)

The kernel packaged for Wheezy is at version 3.2. Source code for a more up-to-date kernel version can be obtained from kernel.org and saved somewhere. You just need to expand the compressed source somewhere (even under your home directory, which is the case in this post).

Either way, the customization procedure is similar; when you expand the source tree (e.g., the tar statement below), you just need to specify the source tarball you wish to use. Here I just follow the Debian way:

$ sudo aptitude install linux-source-3.2

Extract the source tree:

$ cd ~/src
$ tar -jxf /usr/src/linux-source-3.2.tar.bz2
... or use tar -Jxf for .tar.xz file ...
$ ln -s linux-source-3.2 linux
$ cd linux

Edit the EXTRAVERSION entry in Makefile, as in:

EXTRAVERSION = .20120929.1

for example to add .20120929.1 to the kernel version number. This is convenient for keeping the existing, working kernels around when you need to recompile with different options.

Use xconfig or menuconfig to customize the kernel options. Before the make-kpkg lines, setting concurrency (most likely to the number of cores of my processor) is optional but having a higher number typically reduces the compilation time.

$ make mrproper
$ make xconfig    # or make menuconfig
$ export CONCURRENCY_LEVEL=2    # for dual-core; this is optional
$ fakeroot make-kpkg clean
$ fakeroot make-kpkg --initrd kernel_image
... fakeroot might fail; when it does, try running make-kpkg commands as root ...
$ sudo dpkg -i ../linux-image-3.2.20120929.*_amd64.deb

Upon reboot in the GRUB menu you will find the newly installed kernel:

$ sudo reboot

Purging Old Kernel Image from System

For example, if the kernel to be uninstalled is version 2.6.26 and the sub version that I used was 20091112.1, do:

$ sudo dpkg -P linux-image-2.6.26.20091112.1

That’s it. However, it is often a good idea to keep at least one kernel image that I know for sure to work so that when a custom kernel fails, I have something to fall back on. On the other hand, it is also a good idea to purge very old kernel images to save space in /boot.

Power Management on Lenovo T430s with Debian Wheezy

Sun, 30 Sep 2012 00:00:00 +0000

Since tp_smapi does not work properly on T430s, I am using TLP this time. I still want some traditional tools

$ sudo aptitude install powertop

but not others:

$ sudo aptitude remove --purge laptop-mode-tools

To install, follow the instruction. In short, first update the APT source list by adding the following line to /etc/apt/sources.list:

deb http://ppa.launchpad.net/linrunner/tlp/ubuntu lucid main

and run

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 02D65EFF
$ sudo aptitude update
$ sudo aptitude install tlp tlp-rdw tp-smapi-dkms smartmontools ethtool acpi-call-tools</pre>

Upon reboot, TLP starts running.

I am impressed by the ease of installation and how effective TLP is on configuring T430s to start power saving from the get go. According to powertop, T430s is already using just 7 - 8 W on battery. That’s several watts below I had been using on other ThankPads!!

Controlling the behavior of TPL is done by modifying /etc/default/tlp. I mostly use the default except the following changes:

...

# Main battery (values in %)
START_CHARGE_THRESH_BAT0=75
STOP_CHARGE_THRESH_BAT0=90
# Ultrabay battery (values in %)
START_CHARGE_THRESH_BAT1=75
STOP_CHARGE_THRESH_BAT1=90

...

# Radio devices to disable on connect
DEVICES_TO_DISABLE_ON_LAN_CONNECT=&quot;wifi wwan&quot;
DEVICES_TO_DISABLE_ON_WIFI_CONNECT=&quot;wwan&quot;
DEVICES_TO_DISABLE_ON_WWAN_CONNECT=&quot;wifi&quot;

# Radio devices to enable on disconnect
DEVICES_TO_ENABLE_ON_LAN_DISCONNECT=&quot;wifi&quot;
DEVICES_TO_ENABLE_ON_WIFI_DISCONNECT=&quot;&quot;
DEVICES_TO_ENABLE_ON_WWAN_DISCONNECT=&quot;&quot;

Note that in this configuration the battery charge thresholds are intentionally set lower than the factory default to prolong the health of batteries. When I wish to fully charge them, do:

$ sudo tlp fullcharge [ BAT0 | BAT1 ]

and the thresholds will be set back to the factory default. The settings revert to the custom configuration upon next reboot.

Enables RC6 Power Saving

Edit /etc/default/grub and add the option i915.i915_enable_rc6=1 to GRUB:

GRUB_CMDLINE_LINUX_DEFAULT=&quot;quiet i915.i915_enable_rc6=1&quot;

After editing, update Grub:

$ sudo update-grub

This can save about 0.5 W of power!

Fan Control

I have not fine tuned this, but the fan can be controlled manually via thinkfan:

$ sudo aptitude install thinkfan

Create (or edit) /etc/modprobe.d/thinkpad_acpi.conf and add the line:

options thinkpad_acpi fan_control=1

Reboot. The fan can be controlled manually following this Wiki article.

Configuring System on Switch between AC & Battery

For regular daemons (like ssh), TLP provides a way to start/stop based on AC status.

For other personal processes, KDE power management can also do some work by running a script upon dis/connecting with AC. In particular, the sound for the KDE system notifications should be disabled entirely, because they cause quite a number of unwanted interruptions according to powertop. Also, I wish to start/stop programs like flux and Dropbox when on battery just to squeeze more juice. I create a script like this:

#!/usr/bin/env bash
##############################################################################
# This is a script to run when AC/battery switches.  Ideally used
# within the power management configuration setting on KDE.  Requires
# tp_smapi to be loaded for the detection of AC/battery.

USERNAME=johndoe
LATITUDE=37.78
LONGITUDE=-122.42


##############################################################################
# DO NOT MODIFY BELOW

statuspath=&quot;/sys/class/power_supply/AC/online&quot;

if [ ! -f &quot;$statuspath&quot; ]; then
    exit 0
fi

tag=pmpersonal
status=`cat &quot;$statuspath&quot;`

if [ &quot;$status&quot; = &quot;1&quot; ]; then
    logger -t $tag &quot;on ac&quot;

    logger -t $tag &quot;start xflux&quot;
    for i in `pgrep ^xflux$` ; do kill -9 $i ; done
    xflux -l $LATITUDE -g $LONGITUDE

    logger -t $tag &quot;start dropbox&quot;
    dropbox start

    logger -t $tag &quot;enable system notification&quot;
    kwriteconfig --file=&quot;/home/$USERNAME/.kde/share/config/knotifyrc&quot; --group=Sounds --key=&quot;No sound&quot; false
else
    logger -t $tag &quot;on battery&quot;

    logger -t $tag &quot;stop xflux&quot;
    for i in `pgrep ^xflux$` ; do kill -9 $i ; done

    logger -t $tag &quot;stop dropbox&quot;
    dropbox stop

    logger -t $tag &quot;disable system notification&quot;
    kwriteconfig --file=&quot;/home/$USERNAME/.kde/share/config/knotifyrc&quot; --group=Sounds --key=&quot;No sound&quot; true
fi

Save this to a path like /bin/pmpersonal.sh. Go to System Settings -> Power Management -> Energy Saving Settings. There, for each of “On AC,” “On Battery,” and “On Low Battery,” I can set different configurations. I can also specify “Run Script” option for each profile to enable/disable a few desktop related applications and settings. The script should be run on profile load.

Preload Daemon

$ sudo aptitude install preload

In /etc/default/preload, uncomment the line reading

OPTIONS=&quot;-l /dev/null&quot;

This should reduce disk activity.

Disabling Nepomuk, Strigi and Akonadi on KDE

I am not fond of these daemons. Disable them by going to System Settings -> Desktop Search and unchecking “Enable Nepomuk Semantic Desktop.” In ~/.config/akonadi/akonadiserverrc, set:

StartServer=false

so that it doesn’t start at all. On powertop, this shows up as a service running mysqld constantly.

Installing Firefox on Debian Wheezy

Sat, 22 Sep 2012 00:00:00 +0000

I don’t like to get stuck with a very old version of browser, so I install a Firefox binary provided by Mozilla, not the one by Debian.

Make sure to download a 64-bit version (as I use amd64) from here. The tarball expands to a directory named firefox. For example:

$ cd /usr/local/src
$ sudo wget http://releases.mozilla.org/pub/mozilla.org/firefox/releases/latest/linux-x86_64/en-US/firefox-15.0.1.tar.bz2
$ sudo tar -xvjf firefox-15.0.1.tar.bz2
$ sudo mv firefox ..
$ cd /usr/local/bin
$ sudo ln -s ../firefox/firefox .

This makes firefox command available system-wide.

To install flash, do:

$ sudo aptitude install flashplugin-nonfree

To install Java plugin, do:

$ sudo aptitude install icedtea-7-plugin

Using Japanese on Debian Wheezy

Fri, 21 Sep 2012 00:00:00 +0000

The goal is to make the system capable for Japanese input, while letting the base system remain English. For the Japanese input method, I had been using Anthy, but I will be using mozc, which is now better supported and presumably much better (it is).

First, set up locales and install im-config. In addition to my base English locale (en_US.UTF-8), install Japnese locales ja_JP.EUC-JP and ja_JP.UTF-8:

$ sudo aptitude install locales im-config
$ sudo dpkg-reconfigure locales

Set the default locale for the system to en_US.UTF-8.

Install some Japanese fonts:

$ sudo aptitude install ttf-kochi-mincho ttf-kochi-gothic ttf-sazanami-mincho

IBus

Install IBus-related packages and configure for KDE:

$ sudo aptitude install ibus ibus-mozc ibus-qt4 ibus-gtk ibus-gtk3 ibus-clutter
$ im-config
... choose IBus ...

Run the IBus setup program:

$ ibus-setup

If asked, start ibus-daemon. Log out of the KDE session, restart the X session, and log in again. Under the Input Method tab, add “Japanese – Mozc”. Configure IBus to my liking, such as making Ctrl + Space (as opposed to Shift + Space) trigger the Japanese input mode. The IBus icon also shows up in the system tray, from which various settings can be changed.

Mozc

Run

$ sudo aptitude install ibus-mozc mozc-utils-gui
$ /usr/lib/mozc/mozc_tool --mode=config_dialog

to configure Mozc to my liking.

Using Japanese in Emacs

Install ibus-el:

$ sudo aptitude install ibus-el

Within emacs, M-x ibus-mode will toggle IBus on and off. Very easy!

Searching for Nearest-Neighbors between Two Coordinate Catalogs

Thu, 28 Jun 2012 00:00:00 +0000

Say I have two catalogs of points, each in two-dimensional space. For each object in a catalog, I want to find the nearest object(s) in the other catalog. I can do this by computing the distances between every single unique pair of objects and finding the ones within a search radius and possibly doing an additional sort. However, there is a much more efficient way using k-d tree, and Scipy has the class for doing this sort of thing fairly easily.

Here is a Python code example:

#!/usr/bin/env python2.6
from cProfile import Profile
import numpy as np
from numpy.random import random
from scipy.spatial import KDTree, cKDTree


def getnnidx(d1, d2, r):
    t1 = KDTree(d1)
    t2 = KDTree(d2)
    idx = t1.query_ball_tree(t2, r)
    return idx


def cgetnnidx(d1, d2, r, k=5):
    t = cKDTree(d2)
    d, idx = t.query(d1, k=k, eps=0, p=2, distance_upper_bound=r)
    return idx


def main():
    # number of points
    np1 = 4000
    np2 = 2000

    # search radius
    r = 0.01

    # prepare coordinates; the input data for the constructor of
    # KCTree needs to be in the form:
    #
    #   data = [[x0, y0], [x1, y1], ... , [xN, yN]]
    #
    d1 = np.empty((np1, 2))
    d2 = np.empty((np2, 2))
    d1[:, 0] = random(np1)
    d1[:, 1] = random(np1)
    d2[:, 0] = random(np2)
    d2[:, 1] = random(np2)

    # profile two versions of KDTree implementations
    p = Profile()

    result = p.runcall(getnnidx, d1.copy(), d2.copy(), r)
    p.print_stats()

    p.clear()

    result = p.runcall(cgetnnidx, d1.copy(), d2.copy(), r)
    p.print_stats()


if __name__ == '__main__':
    main()

When I run this script, I get the following output:

       706932 function calls (692470 primitive calls) in 1.663 CPU seconds

 Ordered by: standard name

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   1720    0.001    0.000    0.004    0.000 fromnumeric.py:1067(nonzero)
      2    0.000    0.000    0.000    0.000 fromnumeric.py:1147(shape)
  42293    0.046    0.000    0.224    0.000 fromnumeric.py:1314(sum)
      2    0.000    0.000    0.000    0.000 fromnumeric.py:1708(amax)
      2    0.000    0.000    0.000    0.000 fromnumeric.py:1769(amin)
    860    0.001    0.000    0.002    0.000 fromnumeric.py:664(argmax)
  14462    0.008    0.000    0.025    0.000 function_base.py:776(copy)
      2    0.000    0.000    0.045    0.022 kdtree.py:113(__init__)
    862    0.001    0.000    0.001    0.000 kdtree.py:139(__init__)
    860    0.001    0.000    0.001    0.000 kdtree.py:143(__init__)
 1722/2    0.033    0.000    0.045    0.022 kdtree.py:150(__build)
  42293    0.183    0.000    0.992    0.000 kdtree.py:22(minkowski_distance)
  12744    0.085    0.000    0.104    0.000 kdtree.py:36(__init__)
      1    0.002    0.002    1.618    1.618 kdtree.py:437(query_ball_tree)
12743/1    0.270    0.000    1.616    1.616 kdtree.py:462(traverse_checking)
   6371    0.031    0.000    0.157    0.000 kdtree.py:49(split)
  42293    0.372    0.000    0.682    0.000 kdtree.py:7(minkowski_distance_p)
  12743    0.133    0.000    0.430    0.000 kdtree.py:72(min_distance_rectangle)
   7385    0.052    0.000    0.219    0.000 kdtree.py:76(max_distance_rectangle)
 169174    0.095    0.000    0.213    0.000 numeric.py:201(asarray)
      1    0.000    0.000    1.663    1.663 xmatch.py:8(getnnidx)
  57063    0.036    0.000    0.036    0.000 {isinstance}
   5164    0.001    0.000    0.001    0.000 {len}
    860    0.001    0.000    0.001    0.000 {method 'argmax' of 'numpy.ndarray' objects}
  25488    0.019    0.000    0.019    0.000 {method 'astype' of 'numpy.ndarray' objects}
      1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
      2    0.000    0.000    0.000    0.000 {method 'max' of 'numpy.ndarray' objects}
      2    0.000    0.000    0.000    0.000 {method 'min' of 'numpy.ndarray' objects}
   1720    0.003    0.000    0.003    0.000 {method 'nonzero' of 'numpy.ndarray' objects}
  42293    0.146    0.000    0.146    0.000 {method 'sum' of 'numpy.ndarray' objects}
  22165    0.008    0.000    0.008    0.000 {method 'tolist' of 'numpy.ndarray' objects}
      2    0.000    0.000    0.000    0.000 {numpy.core.multiarray.arange}
 183636    0.135    0.000    0.135    0.000 {numpy.core.multiarray.array}
      1    0.000    0.000    0.000    0.000 {range}


       32 function calls in 0.008 CPU seconds

 Ordered by: standard name

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      3    0.000    0.000    0.000    0.000 fromnumeric.py:107(reshape)
      3    0.000    0.000    0.000    0.000 fromnumeric.py:1147(shape)
      1    0.000    0.000    0.000    0.000 fromnumeric.py:1708(amax)
      1    0.000    0.000    0.000    0.000 fromnumeric.py:1769(amin)
      1    0.000    0.000    0.000    0.000 fromnumeric.py:1865(prod)
      1    0.000    0.000    0.000    0.000 fromnumeric.py:32(_wrapit)
      2    0.000    0.000    0.000    0.000 numeric.py:201(asarray)
      4    0.000    0.000    0.000    0.000 numeric.py:314(ascontiguousarray)
      1    0.000    0.000    0.008    0.008 xmatch.py:15(cgetnnidx)
      1    0.000    0.000    0.000    0.000 {getattr}
      1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
      1    0.000    0.000    0.000    0.000 {method 'max' of 'numpy.ndarray' objects}
      1    0.000    0.000    0.000    0.000 {method 'min' of 'numpy.ndarray' objects}
      1    0.000    0.000    0.000    0.000 {method 'prod' of 'numpy.ndarray' objects}
      1    0.007    0.007    0.007    0.007 {method 'query' of 'scipy.spatial.ckdtree.cKDTree' objects}
      3    0.000    0.000    0.000    0.000 {method 'reshape' of 'numpy.ndarray' objects}
      6    0.000    0.000    0.000    0.000 {numpy.core.multiarray.array}

from which it is clear that cKDTree is much faster (1.663 vs. 0.008 seconds!). Not surprising given cKDTree is a C implementation of basically the same thing.

These classes are not exactly equivalent and therefore the outputs are different. KDTree.query_ball_tree returns:

[[],
 [216, 1767],
 [317],
 ...,
 [],
 [367],
 [1465, 1899]]

which is an array of indices of nearest neighbors. If an object is not found within the search radius, it simply returns an array of zero elements.

cKDTree.query, on the other hand, returns something like

[[2000 2000 2000 2000 2000]
 [ 216 1767 2000 2000 2000]
 [ 317 2000 2000 2000 2000]
 ...,
 [2000 2000 2000 2000 2000]
 [ 367 2000 2000 2000 2000]
 [1465 1899 2000 2000 2000]]

which is an array of indices of five nearest neighbors. When not found within a search radius, it uses the number of elements as a place holder. The number of nearest neighbors to retain can be specified by the input argument (k).

Installing Ubuntu (12.04 LTS 64-bit) from an ISO Image on Hard Drive

Mon, 14 May 2012 00:00:00 +0000

Sometimes it’s useful to be able to install Linux off an ISO image on hard drive for whatever reason. My Dell XPS 630i has had some issues when I was installing Debian off an install CD/USB, likely related to how GRUB 2 does not gracefully deals with SATA configuration particular to this machine (the install process doesn’t appear to recognize /dev/sdaX properly at some point, ending up in IO errors or kernel panic). In any case, I could solve this by installing Linux off a pre-partitioned hard drive. This time, I install Ubuntu. (I’m giving my XPS 630i away before moving back to the States!)

Here goes the procedure. I assume that I am installing Ubuntu from scratch and hard drive has been formatted.

First, I need a Linux Live media, whatever from which I can boot the computer. I burn a CD for this particular post. Go to the Ubuntu website, download an ISO image (here ubuntu-12.04-desktop-amd64.iso), and burn a Live CD.

Boot up the computer off Ubuntu Live (do not install at this point!) and prepare the disk partitions, using GParted. It would be best to create full partition scheme at this point, so during the installation I can simply format them without repartitioning. For Ubuntu, the root partition (/) must be at least 2.5 GB, otherwise the installer may complain. Typically, a /boot partition needs to be about 300 MB or so, but I store all my ISO files under /boot/iso, so I will use 3 GB for the partition just in case. (For an Ubuntu ISO image, 700 MB may be enough but for Linux Mint, I may need 2 GB, for example; I can also store multiple ISO images if I wish.) Here’s the partition scheme:

Filesystem            Size   Mounted on
/dev/sda5               3G   /
/dev/sda6               3G   /boot
/dev/sda7              10G   /tmp
/dev/sda8               6G   /var
/dev/sda9              10G   /usr
/dev/sda10             20G   /usr/local
/dev/sda11              8G   Linux Swap
/dev/sda12            690G   /home

Then on Terminal I install GRUB 2 on /dev/sda:

$ sudo mount /dev/sda5 /mnt
$ sudo mkdir /mnt/boot
$ sudo mount /dev/sda6 /mnt/boot
$ sudo grub-install --root-directory=/mnt /dev/sda
$ sudo mkdir /mnt/boot/boot-isos
$ sudo cp ~/Downloads/ubuntu-12.04-desktop-amd64.iso /mnt/boot/boot-isos
$ sudo umount /mnt/boot
$ sudo umount /mnt

I assumed that the ISO image was already downloaded to ~/Downloads directory. Now, reboot and on the GRUB prompt, follow this to boot off the ISO image:

$ set prefix=(hd0,6)/grub
$ insmod linux
$ insmod loopback
$ insmod iso9660
$ loopback loop (hd0,6)/boot-isos/ubuntu-12.04-desktop-amd64.iso
$ set root=(loop)
$ linux /casper/vmlinuz boot=casper iso-scan/filename=/boot-isos/ubuntu-12.04-desktop-amd64.iso noprompt noeject toram
$ initrd /casper/initrd.lz
$ boot

The machine should boot into Ubuntu Live. Once in Ubuntu, I should unmount /isodevice:

$ sudo umount -l -r -f /isodevice

Then I can simply run the installer on Ubuntu. In order to prevent the installer from stalling (see this bug report, I had to format all partitions except /boot. But otherwise everything should go fine.

** Reference

HOWTO: ISO Booting with Grub 2

HOWTO: Boot & Install Ubuntu from the Grub Rescue Prompt

Half-Light Radii for Various Profiles

Mon, 14 Feb 2011 00:00:00 +0000

For a radial profile of $I( r)$, the enclosed flux within the radius $r$ is given by

\begin{equation*} F( r) = \int_{0}^{2 \pi} d\phi \int_{0}^{r} dr r I(r, \phi) \ . \end{equation*}

I’m only concerned about azimuthal symmetric cases, so $F( r) = 2\pi \int_{0}^{r} dr r I( r)$ . The idea is to solve

\begin{equation*} \frac{ F( r_{ 1 / 2 } ) }{ F( \infty ) } = \frac{1}{2} \end{equation*}

for $r_{1/2}$.

Gaussian Profile

Say the Gaussian intensity profile is given by

\begin{equation*} I( r) = I_0 \exp{\left(-\frac{r^2}{2\sigma^2}\right)} \ . \end{equation*}

The enclosed flux within $r$ is given by

\begin{equation*} F( r) = 2\pi I_0 \int_0^r dr r \exp{\left(-\frac{r^2}{2\sigma^2}\right)} = 2\pi I_0 \sigma^2 \left[1 - \exp{\left(-\frac{r^2}{2\sigma^2}\right)}\right]\ . \end{equation*}

Hence

\begin{equation*} \boxed{ r_{1/2} = \sqrt{-2 \ln{0.5}} \sigma \ . } \end{equation*}

Exponential Profile

The radial surface-brightness profile $I( r)$ of a disk galaxy is often fitted by the exponential profile:

\begin{equation*} I( r) = I_d \exp{\left(-\frac{r}{r_d}\right)} \ , \end{equation*}

where $r_d$ is the disk scale length. The flux enclosed within the radius $r$ is given by

\begin{equation*} F( r) = 2\pi I_d \int_0^r dr r \exp{\left(-\frac{r}{r_d}\right)} \ . \end{equation*}

Solving for the half-light radius $r_{1/2}$ leads to the following:

\begin{equation*} \left(1 + \frac{r_{1/2}}{r_d}\right) \exp{\left(-\frac{r_{1/2}}{r_d}\right)} = \frac{1}{2} \ . \end{equation*}

A relevant numerical solution is

\begin{equation*} \boxed{r_{1/2} = 1.67835 r_d \ .} \end{equation*}

Sersic/de Vaucouleurs Profile

The radial surface-brightness profile $I( r)$ of a spheroidal galaxy is often fitted by the Sersic (de Vaucouleurs when $n = 4$ profile):

\begin{equation*} I( r) = I_{1/2} \exp{\left( -k \left( \left(\frac{r}{r_{1/2}}\right)^{1/n} - 1 \right) \right)} \ . \end{equation*}

where $k = 1.992 n - 0.3271$ (Capaccioli 1989). This expression explicitly parametrizes the profile in terms of half-light radius.

Installing Debian Squeeze from Hard Disk

Tue, 14 Dec 2010 00:00:00 +0000

Unfortunately the Debian installer does not work on my Dell desktop box. I still wanted to do a fresh install of Squeeze, and the only sensible option for me was to install off hard disk. I have been having the same issue for several months and I have not been able to solve it myself, and nobody responds to my bug report; however, this hard drive method may work well even if I have a working install USB stick/CD media.

I assume that a working Linux installation exists on the system already; it will only be used to launch the installer. (In practice, I used an existing Lenny installation to go through this procedure, only to wipe it over with Squeeze.) For this note I use the Debian installer beta 2 for Squeeze (AMD64).

Create a partition large enough to hold a netinst .iso image (~ 170 MB) plus maybe 20 MB for kernel image and whatnot. For illustration, I assume it is at /dev/sda11. Move to a temporary working directory, download some files, mount the partition, and copy files to the install partition:

$ wget http://ftp.nl.debian.org/debian/dists/testing/main/installer-amd64/current/images/hd-media/initrd.gz
$ wget http://ftp.nl.debian.org/debian/dists/testing/main/installer-amd64/current/images/hd-media/vmlinuz
$ wget http://cdimage.debian.org/cdimage/squeeze_di_beta2/amd64/iso-cd/debian-squeeze-di-beta2-amd64-netinst.iso
$ mkdir mnt
$ sudo mount /dev/sda11 mnt
$ sudo cp initrd.gz mnt
$ sudo cp vmlinuz mnt
$ sudo cp debian-squeeze-di-beta2-amd64-netinst.iso mnt/netinst.iso
$ sudo umount /dev/sda11

Now I need to modify the GRUB menu to add an entry for the installer that I just made.

For the new GRUB version 2, an entry in /boot/grub/grub.cfg looks a bit complicated:

menuentry 'Squeeze Installer Beta 2' --class debian --class gnu-linux --class os {
  insmod part_msdos
  insmod ext2
  set root='<em>(hd0,msdos11)</em>'
  search --no-floppy --fs-uuid --set <em>SOMELENGTHYHASH</em>
  echo 'Loading Squeeze installer'
  linux /vmlinuz root=UUID=<em>SOMELENGTHYHASH</em>
  echo 'Loading init ramdisk'
  initrd /initrd.gz
}

Here, the root in which the install partition is the eleventh partition of the first hard drive (hence hd0, msdos11). With the lengthy UUID hash for the hard drive, I’d just copy one of the GRUB entries for working installations.

For an old version of GRUB, an entry in /boot/grub/menu.lst would look like this:

title  Squeeze Installer Beta 2
root   (hd0,10)
kernel /vmlinuz
initrd /initrd.gz

This means the install partition is located at the 11th partition of the first hard drive, and the paths to the kernel and initrd images are located at the topmost level of the partition. This is the sort of entry I copy and paste from other entries in /boot/grub/menu.lst.

After reboot, select the menu entry just added and I will be starting the installer right away!

IMPORTANT: When I partition the drive during the installation process, do not mount the install partition. If I do, the partitioning and the subsequent installation will fail.

Installing F.lux on Debian Squeeze

Sat, 20 Nov 2010 00:00:00 +0000

In another feeble attempt to become a “morning person,” I’ve decided to follow the usual advice that I should not be looking at bright LCD screen at night to help myself going into the “sleep mode.”

Hence F.lux.

I want it to launch automatically at the start of a KDE session.

First of all, check the geographical coordinates of your location, perhaps using a search engine like this one. In my case (Halifax, Canada), the latitude and longitude is 44.65 and -63.6 (in degrees).

Download the Linux binary (i.e., the command-line version) from here. As of now, this is just a binary and no source codes. So no compiling is necessary. Do as:

$ wget http://secure.herf.org/flux/xflux.tgz
$ tar -xvzf xflux.tgz
$ sudo cp xflux /usr/local/bin
$ sudo chmod 755 /usr/local/bin/xflux

Now the command xflux is available system-wide.

To make it run at the KDE starup, create a shell script (called here xflux.sh) like this one:

#!/bin/sh
xflux -l 44.65 -g -63.6

Either place this script manually under ~/.kde/Autostart, or go to KDE menu -> Computer -> System Settings -> Advanced -> Autostart, and add the xflux.sh script there.

If for some reason you need to stop this program, do:

$ kill -9 `pgrep xflux`

Running F.lux on 64-bit Debian Squeeze

I may have problem getting xflux to run at all on a 64-bit machine, possibly seeing a xflux: No such file or directory error. The xflux binary is built against 32-bit libraries:

$ file xflux
xflux: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux 2.6.15, not stripped

so if I haven’t done so, I need to install 32-bit environment:

$ sudo aptitude install ia32-libs

After this, I should be able to run xflux without problems.

Installing Bruzual & Charlot 2003 (GALAXEV) on Debian Squeeze

Tue, 19 Oct 2010 00:00:00 +0000

The installation procedure is confusing only if you are not using csh (I use Bash) and if g77 is not available. Otherwise you can simply follow the installation procedure outlined in the official documentation file (i.e., bc03/doc/bc03.ps.gz).

First, you definitely need csh and a FORTRAN compiler:

$ sudo apt-get install csh gfortran

Download from the B&C 2003 website the following tarballs (assuming here they are saved under /usr/local/SED):

bc03.doc.tar.gz
bc03.models.padova_1994_chabrier_imf.tar.gz
bc03.models.padova_1994_salpeter_imf.tar.gz
bc03.src.tar.gz
bc03.templates.tar.gz

Uncompress and untar all these at once by

for i in *.tar.gz; do tar -xvzf $i; done

This will create the directory bc03 under /usr/local/SED and all the subdirectories (bc03/doc, bc03/models, bc03/src, bc03/templates) should be filled with files.

Unfortunately the setup file only exists for csh (bc03/src/.bc_cshrc), so you need to create one for yourself. Save the following to a file at bc03/src/.bc_bash:

export FILTERS="$bc03/FILTERBIN.RES"
export A0VSED="$bc03/A0V_KURUCZ_92.SED"
export RF_COLORS_ARRAYS="$bc03/RF_COLORS.filters"
####################################################
alias  add='csh $bc03/add_bursts.sh'
alias  csp='csh $bc03/csp_galaxev.sh'
alias  vdisp='csh $bc03/vel_disp.sh'
alias  cmev='csh $bc03/cm_evolution.sh'
alias  dgr='csh $bc03/downgrade_resolution.sh'
alias  gpl='$bc03/galaxevpl'
alias  zmag='$bc03/zmag'

(This file is actually provided by default in C&B 2007, which you should be using instead of the 2003 version.) Furthermore, you need to modify a line in bc03/src/Makefile to use gfortran instead of g77:

#FC = g77 -ffixed-line-length-132  # comment out this line and add:
FC = gfortran -ffixed-line-length-none

Then compile the programs:

$ bc03=/usr/local/SED/bc03/src
$ cd $bc03
$ . .bc_bash
$ make all

Add the following lines to ~/.bashrc and all the commands for GALAXEV will be available (as aliases):

export bc03=/usr/local/SED/bc03/src
. $bc03/.bc_bash

That’s it.

Testing if a Point is Inside a Polygon in Python

Thu, 24 Jun 2010 00:00:00 +0000

Finally got around to finding this out by Googling. It’s a useful function so I reproduce it here for copy & paste:

def inside_polygon(x, y, points):
    """
    Return True if a coordinate (x, y) is inside a polygon defined by
    the list of verticies [(x1, y1), (x2, y2), ... , (xN, yN)].

    Reference: http://www.ariel.com.au/a/python-point-int-poly.html
    """
    n = len(points)
    inside = False
    p1x, p1y = points[0]
    for i in range(1, n + 1):
        p2x, p2y = points[i % n]
        if y > min(p1y, p2y):
            if y <= max(p1y, p2y):
                if x <= max(p1x, p2x):
                    if p1y != p2y:
                        xinters = (y - p1y) * (p2x - p1x) / (p2y - p1y) + p1x
                    if p1x == p2x or x <= xinters:
                        inside = not inside
        p1x, p1y = p2x, p2y
    return inside

It is a clever algorithm if you think through it.

Installing Kindle for PC on Debian Squeeze

Fri, 11 Jun 2010 00:00:00 +0000

I sometimes want to run Kindle for PC on Wine. Amazon is so lame that they don’t even bother making a native Linux version of this software!

Here is the procedure. Move to a temporary directory and do:

$ sudo aptitude install wine lib32nss-mdns cabextract bzip2
$ wget http://www.kegel.com/wine/winetricks
$ sh winetricks corefonts
$ wget ./wp-content/uploads/2010/06/KindleForPC-installer.exe.bz2
$ bunzip2 KindleForPC-installer.exe.bz2
$ wine KindleForPC-installer.exe

After going through the above procedure, I need to run Wine Configuration to set Kindle for PC executable to run as a Windows 98 application. I do this under the Applications tab of the Wine configuration. It should be self-explanatory.

The program called winetricks is necessary to install necessary MS fonts once Kindle for PC launches.

The old version of Kindle for PC installer, which I just keep for myself here, is necessary to properly install it on Wine. Unfortunately newer versions won’t work. Hopefully Amazon releases a new version which runs at least on Wine.

Update History

Jan 26, 2011 – Fixing typos and added a couple packages based on a user comment.

Installing Debian Squeeze (AMD64) on Lenovo T410s

Tue, 01 Jun 2010 00:00:00 +0000

So far the only thing that I do not like about T410s is its short battery life. I can only get 2 to 3 hours out of “standard” usage. Hopefully this will improve as I tweak things more.

To Begin With: A Problem

Unfortunately I had a major issue with my T410s recently (November 17, 2010). A few months ago, the LCD started showing a vertical band in which graphics was lost. the issue seems to be fairly prevalent with T410s, which means there was a solution. So, I just got a warranty repair done, which involved replacing the LCD and motherboard. It took exactly two weeks for the whole process using the local Lenovo contractor (in Canada), but now the laptop is fixed. Anyone who got T410s around June 2010 or before should be aware of this LCD issue.

As of November 26, 2010, I decided to reinstall Debian Squeeze from scratch. So this installation note now reflects the nearly stable version of Squeeze installation procedure. Things that did not work before now works from the beginning!

General Hardware Specifications

My Lenovo T410s is 2901-CTO.

Hardware Components	Status under Linux	Notes
Intel Core i5 M520 2.4 GHz	Works
14.1 WXGA+ TFT Display, w/ LED Backlight	Works
Intel HD Graphics (IT 5700MHD, i5-520M AMT)	Works
4 GB PC3-8500 DDR3 (2 DIMM)	Works
80 GB SSD (INTEL SSDSA1M080G2LE, 2CV102J6, max UDMA/133)	Works
Integrated Ethernet Card (Intel 82577LM Gigabit Network Connection (rev 06))	Works
Wireless Network Card (Intel Centrino Ultimate-N 6300 (rev 35))	Works	Need to download firmware to /lib/firmware.
Matsushita DVD-RAM Drive (UJ892)	Works
6 cell Prismatic Battery	Works
Integrated Sound Card (Intel 3b57 (rev 06))	Works
Integrated Speaker & Microphone	Works
Integrated Camera	Works	uvcvideo
ThinkLight	Works
Fingerprint Reader	Not tested
TrackPoint & TrackPad	Works
Keyboard	Not fully tested	Haven’t thoroughly tested special keys

Note: Currently my T410s is operating on kernel version 2.26.36.2 from kernel.org which I customized and compiled myself. I don’t think is necessary to get the basic things working though. On the other hand, I see some problem with kernel 2.26.36.2. When I do dmesg, I get numerous scsi host1: __pm_runtime_resume()! errors, and sometimes T410s locks up for several seconds. In order to alleviate the problem, I have to boot with kernel option pcie_ports=compat.

Why Debian Squeeze?

I have always liked the leanness and stability of Debian, but I admit that I have recently flirted with the idea of migrating to Ubuntu; near the end of Debian release cycle, I have always found myself building from sources various programs for which their Debian versions became obsolete. When I tried, I also liked Ubuntu for its ease of use; I didn’t have to do whole a lot of configurations myself to get my old laptop (a Dell Latitude) working. However, it slightly felt like using Windows on a new computer; a lot of software that I don’t care to install are bundled with Ubuntu. In the end, I decided to stay with Debian for now, but with Squeeze, still a testing distribution (Nov. 17, 2010).

Installing Debian Squeeze

First of all, make factory recovery disks in case you need to restore factory default. There is little reason to keep Windows 7 around, but it’s always a good thing to have a way to restore default setting. In order to do this, launch ThinkVantage and go to Factory Recovery Disks. Check both boot and data. In my case, three blank DVDs were used (one for boot and two for data; it looks like the boot disk only needs to be of very small capacity). After making the disks, it is recommended to test if they work or not by rebooting off the boot disk just created. Mine passed the test and factory reset was completely working. I also updated BIOS to its latest version on Windows to avoid problems later.

Debian Squeeze will soon become the stable distribution, but till that happens I fetched the installer from the testing distribution site, and made a install USB stick, following this article. For this installation note, I use the Beta1 release of Squeeze.

Boot off the USB stick by pressing F12 to start the installation.

Installation Customization Options

Debian installation has gotten easy enough that I don’t feel the need for this any more, but just for completeness. Obviously you should customize to your liking. I use a graphical installer here.

Language: English
Country, territory or area: (your choice)
Keyboard layout: USA
Primary network interface: wlan0 (for wireless network installation, only WEP or manual appears working, for which you specify the wireless key and password perhaps.) or eth0 (wired network installation is often easier)
Hostname: (your choice)
Domain name: (your choice)
Root password: (your choice)
Full name for the new user: (your choice)
Username for your account: (your choice)
Choose a password for the new user: (your choice)
Select your time zone: (your choice)
Partitioning method: Manual

I’m not doing dual boot, so my partition scheme is very simple:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             323M  149M  157M  49% /
/dev/sda6              46M   27M   17M  62% /boot
/dev/sda7             4.6G  138M  4.3G   4% /tmp
/dev/sda8             2.8G  1.8G  841M  69% /var
/dev/sda9             5.5G  4.4G  914M  83% /usr
/dev/sda10            6.9G  4.4G  2.2G  67% /usr/local
/dev/sda11              8G                  Linux Swap
/dev/sda12             46G   19G   26G  43% /home

Policy for handling keymaps: Select keymap from arch list -> qwerty -> US american -> Standard -> Standard
Debian archive mirror country: (your choice)
Participate in the package usage survey: No
Choose software to install: (I want a fairly minimal install. I only check Laptop.)
Install the GRUB boot loader to the master boot record: yes

Reboot and a very simple Debian box is ready!

Post-Install Admin Tweaks

# aptitude install sudo ssh rsync wget wireless-tools

If I need to run 32-bit applications, it is also a good idea to do:

# aptitude install ia32-libs

Setting up `sudo`

At this point I add myself as an sudoer by adding my regular username to /etc/sudoers, just under the entry for root, giving full admin privileges (i.e., just copy the line for root).

# chmod 640 /etc/sudoers
# emacs -nw /etc/sudoers
... add a regular user to the list ...
# chmod 440 /etc/sudoers

I believe in using sudo rather than becoming root to do admin tasks.

Setting up APT Sources

It is also a good idea to modify /etc/apt/sources.list now. Mine looks like this:

deb http://ftp.us.debian.org/debian/ squeeze main contrib non-free
deb-src http://ftp.us.debian.org/debian/ squeeze main contrib non-free

deb http://security.debian.org/ squeeze/updates main contrib non-free
deb-src http://security.debian.org/ squeeze/updates main contrib non-free

deb http://www.debian-multimedia.org squeeze main non-free

As I see, the debian-multimedia.org repository is added. Though this isn’t absolutely essential, I may wish to do

# wget http://www.debian-multimedia.org/pool/main/d/debian-multimedia-keyring/debian-multimedia-keyring_2008.10.16_all.deb
# dpkg -i debian-multimedia-keyring_2008.10.16_all.deb
# aptitude update

in order to avoid GPG error accessing this repository in future.

Setting up DHCP Hostname

Modify /etc/dhcp/dhclient.conf to have the line:

send host-name "yourcomputershostname";

This way, identify this computer on a router, for example, will be easier.

Intel Centrino Ultimate-N 6300

The built-in kernel for Squeeze (2.6.32) appears to have the driver for this WiFi card, but firmware need to be loaded:

$ sudo aptitude install firmware-iwlwifi

Or alternatively I can download firmware, and manually copy the file iwlwifi-6000-4.ucode to /lib/firmware. For example:

$ wget http://intellinuxwireless.org/iwlwifi/downloads/iwlwifi-6000-ucode-9.221.4.1.tgz
$ tar -xvzf iwlwifi*.tgz
$ sudo cp iwlwifi-6000-ucode-9.221.4.1/iwlwifi-6000-4.ucode /lib/firmware

Either way, WiFi should be working after reboot.

Integrated Webcam

To check if the webcam works, the easiest is to use Kopete on KDE (Menu -> Applications -> Internet -> Kopete). Go to Settings -> Configure -> Video, and see if the preview works.

KDE

To install KDE,

# aptitude update
# aptitude install kdebase kdenetwork kmix
# reboot

Upon reboot, the KDM login screen appears.

KDE Network Manager

If I don’t see any network manager on the system tray of the bottom bar, click on the system tray (this one is located to the left of the clock by default). Right click -> System Tray Settings -> Plasma Widgets and check Network Management. Try to add a wireless access point.

Suspend and Resume

I have always been a type who turns off computer when not used, but I am now using suspend/resume frequently. Fortunately I have few problems using suspend to RAM/disk on T410s so far.

TrackPoint Scrolling

I can use the red TrackPoint for scrolling as well! There are several ways to do this, but I’m going to use xinput. If it is not installed, do

$ sudo aptitude install xinput

See what ID is associated with TrackPoint:

$ xinput list
⎡ Virtual core pointer                          id=2    [master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer                id=4    [slave  pointer  (2)]
⎜   ↳ SynPS/2 Synaptics TouchPad                id=11   [slave  pointer  (2)]
⎜   ↳ TPPS/2 IBM TrackPoint                     id=13   [slave  pointer  (2)]
⎣ Virtual core keyboard                         id=3    [master keyboard (2)]
    ↳ Virtual core XTEST keyboard               id=5    [slave  keyboard (3)]
    ↳ Power Button                              id=6    [slave  keyboard (3)]
    ↳ Video Bus                                 id=7    [slave  keyboard (3)]
    ↳ Sleep Button                              id=8    [slave  keyboard (3)]
    ↳ Integrated Camera                         id=9    [slave  keyboard (3)]
    ↳ AT Translated Set 2 keyboard              id=10   [slave  keyboard (3)]
    ↳ ThinkPad Extra Buttons                    id=12   [slave  keyboard (3)]
    ↳ ACPI Virtual Keyboard Device              id=14   [slave  keyboard (3)]

The TrackPoint’s ID is 13 in my current configuration. Then add the following lines in ~/.xsessionrc:

xinput set-int-prop 13 "Evdev Wheel Emulation" 8 1
xinput set-int-prop 13 "Evdev Wheel Emulation Button" 8 2
xinput set-int-prop 13 "Evdev Wheel Emulation Axes" 8 6 7 4 5

After restarting X, moving TrackPoint while pressing the blue middle button acts as scroll, either vertically or horizontally.

Now that TrackPoint can be used for scrolling, I see little reason to enable TrackPad, as I never use it. It may be a good idea to disable TrackPad in BIOS to save power consumption as well as to prevent accidentally touching it with my hands while typing. Note that after disabling TrackPad, ID (see above) may change, so ~/.xsessionrc may need to be rewritten.

Power Consumption Tips

I now have a separate article, Power Management on Lenovo T410s, for this topic.

`snd_pcsp` Interrupts

Under certain conditions (which I cannot always reproduce), the system becomes sluggish after playing sounds off Flash on Firefox, for example, and until KDE restarts, sounds are not working. Looking at the processes with powertop, I found some process using snd_pcsp tends to make frequent interrupts. A solution to this problem is to ensure that the snd_pcsp kernel module gets loaded after snd_hda_intel gets loaded by feeding index=2 option. Make sure that the file /etc/modprobe.d/snd-pcsp has a line like this:

options snd-pcsp index=2

Installing Applications

Just a list of applications to install on my machine:

Document Update History

June 1, 2010 – First version.

June 6, 2010 – Added a note on suspend and resume.

June 9, 2010 – Added ThinkWiki reference. Added a section on TrackPoint scrolling. Created Power Consumption Tips. Added the preload daemon section.

November 27, 2010 – Major update after a warranty repair of the vertical band issue with LCD.

December 19, 2010 – Moved power consumption tips to another article.

Installing ATLAS on Mac OS X Snow Leopard

Mon, 22 Mar 2010 00:00:00 +0000

Installing from Fink

Do:

$ fink install atlas atlas-shlibs

That’s it. They will be installed under /sw/lib.

Installing from Source (Recommended if Building Other Software That Depends on ATLAS)

First of all, I need a FORTRAN compiler, so install it. I recommend using the Mac installer from the GCC Wiki site, so that GFortran won’t end up having hard-to-track-down dependencies.

I download tarballs for ATLAS and LAPACK from the official repositories. Here I use ATLAS 3.9.32 and LAPACK 3.3.0. I assume the compiled library and header files will be installed at /usr/local/atlas.

Here is the procedure:

$ mkdir tmp
$ cd tmp
$ wget http://www.netlib.org/lapack/lapack.tgz
... download the tarball for ATLAS from sourceforge ...
$ tar -xvjf atlas3.9.32.tar.bz2
$ mv ATLAS atlas-3.9.32
$ ln -s atlas-3.9.32 atlas
$ cd atlas
$ mkdir bld
$ cd bld
$ ../configure -b 64 --shared --prefix=/usr/local/atlas-3.9.32 \
    --with-netlib-lapack-tarfile=../../lapack.tgz
$ make build

If I wish to do some checking, do:

$ make check
...
$ make time
...

(I may wish to compare the output with doc/atlas_install.pdf.)

If I don’t find any problem, do:

$ sudo make install
$ sudo ln -s /usr/local/atlas-3.9.32 /usr/local/atlas

That’s it.

Update History

November 24, 2010 – Added the Fink section. Using the most recent versions of ATLAS and LAPACK. Made the install directory more explicit to the version installed.

Installing GFortran on Mac OS X Snow Leopard

Mon, 22 Mar 2010 00:00:00 +0000

Installing from GFortran Wiki (My Current Preferred Way)

GCC Wiki has a link to Mac OS X installer. As the command gfortran will be installed at /usr/local/gfortran and causes less interference with other libraries and software, this is my preferred way.

Installing from Fink

I use Fink (as opposed to MacPorts) to manage most Linux-like tools.

$ fink install netcdf-gfortran

This will make gfortran available at /sw/bin/gfortran. It may take a bit of time to compile binaries though.

Installing Binary from High Performance Computing for Mac OS X

This site has a bunch of pre-compiled binaries useful for scientific computing, and GFortran is one of them.

Installing Binary from the R Project

You may simply use a binary made available by the R people. Download a .dmg file from here. This version will install at /usr/local/bin.

Update History

November 24, 2010 – Added different install options.

Installing SExtractor on Mac OS X Snow Leopard

Mon, 22 Mar 2010 00:00:00 +0000

(November 25, 2010) I screwed up SExtractor installation for no reason and in an attempt to fix it, I’ve found a few things. After wasting a few hours again, I think this installation note can work for most people who can actually build software from source… I think the usefulness of this note will be limited though, as a new release of SExtractor appears imminent.

Installing from Fink

Fink has a version of SExtractor. I can install this via the command fink install.

$ fink install sextractor

If Fink is already installed, this may be the most trouble free way of installing SExtractor.

Installing from Source

IMPORTANT: I need FFTW and ATLAS installed already, and assume that FFTW was installed at /usr/local/fftw, and ATLAS at /usr/local/atlas following the source install procedures described in those notes. Make sure that you build both double and single precision versions of FFTW by going through the make procedure twice to install both. Build both of them from source, and do not use the libraries from, say, Fink; otherwise dependency hell might arise. So, if I have Fink or Scisoft installed, put them away during the installation to avoid path problems. For example, temporarily rename /sw to /sw.tmp to put Fink away, /usr/local/scisoft to /usr/local/scisoft.tmp to put Scisoft away. (Or alternatively I can “comment out” from your start up shell script those lines that set up Fink and Scisoft, so that environment variables won’t include paths to these things.) Then restart the computer, and start the whole installation process (that is, including FFTW and ATLAS).

Download the SExtractor source into a temporary directory:

$ mkdir tmp
$ cd tmp
... download sextractor-2.8.6.tar.gz
$ tar -xvzf sex*.tar.gz
$ cd sextractor-2.8.6
$ ./configure --prefix=/usr/local/sextractor-2.8.6 \
    --with-atlas=/usr/local/atlas/lib --with-atlas-incdir=/usr/local/atlas/include \
    --with-fftw=/usr/local/fftw/lib --with-fftw-incdir=/usr/local/fftw/include \
    --enable-threads
$ make
$ sudo make install
$ ln -s /usr/local/sextractor-2.8.6 /usr/local/sextractor
$ sudo cp -rp config /usr/local/sextractor

This will install SExtractor at /usr/local/sextractor-2.8.6 (which is aliased by /usr/local/sextractor) and an executable command sex will be located at /usr/local/sextractor/bin/sex. Make a simlink to that binary /usr/local/bin to make it available system-wide. I might also want to make a simlink to /usr/local/sextractor/bin/ldactoasc, which is a utility command that might be useful. The last line copies the SExtractor configuration files and convolution masks to /usr/local/sextractor/config. The default convolution masks are often used so it may be a good idea to keep them under the same installation directory.

Hope this is it!

What if Make Stops with an Error?

There may be a bug in ./configure script that makes the --with-fftw-incdir option to be not honored. If the make stops with errors complaining as in error: fftw3.h: No such file or directory despite that I have installed FFTW properly, copy /usr/local/fftw/include/fftw3.h to the sextractor-2.8.6/src directory before doing make. It will compile fine then.

Update History

November 25, 2010 — Made the procedure description a bit more elaborate where things may fail.

Installing Squid Proxy Server on Mac OS X Snow Leopard

Mon, 22 Mar 2010 00:00:00 +0000

My goal is to set up a very basic proxy server on my Mac box on campus, so that I can have full access to subscription-based academic journals via the proxy on my laptop even when I am off campus. Some schools provide such (library) proxies but my school unfortunately does not. I want to set it up such that the proxy requires a password authentication in order not to make it wide open to the public.

Getting and Installing Squid

Download a tarball for a stable version from the repository. The version that I use here is 2.7. I assume the file is downloaded to /usr/local/src/squid.

$ cd /usr/local/src/squid
$ gunzip -c squid-2.7.STABLE9.tar.gz | tar xvf -
$ cd squid-2.7.STABLE9
$ ./configure
$ make
$ sudo make install
$ cd helpers/basic_auth/NCSA
$ make
$ sudo make install
$ sudo chown -R nobody /usr/local/squid/var
$ sudo /usr/local/squid/sbin/squid -z

Squid will be installed at /usr/local/squid. (The last command is necessary to run a daemon as user nobody.)

Configure Squid

First, prepare a NCSA-compliant encrypted password file for a user (here with username johndoe):

$ cd /usr/local/squid/etc
$ sudo touch squid_passwd
$ sudo chmod o+r squid_passwd
$ sudo htpasswd squid_passwd johndoe
New password:
Re-type new passwod:
Adding passwod for user johndoe

Now, edit /usr/local/squid/etc/squid.conf. The following lines need to be added:

# Add this to the auth_param section
auth_param basic program /usr/local/squid/libexec/ncsa_auth /usr/local/squid/etc/squid_passwd

# Add this to the bottom of the ACL section
acl ncsa_users proxy_auth REQUIRED

# Add this at the top of the http_access section
http_access allow ncsa_users

Finally, run the server:

$ sudo /usr/local/squid/sbin/squid -N -d 1 -D

Firewall will prompt me to see if I allow incoming connections to squid. Say “allow.”

The IP address or hostname of your Mac box at the port 3128 will be available as a proxy server now.

Launch Squid on Startup with launchd

Under the directory /Library/LaunchDaemons, create a file named squid.plist with the following content:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>squid</string>
    <key>OnDemand</key>
    <false/>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/local/squid/sbin/squid</string>
      <string>-N</string>
      <string>-d 1</string>
      <string>-D</string>
    </array>
    <key>ServiceIPC</key>
    <false/>
  </dict>
</plist>

Then issuing

$ sudo launchctl load -w /Library/LaunchDaemons/squid.plist

will launch squid. On reboot, the proxy should also be working automatically.

Customizing & Installing Linux Kernel on Debian Squeeze

Sun, 21 Mar 2010 00:00:00 +0000

Here is a quickie for customizing and install Linux kernel 2.6.x on Squeeze.

Install some packages and kernel source:

$ sudo aptitude update
$ sudo aptitude install kernel-package bzip2 g++ libqt3-mt-dev libncurses5-dev sudo fakeroot

If you do not belong to group “src”, add yourself to the group as we work under /usr/src:

$ sudo adduser username src

You need to logout and login for this change to take effect. You also need to be able to use sudo or su to install the new kernel in the end.

The kernel packaged for Squeeze is at version 2.6.32. Alternatively a newer kernel could be obtained from kernel.org and saved to /usr/src. The customization procedure should be similar. Here I just follow the Debian way:

$ sudo aptitude install linux-source-2.6.32

Extract the source tree:

$ sudo chgrp -R src /usr/src
$ sudo chmod -R g+w /usr/src
$ cd /usr/src
$ rm linux
$ tar -jxf linux-source-2.6.32.tar.bz2
$ ln -s linux-source-2.6.32 linux
$ cd linux

Edit the EXTRAVERSION entry in Makefile, as in:

EXTRAVERSION = .20100321.1

for example to add .20100321.1 to the kernel version number. This is convenient for keeping the existing, working kernels around when you need to recompile with different options.

Use xconfig or menuconfig to customize the kernel options. Before the make-kpkg lines, setting concurrency (most likely to the number of cores of my processor) is optinal but having a higher number typically reduces the compilation time.

$ make mrproper
$ make xconfig    # or make menuconfig
$ export CONCURRENCY_LEVEL=4    # this is optional
$ fakeroot make-kpkg clean
$ fakeroot make-kpkg --initrd kernel_image
$ cd ..
$ sudo dpkg -i linux-image-2.6.32.20100321.1_2.6.32.20100321.1-10.00.Custom_amd64.deb

Upon reboot in the GRUB menu you will find the newly installed kernel:

$ sudo reboot

Purging Old Kernel Image from System

For example, if the kernel to be uninstalled is of version 2.6.26 and the extra version that I used was 20091112.1, do:

$ sudo dpkg -P linux-image-2.6.26.20091112.1

That’s it. However it is often a good idea to keep at least one kernel image that I know for sure to work so that when a custom kernel fails, I have something to fall back on. On the other hand, it is also a good idea to purge very old kernel images to save space in /boot.

Giving Kernel Boot Options with Grub (Version 2)

The new Grub has a slightly different way of specifying boot options (/boot/grub/menu.lst no longer exists for version 2!).

$ sudo chmod +w /boot/grub/grub.cfg
... open and edit /boot/grub/grub.cfg ...
$ sudo chmod -w /boot/grub/grub.cfg

If I need to provide a kernel with boot options, find the menuentry for the kernel, and change from

linux /vmlinuz-2.6.36.1.3 root=UUID=somehash ro quiet

linux /vmlinuz-2.6.36.1.3 root=UUID=somehash ro quiet usbcore.autosuspend=1

in order to set the usbcore.autosuspend option, for example.

Playing with CONCURRENCY_LEVEL

With Core i5 (2 cores but virtually 4 cores with hyperthreading), I get:

$ fakeroot make-kpkg clean
$ export CONCURRENCY_LEVEL=4
$ time fakeroot make-kpkg --initrd kernel_image
...
real    6m50.028s
user    21m45.186s
sys     2m42.246s

$ fakeroot make-kpkg clean
$ export CONCURRENCY_LEVEL=1
$ time fakeroot make-kpkg --initrd kernel_image
...
real    13m55.216s
user    13m23.638s
sys     1m50.359s

I say significant speed up!

What if Kernel Panics after Reboot?

(NOTE: This was true back when I was playing around when Squeeze was still far from stable. I don’t think this section is relevant as of November 2010, since Squeeze is very close to stable now. Just leaving the note in case for its archival value.)

Well, it turns out make-kpkg --initrd does not automatically create a /boot/inird.img-* file for my current configuration. I wondered for a while why my new custom kernel keeps panicking upon reboot but finally realized that was the cause.

Fortunately, I can create the image myself easily:

$ sudo update-initramfs -c -k 2.6.32.20100321.1
$ sudo update-grub

This should work.

Installing FFTW on Mac OS X Snow Leopard

Thu, 18 Mar 2010 00:00:00 +0000

No FORTRAN compiler appears to be included in Xcode, so I need to install it if I haven’t done so. I installed GFortran. In the configure statement below, set F77 to gfortran.

Installing from Fink

For FFTW version 3.x, do:

$ fink install fftw3 fftw3-shlibs

The library will be available under /sw/lib.

Installing from Source

The version of FFTW we use here is 3.2.2. Follow this:

$ mkdir tmp
$ cd tmp
$ wget http://www.fftw.org/fftw-3.2.2.tar.gz
$ tar -xvzf fftw-3.2.2.tar.gz
$ cd fftw-3.2.2
$ ./configure --prefix=/usr/local/fftw-3.2.2 --enable-threads \
    CC="gcc -arch i386 -arch x86_64" CXX="g++ -arch i386 -arch x86_64" \
    CPP="gcc -E" CXXCPP="g++ -E" F77=gfortran
$ make
$ sudo make install
$ sudo ln -s /usr/local/fftw-3.2.2 /usr/local/fftw

This will install FFTW under /usr/local/fftw-3.2.2. If a single precision version also needs to be installed, add --enable-single to the configure statement above and do the configure/make/make install sequence again (it’s a good idea to run make clean before the second time around though).

Update History

November 24, 2010 – Added the Fink version. Changed the installation directory to /usr/local/fftw to avoid potential conflict later.

Installing HOTPANTS 5.1.10 on Mac OS X Leopard

Wed, 10 Mar 2010 00:00:00 +0000

You must love the name, but what the software does is far from sultry; it’s a piece of geekware.

Anyways, I needed to edit files when I install HOTPANTS, photometric alignment utility by Andrew Becker. Here is my installation note before I forget.

HOTPANTS requires CFITSIO to be already installed. On Leopard I use Scisoft OS X, so I don’t have to install CFITSIO separately. Otherwise you have to install CFITSIO first.

Download the tarball and expand it into a temporary directory.

First, you need to delete (or comment out) the line which reads

#include<malloc.h>

from all *.c source code files.

Then modify Makefile. To link CFITSIO properly, modify CFITSIOINCDIR and LIBDIR:

CFITSIOINCDIR=/Applications/scisoft/i386/Packages/cfitsio/include
LIBDIR=/Applications/scisoft/i386/Packages/cfitsio/lib

This is fine if you have Scisoft OS X. If you installed CFITSIO elsewhere, you have to find the paths yourself.

You should also remove the -pedantic-errors option from COPTS so that the line reads like:

COPTS= -funroll-loops -O3 -ansi -Wall -I$(CFITSIOINCDIR)

Then run make within the same directory.

That’s it. You might wish to copy the binaries hotpants, extractkern, and maskim to one of those bin directories in your PATH.

Building 64-bit HOTPANTS

The 64-bit version of HOTPANTS is twice as sultry, I need to try it. The change is that you need to first build 64-bit version of CFITSIO.

The procedure is basically similar to the one above, with a couple of changes:

CFITSIOINCDIR=/usr/local/src/cfitsio/cfitsio/include
LIBDIR=/usr/local/src/cfitsio/cfitsio/lib

Basically you need to point them to the locations where you installed the 64-bit CFITSIO; see the CFITSIO installation note using the link above. Another change is in the compiler options:

COPTS= -m64 -funroll-loops -O3 -ansi -Wall -I$(CFITSIOINCDIR)

These should be the only difference.

Setting Up Squid Proxy Server on Mac OS X Leopard

Thu, 30 Jul 2009 00:00:00 +0000

Getting and Installing Squid

Download a tarball for a stable version from the repository. The version that I use here is 2.7. I assume the file is downloaded to /usr/local/src/squid.

$ cd /usr/local/src/squid
$ gunzip -c squid-2.7.STABLE6.tar.gz | tar xvf -
$ cd squid-2.7.STABLE6
$ ./configure
$ make
$ sudo make install
$ cd helpers/basic_auth/NCSA
$ make
$ sudo make install
$ sudo chown -R nobody /usr/local/squid/var
$ sudo /usr/local/squid/sbin/squid -z

Squid will be installed at /usr/local/squid. (The last command is necessary to run a daemon as user nobody.)

Configure Squid

First, prepare a NCSA-compliant encrypted password file for a user (here with username johndoe):

$ cd /usr/local/squid/etc
$ sudo touch squid_passwd
$ sudo chmod o+r squid_passwd
$ sudo htpasswd squid_passwd johndoe
New password:
Re-type new passwod:
Adding passwod for user johndoe

Now, edit /usr/local/squid/etc/squid.conf. The following lines need to be added:

# Add this to the auth_param section
auth_param basic program /usr/local/squid/libexec/ncsa_auth /usr/local/squid/etc/squid_passwd

# Add this to the bottom of the ACL section
acl ncsa_users proxy_auth REQUIRED

# Add this at the top of the http_access section
http_access allow ncsa_users

Finally, run the server:

$ sudo /usr/local/squid/sbin/squid -N -d 1 -D

Firewall will prompt me to see if I allow incoming connections to squid. Say “allow.”

The IP address or host name of your Mac box at the port 3128 will be available as a proxy server now.

Launch Squid on Startup with `launchd`

Under the directory /Library/LaunchDaemons, create a file named squid.plist with the following content:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>squid</string>
    <key>OnDemand</key>
    <false/>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/local/squid/sbin/squid</string>
      <string>-N</string>
      <string>-d 1</string>
      <string>-D</string>
    </array>
    <key>ServiceIPC</key>
    <false/>
  </dict>
</plist>

Then issuing

$ sudo launchctl load -w /Library/LaunchDaemons/squid.plist

will launch squid. On reboot, the proxy should also be working automatically.

Launch Squid on Startup with SystemStarter

This method should be ignored in favor of the method with launchd described above. This one is incomplete anyways…

This is a server so it would be convenient if the proxy starts up upon reboot automatically. Here is a Mac way to do it:

$ sudo mkdir /Library/StartupItems/squid
$ sudo touch /Library/StartupItems/squid/squid
$ sudo touch /Library/StartupItems/squid/StartupParameters.plist
$ sudo chmod +x /Library/StartupItems/squid/squid

The newly created files should have contents as follows.

squid:

#!/bin/sh

. /etc/rc.common

StartService()
{
 ConsoleMessage "Starting squid"
 /usr/local/squid/bin/RunCache &
}

StopService()
{
 ConsoleMessage "Stopping squid"
 # TODO: add a command to stop squid
}

RestartService()
{
 ConsoleMessage "Restarting squid"
 # TODO: add a command to restart squid
 StopService
 StartService
}

RunService "$1"

StartupParameters.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist
 SYSTEM "file://localhost/System/Library/DTDs/PropertyList.dtd">
<plist version="0.9">
 <dict>
 <key>Description</key>
 <string>squid</string>
 <key>Provides</key>
 <array>
 <string>squid</string>
 </array>
 <key>Requires</key>
 <array>
 <string>Network</string>
 </array>
 <key>OrderPreference</key>
 <string>Last</string>
 </dict>
</plist>

References

Quick HOWTO: Ch32 : Controlling Web Access with Squid – Linux Home Networking

Edits

March 22, 2010. A minor error on the permission of var directory corrected.

Formatting a USB Drive with ext4 for Use with Linux

Fri, 27 Mar 2009 00:00:00 +0000

I wish to use an external USB drive exclusively on a Linux system (in my case, Debian). I want to use the ext4 filesystem. Note that this method wipes out the filesystem on the drive.

Install hotplug-type package such that upon plugging the USB drive in the device is attached to somewhere in the system. I use KDE and it automatically prepares such an environment. When a USB drive is plugged in, I should see a message like this one in the system log:

$ sudo tail /var/log/messages
Dec 16 02:07:51 kernel: [ 3000.544361] scsi11 : usb-storage 3-2:1.0
...
Dec 16 02:07:56 mochi kernel: [ 3005.882879]  sdb: sdb1
...

or I may see something like this instead (I’m using USB 3.0 adapter now):

$ sudo tail -n 100 /var/log/messages | grep sd
...
May  3 12:38:33 mochi kernel: [   12.773057] sd 6:0:0:0: [sdb] Spinning up disk...
May  3 12:38:39 mochi kernel: [   18.782284] sd 6:0:0:0: [sdb] 3906963456 512-byte logical blocks: (2.00 TB/1.81 TiB)
May  3 12:38:39 mochi kernel: [   18.783052] sd 6:0:0:0: [sdb] Write Protect is off
May  3 12:38:39 mochi kernel: [   18.794337]  sdb: sdb1
May  3 12:38:39 mochi kernel: [   18.795569] sd 6:0:0:0: [sdb] Attached SCSI disk

Here, I find the device is at /dev/sdb and there is only one partition, /dev/sdb1, for example. I can also see all the disk devices as follows:

$ sudo fdisk -l

Disk /dev/sda: 80.0 GB, 80026361856 bytes
240 heads, 63 sectors/track, 10337 cylinders
Units = cylinders of 15120 * 512 = 7741440 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xbd3cc0bb

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1       10338    78149633    5  Extended
/dev/sda5               1          46      340992   83  Linux
/dev/sda6              46          52       48128   83  Linux
/dev/sda7              52         698     4881408   83  Linux
/dev/sda8             698        1086     2928640   83  Linux
/dev/sda9            1086        1861     5858304   83  Linux
/dev/sda10           1861        2830     7323648   83  Linux
/dev/sda11           9305       10338     7812096   82  Linux swap / Solaris
/dev/sda12           2830        9304    48949248   83  Linux

Partition table entries are not in disk order

Disk /dev/sdb: 2000.4 GB, 2000365289472 bytes
255 heads, 63 sectors/track, 243197 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0005f107

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1      243198  1953480704    7  HPFS/NTFS

When the USB drive is new, the partition is most likely formatted in FAT or NTFS unless the drive is pre-formatted specifically for OS X. Now, prepare the partition for Linux:

$ sudo fdisk /dev/sdb

See help for detail by pressing m on the fdisk command shell. For example, press p to see the list of partitions in the disk. To create a new one press n (probably after deleting one or more partitions by pressing d). Finally press w to actually make changes to the drive. To use ext4 filesystem, do:

$ sudo mkfs.ext4 /dev/sdb1

after exiting fdisk.

If I always mount this drive on boot, add a line like this to /etc/fstab:

/dev/sdc1 /media/usbdrive ext4 defaults 0 0

assuming /media/usbdrive is the mount point. If I use KDE, adding an entry to /etc/fstab is no longer necessary; I can easily manage external drives with Device Notifier.

Update History

Feb 20, 2014 — Updated following a comment by Ugly Mug.

May 3, 2012 — Updated for my current system.

December 17, 2010 — Modified to use ext4 filesystem.

	Not converted (\(f\))	Converted (\(t\))
\(a\)	\(4,514\)	\(486\)
\(b\)	\(4,473\)	\(527\)

	Not converted (\(f\))	Converted (\(t\))
\(a\)	\(P(V=a, C=f) = 0.4514\)	\(P(V=a, C=t) = 0.0486\)
\(b\)	\(P(V=b, C=f) = 0.4473\)	\(P(V=b, C=t) = 0.0527\)

	Not converted (\(f\))	Converted (\(t\))
\(a\)	\(P(V=a) P(C=f)\)	\(P(V=a) P(C=t)\)
\(b\)	\(P(V=b) P(C=f)\)	\(P(V=b) P(C=t)\)

	\(D_1\)	\(D_2\)	\(\dots\)	\(D_l\)
\(h(s_1)\)	\(1\)	\(0\)	\(\dots\)	\(1\)
\(h(s_2)\)	\(1\)	\(1\)	\(\dots\)	\(0\)
\(\vdots\)	\(\dots\)	\(\dots\)	\(\dots\)	\(\dots\)
\(h(s_m)\)	\(1\)	\(1\)	\(\dots\)	\(0\)

	\(D_1\)	\(D_2\)
\(h(s_1)\)	\(0\)	\(0\)
\(h(s_2)\)	\(1\)	\(0\)
\(h(s_3)\)	\(1\)	\(1\)
\(h(s_4)\)	\(0\)	\(1\)
\(h(s_5)\)	\(1\)	\(0\)
\(h(s_6)\)	\(0\)	\(0\)
\(h(s_7)\)	\(1\)	\(1\)

	\(D_1\)	\(\dots\)	\(D_j\)	\(D_{j+1}\)	\(\dots\)	\(D_l\)
\(h_1\)	\(98,273\)	\(\dots\)	\(23\)	\(23\)	\(\dots\)	\(63,483\)
\(h_2\)	\(2,763\)	\(\dots\)	\(524\)	\(524\)	\(\dots\)	\(2,345\)
\(\vdots\)	\(\vdots\)	\(\dots\)	\(\vdots\)	\(\vdots\)	\(\dots\)	\(\vdots\)
\(h_{n-1}\)	\(325\)	\(\dots\)	\(567,849\)	\(567,849\)	\(\dots\)	\(987\)
\(h_n\)	\(876\)	\(\dots\)	\(7,849\)	\(32\)	\(\dots\)	\(897,347\)

	\(D_1\)	\(\dots\)	\(D_j\)	\(D_{j+1}\)	\(\dots\)	\(D_l\)
\(H_1\)	\(390\)	\(\dots\)	\(57,232\)	\(57,232\)	\(\dots\)	\(33,719\)
\(H_2\)	\(62,509\)	\(\dots\)	\(453\)	\(453\)	\(\dots\)	\(51,954\)
\(H_3\)	\(453\)	\(\dots\)	\(13,009\)	\(23,905\)	\(\dots\)	\(12,174\)

Biboroku

Polymorphism with Single Dispatch in Python

Creating a Debian Boot Partition on a USB Flash Drive

What You Need

Creating a Boot Partition on a USB Flash Drive

Creating an Encrypted Partition on an SSD Drive

Creating the Encrypted Volume

Configuring the Logical Volume Manager

Creating a volume group

Creating logical volumes

Creating the Root and SWAP Partitions in Logical Volumes

Storing SSD Drive Encryption Key File in the Boot USB Flash Drive

Removing the Boot USB Flash Drive

Waking up from Hibernation

Creating Spare USB Flash Drives

Handling Kernel Updates

Snipping Transcripts with Snipd

Attribute Access with Dict

About Biboroku

Conventions Used in This Blog

On Lazy Logging Evaluation

Lazy Function Evaluation with logging

Installing Debian/Jessie on ThinkPad T460s

Intel Wireless 8260

TrackPoint scrolling and disabling TrackPad

Creating a Debian Bootable USB Stick with Non-Free Firmware

TODO

Reference

Install Google Chrome on Debian Jessie

Installing Adobe Reader on Debian/Jessie

Interpreting A/B Test using Python

Background

Python Implementation

Brand Positioning by Correspondence Analysis

PCA and Biplot using Python

Near-Duplicate Detection using MinHash: Background

Jaccard Similarity Index

Shingling

Hashing Shingles

Characteristic Matrix

MinHash as a Jaccard Index Estimator

Locality-Sensitive Hashing

Reference

A Trick for Computing the Sum of Geometric Series

Creating Debian Wheezy Installation USB Stick

Customizing & Installing Linux Kernel on Debian Wheezy

Purging Old Kernel Image from System

Power Management on Lenovo T430s with Debian Wheezy

Enables RC6 Power Saving

Fan Control

Configuring System on Switch between AC & Battery

Preload Daemon

Disabling Nepomuk, Strigi and Akonadi on KDE

Installing Firefox on Debian Wheezy

Using Japanese on Debian Wheezy

IBus

Mozc

Using Japanese in Emacs

Searching for Nearest-Neighbors between Two Coordinate Catalogs

Installing Ubuntu (12.04 LTS 64-bit) from an ISO Image on Hard Drive

Half-Light Radii for Various Profiles

Gaussian Profile

Exponential Profile

Sersic/de Vaucouleurs Profile

Installing Debian Squeeze from Hard Disk

Installing F.lux on Debian Squeeze

Running F.lux on 64-bit Debian Squeeze

Installing Bruzual & Charlot 2003 (GALAXEV) on Debian Squeeze

Testing if a Point is Inside a Polygon in Python

Installing Kindle for PC on Debian Squeeze

Update History

Installing Debian Squeeze (AMD64) on Lenovo T410s

To Begin With: A Problem

General Hardware Specifications

Why Debian Squeeze?

Installing Debian Squeeze

Installation Customization Options

Post-Install Admin Tweaks

Setting up sudo

Setting up APT Sources

Lazy Function Evaluation with `logging`

Setting up `sudo`

`snd_pcsp` Interrupts

Launch Squid on Startup with `launchd`