Articles in the linux category

Running cloud images locally

Fedora and Ubuntu both provide compact cloud images that are useful for spinning up small VM's quickly (much quicker than installing from a huge ISO or even net-install).

Both these pages have single-click buttons to simply launch the VM's at Amazon, which is very cool. However, you may have reason to use them "offline" with a local KVM (outside of EC2 or OpenStack) for testing, or on a plane, etc. I didn't find a lot of information on this (although it is all available once you know what you're looking for).

So here's my quick guide for the complete newbie. Firstly, download your image of choice and put it in /var/lib/libvirt/images. Open up virt-manager and you can create your new VM via "import existing image".

You can try booting it, but the problem you'll hit is the images have no default password or way to log-in because, very sensibly, you're expected to inject that. Booting you'll see cloud-init attempting to contact 169.254.169.254 to collect the meta-data to initalize the VM; eventually it will time out and you're left at a login prompt you can't log in at. cloud-init does many things and is very flexible with lots of different plugins to initialise images in various ways. The particular one we're interested in is the No cloud plugin. This will look for data from a locally attached CD drive, which works very easily for this scenario.

Now that you know that's what you're looking for, the instructions are fairly clear; I'll repeat them just for clarity. Firstly you want to create a minimal meta-data file that has a host-name and an instance-id:

$ cat meta-data
instance-id: iid-local01
local-hostname: myhost

Modifying the instance-id will signal to cloud-init that it should run again because something changed. This is also where you can put static IP address data; see the documentation.

Secondly, create a user-data file that has the commands to enable password login; set a password, and inject your ssh public key so you can easily log-in:

$ cat user-data
#cloud-config
password: mypassword
ssh_pwauth: True
chpasswd: { expire: False }

ssh_authorized_keys:
  - ssh-rsa ... foo@foo.com (insert ~/.ssh/id_rsa.pub here)

Note for new players: that #cloud-config isn't an optional comment, it's a directive. If you leave it out, you'll probably see a little something about Unhandled non-multipart userdata starting... and your changes won't apply.

Insert those two into files into an iso:

$ genisoimage -output init.iso -volid cidata -joliet -rock user-data meta-data

Then copy init.iso into /var/lib/libvirtd/images as well, and connect that as a virtual-cd drive to your virtual-machine created from the image.

When you boot you should see some output as it injects your key. It should get an address via DHCP and then you should be able to log-in with fedora or ubuntu users. Then you can start getting more complicated with things like static addresses, package installations, running commands, etc by reading the documentation.

Skipping bash history for command-lines starting with space

I was recently reading about a case where part of the evidence appears to be a deleted bash history-file. From what I gather, the accused says that the removal was a clean-up job to remove inadvertently stored passwords rather than an attempt to hide nefarious activity.

What had managed to pass me by in 15 or so years of using bash but not reading man bash properly is the HISTCONTROL variable. If it is set to ignorespace then commands entered with a leading-space will not be stored in the history. Like all the best discoveries I found this only by accident on a machine where it was turned on.

It's been quite useful in keeping my history pruned. However, not usually for security reasons around hiding passwords — passwords on the command-line have other problems and I'm sure any security person would tell you not to use them just from a defense-in-depth perspective. That said, from a "keeping ctrl-r reversi-search useful" point-of-view it's often helpful; for example I've mostly trained my fingers to <space>rm -rf because I'm sure I'm not the only one who has deleted the wrong thing via a history-recall-combined-with-typing-to-fast scenario.

So, HISTCONTROL=ignoreboth (which also ignores duplicates) is certainly a useful one to slip into your .bashrc. At least it would be one-less thing to explain to the FBI!

Stuck keyboard with Fedora 17 under VirtualBox

For a while now I've been plagued with this ridiculous problem of the keyboard becoming "stuck" in my Fedora 17 VM running under VirtualBox. Keys would stop working until you held them down for several seconds, making typing close to impossible.

Of course my first thought was to blame VirtualBox, since it has a lot to do with the keyboard. I found out some interesting things, like you can send scan-codes directly to virtual-machines with something like

$ VBoxManage controlvm UUID keyboardputscancode aa

(get the UUID via VBoxManage list vms; figuring out scan-codes is left to the reader!).

I noticed the problem primarily happening while emacs was in the foreground, meaning I was working on code or an email ... my theory was when I typed too fast some race got hit and put the keyboard into a weird state that a reboot fixed.

Well it turns out the problem is almost the exact opposite and nothing to do with VirtualBox, scan-codes or race-conditions. Luckily, today I noticed "slow keys enabled" warning that sprung-up and went away quickly just before the keyboard stopped. Once I saw that the game was up; turns out this is a well-known bug that is easily fixed with xkbset -sl. It happens because I was typing too slowly; holding down the shift-key while I thought about something probably.

Hopefully this saves someone else a few hours!

Debugging puppetmaster with Foreman

This is a little note for anyone trying to get some debugging out of the puppetmaster when deploying with Foreman.

The trick, much as it is, is that Foreman is running puppet via Apache; so if you're trying to start a puppet master daemon outside that it won't be able to bind to port 8140. You thus want to edit the config file Apache is using to launch puppet /etc/puppet/rack/config.ru. It's probably pretty obvious what's happening when you look in there; simply add

ARGV << "--debug"

and you will start to get debugging output. One issue is that this goes to syslog (/var/log/messages) by default and is a lot of output; so much so that it might get throttled. Although you can certainly reconfigure your syslog daemon to split out puppet logs, an easier way is to just skip syslog while you're debugging. Don't be fooled by config options; simply add

ARGV << "--logdest" << "/var/log/puppet/puppet-master.debug"

to the same file to get the logs going to a separate file. Don't forget to restart Apache so the changes stick.

Shared libraries and execute permissions

In the few discussions you can find on the web about shared-libraries and execute-permissions, you can find a range of various opinions but not a lot about what goes when you execute a library.

The first thing to consider is how the execute-permission interacts with the dynamic loader. When mapping a library, the dynamic-loader doesn't care about file-permissions; it cares about mapping specific internal parts of the .so. Specifically, it wants to map the PT_LOAD segments as-per the permissions specified by the program-header. A random example:

 $ readelf --segments /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0

Elf file type is DYN (Shared object file)
Entry point 0x77c00
There are 7 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x00000000001b5ea4 0x00000000001b5ea4  R E    200000
  LOAD           0x00000000001b6b60 0x00000000003b6b60 0x00000000003b6b60
                 0x0000000000028dc4 0x000000000002c998  RW     200000

The permissions to load the code and data segments are given the Flags output. Code has execute-permissions, and data has write-permissions. These flags are mapped into flags for the mmap call, which the loader then uses to map the various segments of the file into memory.

So, do you actually need execute-permissions on the underlying file to mmap that segment as executable? No, because you can read it. If I can read it, then I can copy the segment to another area of memory I have already mapped with PROT_EXEC and execute it there anyway.

Googling suggests that some systems do require execute-permissions on a file if you want to directly mmap pages from it with PROT_EXEC (and if you dig into the kernel source, there's an ancient system call uselib that looks like it comes from a.out days, given it talks about loading libraries at fixed addresses, that also wants this). This doesn't sound like a terrible hardening step; I wouldn't be surprised if some hardening patches require it. Maximum compatability and historical-features such as a.out also probably explains why gcc creates shared libraries with execute permissions by default.

Thus, should you feel like it, you can run a shared-library. Something trivial will suffice:

int function(void) {
      return 100;
}
$ gcc -fPIC -shared -o libfoo.so foo.c

$ ./libfoo.so
Segmentation fault

This is a little more interesting (to me anyway) to dig into. At a first pass, why does this even vaguely work? That's easy -- an ELF file is an ELF file, and the kernel is happy to map those PT_LOAD segments in and jump to the entry point for you:

$ readelf --segments ./libfoo.so

Elf file type is DYN (Shared object file)
Entry point 0x570
There are 6 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x000000000000070c 0x000000000000070c  R E    200000
  LOAD           0x0000000000000710 0x0000000000200710 0x0000000000200710
                 0x0000000000000238 0x0000000000000240  RW     200000

What's interesting here is that the shared-library has an entry point (e_entry) at all. ELF defines it as:

This member gives the virtual address to which the system first transfers control, thus starting the process. If the file has no associated entry point, this member holds zero.

First things first, where did that entry point come from? The ld manual tells us that the linker will set the entry point based upon the following hierarchy:

  • the -e entry command-line option;
  • the ENTRY(symbol) command in a linker script;
  • the value of a target specific symbol
  • the address of the first byte of the .text section, if present;
  • The address 0.

We know we're not specifying an entry point. The ENTRY command is interesting; we can check our default-link script to see if that is specified:

$ ld --verbose | grep ENTRY
ENTRY(_start)

Interesting. Obviously we didn't specify a _start; so do we have one? A bit of digging leads to the crt files, for C Run-Time. These are little object files automatically linked in by gcc that actually do the support-work to get a program to the point that main is ready to run.

So, if we go and check out the crt files, one can find a definition of _start in crt1.o

$ nm /usr/lib/x86_64-linux-gnu/crt1.o
crt1.o:
0000000000000000 R _IO_stdin_used
0000000000000000 D __data_start
                 U __libc_csu_fini
                 U __libc_csu_init
                 U __libc_start_main
0000000000000000 T _start
0000000000000000 W data_start
                 U main

But do we have that for our little shared-library? We can get a feel for what gcc is linking in by examining the output of -dumpspecs. Remembering gcc is mostly just a driver that calls out to other things, a``specs`` file is what gcc uses to determine which arguments pass around to various stages of a compile:

$ gcc -dumpspecs
...
*startfile:
%{!shared: %{pg|p|profile:gcrt1.o%s;pie:Scrt1.o%s;:crt1.o%s}}
 crti.o%s %{static:crtbeginT.o%s;shared|pie:crtbeginS.o%s;:crtbegin.o%s}

The format isn't really important here (of course you can read about it); but the gist is that various flags, such as -static or -pie get passed different run-time initailisation helpers to link-in. But we can see that if we're creating a shared library we won't be getting crt1.o. We can double-confirm this by checking the output of gcc -v (cut down for clarity).

$ gcc -v -fPIC -shared -o libfoo.so foo.c
Using built-in specs.
 ...
/usr/lib/gcc/x86_64-linux-gnu/4.4.5/collect2 -shared -o libfoo.so
/usr/lib/gcc/x86_64-linux-gnu/4.4.5/../../../../lib/crti.o
/usr/lib/gcc/x86_64-linux-gnu/4.4.5/crtbeginS.o
/tmp/ccRpsQU3.o
/usr/lib/gcc/x86_64-linux-gnu/4.4.5/crtendS.o
/usr/lib/gcc/x86_64-linux-gnu/4.4.5/../../../../lib/crtn.o

So this takes us further down ld's entry-point logic to pointing to the first bytes of .text, which is where the entry-point comes from. So that solves the riddle of the entry point.

There's one more weird thing you notice when you run the library, which is the faulting address in kern.log:

libfoo.so[8682]: segfault at 1 ip 0000000000000001 sp 00007fffcd63ec48 error 14 in libfoo.so[7f54c51fa000+1000]

The first thing is decoding error; 14 doesn't seem to have any relation to anything. Of course everyone has the Intel 64 Architecture Manual (or mm/fault.c that also mentions the flags) to decode this into 1110 which means "no page found for a user-mode write access with reserved-bits found to be set" (there's another post in all that someday!).

So why did we segfault at 0x1, which is an odd address to turn up? Let's disassemble what actually happens when this starts.

00000000000004a0 <call_gmon_start>:
 4a0:   48 83 ec 08             sub    $0x8,%rsp
 4a4:   48 8b 05 2d 03 20 00    mov    0x20032d(%rip),%rax        # 2007d8 <_DYNAMIC+0x190>
 4ab:   48 85 c0                test   %rax,%rax
 4ae:   74 02                   je     4b2 <call_gmon_start+0x12>
 4b0:   ff d0                   callq  *%rax
 4b2:   48 83 c4 08             add    $0x8,%rsp
 4b6:   c3                      retq

We're moving something in rax and testing it; if true we call that value, otherwise skip and retq. In this case, objdump is getting a bit confused telling us that 2007d8 is related to _DYNAMIC; in fact we can check the relocations to see it's really the value of __gmon_start__:

$ readelf --relocs ./libfoo.so

Relocation section '.rela.dyn' at offset 0x3f0 contains 4 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000200810  000000000008 R_X86_64_RELATIVE                    0000000000200810
0000002007d8  000200000006 R_X86_64_GLOB_DAT 0000000000000000 __gmon_start__ + 0
0000002007e0  000300000006 R_X86_64_GLOB_DAT 0000000000000000 _Jv_RegisterClasses + 0
0000002007e8  000400000006 R_X86_64_GLOB_DAT 0000000000000000 __cxa_finalize + 0

Relocation section '.rela.plt' at offset 0x450 contains 1 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000200808  000400000007 R_X86_64_JUMP_SLO 0000000000000000 __cxa_finalize + 0

Thus call_gmon_start, rather unsurprisingly, checks the value of __gmon_start__ and calls it if it is set. Presumably this is set as part of profiling and called during library initialisation -- but it is clearly not an initialiser by itself. The retq ends up popping a value off the stack and jumping to it, which in this case just happens to be 0x1 -- which we can confirm with gdb by putting a breakpoint on the first text address and examining the stack-pointer:

(gdb) x/2g $rsp
0x7fffffffe7d8:        0x0000000000000000      0x0000000000000001

So that gives us our ultimate failure.

Of course, if you're clever, you can get around this and initalise yourself correctly and actually make your shared-library do something when executed. The canonical example of this is libc.so itself:

$ /lib/x86_64-linux-gnu/libc-2.13.so
GNU C Library (Debian EGLIBC 2.13-37) stable release version 2.13, by Roland McGrath et al.
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
...

You can trace through how this actually does work in the same way as we traced through why the trivial example doesn't work.

If you wondering my opinion on executable-bits for shared-libraries; I would not give them execute permissions. I can't see it does anything but open the door to confusion. However, understanding exactly why the library segfaults the way it does actually ends up being a fun little tour around various parts of the toolchain!

Tagging handbrake output for iTunes

If you rip your DVD's with a unix-y name like longish-show-name-season-01-01.m4v then the iPad and iPhone video player only shows the prefix of the file-name to it's string limit, so you only ever see longish-show-na and can never tell what episode is what in the overview list.

You can edit the files in iTunes to add meta-data information, but that's annoying and non-unix-y. It is of course easy, but I went down several dead-ends with tools that didn't recognise file-extensions, corrupted files and plain didn't work.

The package I had most success with is mp4v2-utils and the mp4tags utility. The help is fairly self-explanatory, but something like

mp4tags -show "Show Name" -season <season> -episode <episode> -type "tvshow"  file.m4v

will mean that when you import your file into iTunes it displays correctly as a TV show (and not a movie), is grouped correctly into series and has a name you can see.

Unfortunately, it doesn't seem my Nexus 7 tablet picks up these tags. And both players interfaces remain terrible at indicating which episodes have been watched and which have not. But this has allowed me to re-visit my old box-set of Yes Minister which, despite a few odd references to Quango's still seems as relevant as ever!

Upgrading Samsung Series 9 Laptop BIOS without Windows

Having recently acquired a Samsung Series 9 laptop, I'd like to update the BIOS. This is proving rather difficult without Windows.

The first thing you download from the Samsung website isn't actually BIOS update at all; it's a Windows application which then goes and finds a BIOS update for your machine.

Some time with a Windows VM, tcpdump and you'll end up at http://sbuservice.samsungmobile.com/. I'll leave interested parties to play around at that website, but with a little fiddling you can find the BIOS you're looking for. In this case, it is ITEM_20121024_754_WIN_P08AAH.exe which is the latest version, P08AAH.

Unfortunately, this still won't run in any useful way -- in a virtual-machine it errors out saying the battery isn't connected. On previous endeavors such as this, running various incarnations of zip on the .exe file has found interesting components, but not in this case. Not to be deterred, some work with strings will yield the following

$ strings ./ITEM_20121024_754_WIN_P08AAH.exe | grep Copyright
inflate 1.2.7 Copyright 1995-2012 Mark Adler
deflate 1.2.7 Copyright 1995-2012 Jean-loup Gailly and Mark Adler

Now this is interesting, because it tells us that the components aren't using some fancy compression scheme we have no hope of figuring out. Checking RFC1952 we see the following header mentioned

Each member has the following structure:

+---+---+---+---+---+---+---+---+---+---+
|ID1|ID2|CM |FLG|     MTIME     |XFL|OS | (more-->)
+---+---+---+---+---+---+---+---+---+---+

We can easily look for that in the file. Expanding the values, what we're looking for is 0x1f 0x8b 0x08 0x0 0x0 0x0 0x0 0x0 0x0 0x0 which represents the compressed stream (the OS byte is 0xb for NTFS which we can just ignore). A quick hack will get that done and give us the offsets we need

$ ./look-for-headers
Found potential header at 455212
Found potential header at 677136
Found potential header at 1293930
Found potential header at 1294980
Found potential header at 2771193
Found potential header at 2779874
Found potential header at 2789142
Found potential header at 2935778
Found potential header at 3078639
Found potential header at 3083039
Found potential header at 3338552
Found potential header at 3339443
Found potential header at 3352440
Found potential header at 3366757
Found potential header at 3391734
Found potential header at 3444255
Found potential header at 3494372

So, we can feed this into a shell script and extract the streams

FILE=ITEM_20121024_754_WIN_P08AAH.exe

for offset in 455212 677136 1293930 1294980 \
    2771193 2779874 2789142 2935778 3078639 \
    3083039 3338552 3339443 3352440 3366757 \
    3391734 3444255 3494372
do
    echo "create $offset.gz"
    dd if=$FILE  bs=1 skip=$offset | zcat > $offset.data
done

We don't have to worry about the end-point, because zcat will handily stop outputting and ignore junk at the end of the stream. That will yield the following output files:

$ md5sum *
c014e900b02ae39d6e574c63154809b2  1293930.data
982747cd5215c7d51fa1fba255557fef  1294980.data
942c6a8332d5dd06d8f4b2a9cb386ff4  2771193.data
d98d2f80b94f70780b46d1f079a38d93  2779874.data
33ce3174b0e205f58c7cedc61adc6f32  2789142.data
1d6851288f40fa95895b2fadee8a8b82  2935778.data
3a0a9b2fa9b40bdf5721068acde351ce  3078639.data
fc166141d06934e9c7238326f18a4e68  3083039.data
8cf7f86f84afa81363d3cef44b257d50  3338552.data
85e09e18fd495599041e13b4197e1746  3339443.data
0c72004eae0131d52817f23faf8a46a7  3352440.data
26347f93721e7ac49134a17104f56090  3366757.data
26973e301da4948a0016765e40700e8c  3391734.data
c8e4675cda3aa367d523a471bb205fba  3444255.data
7bf5ea753d4cc056b9462a02ac51b160  3494372.data
a2c6562f7f43aaa0e636cc1995b7ee3d  455212.data
d5c91b007dac3a5c6317f8a8833f9304  677136.data
65c0dcd1e7e9f8f87a8e9fb706eb8768  ITEM_20121024_754_WIN_P08AAH.exe

$ file *
1293930.data:                     MS-DOS executable
1294980.data:                     data
2771193.data:                     PE32 executable (native) Intel 80386, for MS Windows
2779874.data:                     PE32+ executable (native) x86-64, for MS Windows
2789142.data:                     PE32 executable (console) Intel 80386 (stripped to external PDB), for MS Windows
2935778.data:                     PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows
3078639.data:                     MS-DOS executable
3083039.data:                     PE32 executable (GUI) Intel 80386, for MS Windows
3338552.data:                     ASCII text, with CRLF line terminators
3339443.data:                     PE32 executable (native) Intel 80386, for MS Windows
3352440.data:                     PE32+ executable (native) x86-64, for MS Windows
3366757.data:                     PE32 executable (DLL) (GUI) Intel 80386, for MS Windows
3391734.data:                     PE32 executable (GUI) Intel 80386, for MS Windows
3444255.data:                     PE32 executable (GUI) Intel 80386, for MS Windows
3494372.data:                     PE32 executable (DLL) (GUI) Intel 80386, for MS Windows
455212.data:                      PE32 executable (DLL) (console) Intel 80386, for MS Windows
677136.data:                      data
ITEM_20121024_754_WIN_P08AAH.exe: PE32 executable (GUI) Intel 80386, for MS Windows

This is where I get a bit stuck. There is clearly a bunch of WinFlash stuff that Wine kind-of recognises, but so far I haven't managed to figure out how I might go about flashing this from a DOS boot disk.

So far I know it's a "Phoenix SecureCore Tiano" based BIOS, and there is some indication that there is a UEFI app to flash the BIOS, but unfortunately I didn't set the laptop up with UEFI (note to anyone setting this up; probably a good idea to do that).

I believe that if you have Windows, you can halt the install part-way through and see these files extracted in a temp directory. Possibly if we can put some file-names to the blobs it might help.

There may also be a much easier way to do this that I'm completely missing. If anything comes up, I'll update this post; hopefully a future Googlenaut who enjoys BIOS hacking will drop a comment...

Update 2012-11-27 I'm unfortunately less optimistic about being able to update this.

Certainly the files identified above as MS-DOS executable are EFI binaries; however they are both 32-bit binaries. You can easily get an EFI shell up and running by putting a 64-bit EFI shell binary on a FAT formatted USB disk in the efi/boot/bootx64.efi but it will not run either of those binaries, presumably because they're 32-bit. A 32-bit EFI binary doesn't seem to be recognised for boot, so no luck there.

A comment also mentioned some issues with Samsung's and UEFI boot in this bug though it's not clear what models are affected.

Another comment mentioned binwalk which also does a nice job of finding the embedded zip streams.

$ strings ./1293930.data
...
RSDS
D:\flash\Source\ENV\src\Efi863\edk\phoenix\edk\Sample\Platform\DUET\ldr16\IA32\oemslp20.pdb
oemslp20.dll
InitializeDriver
2n3u3

$ strings ./3078639.data
en-us;zh-Hans
slp20.marker
slp20.size
slp20.offset
slp20.offset2
read marker fail!
Marker file '%s''s size(%d)!=%d
Not a valided OemMarker!
Update image data error.
marker file is updated.
marker file size too big!
Not a valided OemMarker in ROM.
Update image data error.
SLP marker is reserved.
-PROT
Prot flag is found, mProgFlag is true.
Prot flag is not found, mProgFlag is false.
Please specify input file name.
-PROT
slpFile
OEM Activation 2.0 processing
Failed to Locate Flash Support protocol.
...
D:\flash\Source\ENV\src\Efi863\edk\phoenix\edk\Sample\Platform\DUET\ldr16\IA32\SLP20.pdb

Exploring $ORIGIN

Anyone who works with building something with a few libraries will quickly become familiar with rpath's; that is the ability to store a path inside a binary for finding dependencies at runtime. Slightly less well known is that the dynamic linker provides some options for this path specification; one in particular is a path with the special variable $ORIGIN. ld.so man page has the following to say about the an $ORIGIN appearing in the rpath:

ld.so understands the string $ORIGIN (or equivalently ${ORIGIN}) in an rpath specification to mean the directory containing the application executable. Thus, an application located in somedir/app could be compiled with gcc -Wl,-rpath,'$ORI‐ GIN/../lib' so that it finds an associated shared library in somedir/lib no matter where somedir is located in the directory hierarchy.

As a way of a small example, consider the following:

$ cat Makefile
main: main.c lib/libfoo.so
    gcc -g -Wall -o main -Wl,-rpath='$$ORIGIN/lib' -L ./lib -lfoo main.c

lib/libfoo.so: lib/foo.c
    gcc -g -Wall -fPIC -shared -o lib/libfoo.so lib/foo.c

$ cat main.c
void function(void);

int main(void)
{
   function();
   return 0;
}

$ cat lib/foo.c
#include <stdio.h>

void function(void)
{
   printf("called!\n");
}

$ make
gcc -g -Wall -fPIC -shared -o lib/libfoo.so lib/foo.c
gcc -g -Wall -o main -Wl,-rpath='$ORIGIN/lib' -L ./lib -lfoo main.c

$ ./main
called!

Now, wherever you should choose to locate main, as long as the library is in the right relative location it will be found correctly. If you are wondering how this works, examining the back-trace from the linker function that expands it is helpful:

$ gdb ./main
(gdb) break _dl_get_origin
Function "_dl_get_origin" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y

Breakpoint 1 (_dl_get_origin) pending.
(gdb) r
Starting program: /tmp/test-origin/main

Breakpoint 1, _dl_get_origin () at ../sysdeps/unix/sysv/linux/dl-origin.c:37
37  ../sysdeps/unix/sysv/linux/dl-origin.c: No such file or directory.
    in ../sysdeps/unix/sysv/linux/dl-origin.c
(gdb) bt
#0  _dl_get_origin () at ../sysdeps/unix/sysv/linux/dl-origin.c:37
#1  0x00007ffff7de61f4 in expand_dynamic_string_token (l=0x7ffff7ffe128,
    s=<value optimized out>) at dl-load.c:331
#2  0x00007ffff7de62ea in decompose_rpath (sps=0x7ffff7ffe440,
    rpath=0xffffffc0 <Address 0xffffffc0 out of bounds>, l=0x0,
    what=0x7ffff7df780a "RPATH") at dl-load.c:554
#3  0x00007ffff7de7051 in _dl_init_paths (llp=0x0) at dl-load.c:722
#4  0x00007ffff7de2124 in dl_main (phdr=0x7ffff7ffe6b8,
    phnum=<value optimized out>, user_entry=0x7ffff7ffeb28) at rtld.c:1391
#5  0x00007ffff7df3647 in _dl_sysdep_start (
    start_argptr=<value optimized out>, dl_main=0x7ffff7de14e0 <dl_main>)
    at ../elf/dl-sysdep.c:243
#6  0x00007ffff7de0423 in _dl_start_final (arg=0x7fffffffe7c0) at rtld.c:338
#7  _dl_start (arg=0x7fffffffe7c0) at rtld.c:564
#8  0x00007ffff7ddfaf8 in _start () from /lib64/ld-linux-x86-64.so.2

Just from a first look we can divine that this has found the RPATH header in the dynamic section and has decided to expand the string $ORIGIN.

$ readelf --dynamic ./main

Dynamic section at offset 0x7d8 contains 23 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libfoo.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/lib]
 0x000000000000000c (INIT)               0x4004d8
 0x000000000000000d (FINI)               0x4006f8

At this point it proceeds to do a readlink on /proc/self/exe to get the path to the currently running binary, effectively does a basename on it and replaces the value of $ORIGIN. Interestingly, if this should fail, it will fall back on the environment variable LD_ORIGIN_PATH for unknown reasons. This might be useful if you were in a bad situation where /proc was not mounted and you still had to run your binary, although you could probably achieve the same thing via the use of LD_LIBRARY_PATH just as well.

This feature should be used with some caution to avoid turning your application in a mess that can't be packaged in any standard manner. One common example of use is probably your java virtual machine to find its implementation libraries.

$ readelf --dynamic /usr/lib/jvm/java-6-sun/bin/java

Dynamic section at offset 0x9a28 contains 25 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libjli.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [lib.so]
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/../lib/amd64/jli:$ORIGIN/../jre/lib/amd64/jli]
 ...

There is a small but interesting security issue. If an attacker can hardlink to your binary that is using $ORIGIN then any path expansion is now relative to the attackers directory. Hence it is possible to make the linker read in arbitrary libraries and hence run arbitrary code. Clearly if your program is running setuid as someone else, such as root, you've just given up the keys to the house!

Luckily, the dynamic linker will not let you shoot yourself in the foot like this, and prevents expansion of the $ORIGIN field if the program is running setuid.

$ sudo chown root ./main
$ sudo chmod +s ./main
$ ./main
./main: error while loading shared libraries: libfoo.so: cannot open shared object file: No such file or directory

However, this is a good example of why you might want to keep users home and temp directories on separate file systems away from where binaries live.

There is also a similar variable $PLATFORM, which describes the current running system. This is gained via the auxiliary vector, which I have written about here and here. Setting LD_SHOW_AUXV=1 will allow you to examine this value.

$ LD_SHOW_AUXV=1 ls
...
AT_PLATFORM:     x86_64
...