Toolchains, libc and interesting interactions

When building a user space binary, the -lc that gcc inserts into the final link seems pretty straight forward — link the C library. As with all things system-library related there is more to investigate.

If you look at /usr/lib/libc.so; the "library" that gets linked when you specify -lc, it is not a library as such, but a link script which specifies the libraries to link, which also includes the dynamic linker itself:

$ cat /usr/lib/libc.so
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf64-x86-64)
GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a  AS_NEEDED ( /lib/ld-linux-x86-64.so.2 ) )

On very old glibc's the AS_NEEDED didn't appear; so every binary really did have a DT_NEEDED entry for the dynamic linker itself. This can be somewhat confusing if you're ever doing forensics on an old binary which seems to have these entries for not apparent reason. However, we can see that that /lib/libc.so itself does actually require symbols from the dynamic linker:

$ readelf --dynamic /lib/libc.so.6 | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]

Although you're unlikely to encounter it, a broken toolchain can cause havoc if it gets the link order wrong here, because the glibc dynamic linker defines minimal versions of malloc and friends -- you've got a chicken-and-egg problem using libc's malloc before you've loaded it! You can simulate this havoc with something like:

$ cat foo.c
#include <stdio.h>
#include <syslog.h>

int main(void)
{
   syslog(LOG_DEBUG, "hello, world!");
   return 0;
}

$ cat libbroken.so
GROUP ( /lib/ld-linux-x86-64.so.2 )

$ gcc -o -Wall -Wl,-rpath=. -L. -lbroken -g -o foo foo.c

$ ./foo
Inconsistency detected by ld.so: dl-minimal.c: 138: realloc: Assertion `ptr == alloc_last_block' failed!

Depending on various versions of things, you might see that assert or possibly just strange, corrupt output in your logs as syslog calls the wrong malloc. You could debug something like this by asking the dynamic linker to show you its bindings as it resolves them:

$ LD_DEBUG_OUTPUT=foo.txt LD_DEBUG=bindings ./foo
Inconsistency detected by ld.so: dl-minimal.c: 138: realloc: Assertion `ptr == alloc_last_block' failed!

$ cat foo.txt.11360 | grep "\`malloc'"
     11360: binding file /lib/libc.so.6 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol `malloc' [GLIBC_2.2.5]
     11360: binding file /lib64/ld-linux-x86-64.so.2 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol `malloc' [GLIBC_2.2.5]
     11360: binding file /lib/libc.so.6 [0] to /lib64/ld-linux-x86-64.so.2 [0]: normal symbol `malloc' [GLIBC_2.2.5]

Above, because the dynamic loader comes first in the link order, libc.so.6's malloc has bound to the minimal implementation it provides, rather the full-featured one it provides internally.

As an aside, AFAICT, there is really only one reason why a normal library will link against the dynamic loader -- for the thread-local storage support function __tls_get_addr. You can try this yourself:

$ cat tls.c
char __thread *foo;

char* moo(void) {
    return foo;
}
$ gcc -fPIC -o libtls.so -shared tls.c
$ readelf -d ./libtls.so  | grep NEED
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
 0x00000001 (NEEDED)                     Shared library: [ld-linux.so.2]
 0x6ffffffe (VERNEED)                    0x314
 0x6fffffff (VERNEEDNUM)                 2

Thread-local storage is worthy of a book of its own, but the gist is that this support function says "hey, give me an address of foo in libtls.so", the magic being that if the current thread has never accessed foo then it may not actually have any storage for foo yet, so the dynamic linker can allocate some memory for it lazily and then return the right thing. Otherwise, every thread that started would need to reserve room for foo "just in case", even if it never cares about moo.

But looking a little closer at the symbols of libc.so is also interesting. libc.so doesn't actually have many functions you can override. You can see what it is possible to override by checking the relocations against the procdure-lookup table (PLT).

Relocation section '.rela.plt' at offset 0x1e770 contains 8 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
00000035b000  084600000007 R_X86_64_JUMP_SLO 00000000000a2100 sysconf + 0
00000035b008  02e000000007 R_X86_64_JUMP_SLO 0000000000075e50 calloc + 0
00000035b010  01dd00000007 R_X86_64_JUMP_SLO 0000000000077910 realloc + 0
00000035b018  029300000007 R_X86_64_JUMP_SLO 00000000000661c0 feof + 0
00000035b020  046f00000007 R_X86_64_JUMP_SLO 00000000000768c0 malloc + 0
00000035b028  000400000007 R_X86_64_JUMP_SLO 0000000000000000 __tls_get_addr + 0
00000035b030  01b400000007 R_X86_64_JUMP_SLO 0000000000076dd0 memalign + 0
00000035b038  086000000007 R_X86_64_JUMP_SLO 00000000000767e0 free + 0

i.e. instead of jumping directly to the malloc defined in the libc code section, any internal calls will jump to this stub which, the first time, asks the dynamic linker to go out and find the address of malloc (it then saves it, so the second time the stub just jumps to the saved location).

This is an interesting list, seemingly without much order. feof, for example, stands out as worth checking out a bit closer — why would that be there when fopen isn't, say? We can track down where it comes from with a bit of detective work; knowing that the value of the symbol feof will be placed into 0x35b018 we can disassemble libc.so to see that this address is used by the feof PLT stub at 0x1e870 (luckily, objdump has done the math to offest from the rip for us; i.e. 0x1e876 + 0x33c7a2 = 0x35b018)

000000000001e870 <feof@plt>:
   1e870:       ff 25 a2 c7 33 00       jmpq   *0x33c7a2(%rip)        # 35b018 <_IO_file_jumps+0xb18>
   1e876:       68 03 00 00 00          pushq  $0x3
   1e87b:       e9 b0 ff ff ff          jmpq   1e830 <h_errno+0x1e7dc>

From there we can search for anyone jumping to that address, and find out the caller:

$ objdump --disassemble-all /lib/libc.so.6 | grep 1e870
000000000001e870 <feof@plt>:
   1e870:   ff 25 a2 c7 33 00       jmpq   *0x33c7a2(%rip)        # 35b018 <_IO_file_jumps+0xb18>
   f19f7:   e8 74 ce f2 ff          callq  1e870 <feof@plt>

$ addr2line -e /lib/libc.so.6 f19f7
/home/aurel32/eglibc/eglibc-2.11.2/sunrpc/bindrsvprt.c:70

Which turns out to be part of a local patch which probably gets things a little wrong, as described below. The sysconf relocation is from a similar add-on patch (being used to find the page size, it seems).

libc, like all sensible libraries, uses the hidden attribute on symbols to restrict what is exported by the library. The benefit of this is that when the linker knows you are referencing a hidden symbol it knows that the value can never be overridden, and thus does not need to emit extra code to do indirection just in case you ever wish to redirect the symbols. In the above, it appears that feof has never been marked as hidden, probably because no internal glibc functions used it until that add-on patch, and since it is not considered an internal function the linker must allow for the possibility that it will be overridden at run time and provide a PLT slot for it. There are consequences; if this was on a fast-path then the extra jumps required to go via the PLT may matter for performance and it may also cause strange behaviour if somebody had preloaded something that took over feof.

Note, this is different from saying that your library can override symbols provided by libc.so; such as when you LD_PRELOAD a library to wrap an open call. What you can not override is the open call that say, the internal libc.so function getusershell does to read /etc/shells.

Having the malloc related calls as preemptable seems intentional and sane; although I can not find a comment to the effect of "we deliberately leave these like this so that users may use alternative malloc implementations", it makes sense so that libc.so is internally using the same malloc as everything else if you choose to use something such as tc-malloc.

tl;dr? Digging into your system libraries is always interesting. Be careful with your link order when creating toolchains, and be careful about symbol visibility when you're working with libraries.

Converting DICOM images, part 2

In a previous post I discussed converting medical images. I tried again today and hit another issue which could do with some google help.

If you see the following:

$ dcm2pnm --all-frames input.dcm
E: can't change to unencapsulated representation for pixel data
E: can't determine 'PhotometricInterpretation' of decompressed image
E: can't decompress frame 0: Illegal call, perhaps wrong parameters
E: can't convert input pixel data, probably unsupported compression
F: Invalid DICOM image

and dcmdump reports somewhere:

# Dicom-Data-Set
# Used TransferSyntax: JPEG Baseline

Try the ``dcmj2pnm`` program, rather than dcm2pnm. The man page does mention this, but that assumes that you know you have a JPEG encoded image :) It works the same, can extract multiple frames and can be converted to a movie as discussed in the previous post. Easy when you know how!

Pyblosxom Recaptcha Plugin

I think maybe the world has moved on from Pyblosxom and, as my logs can attest, spammers seem to have moved on past Recaptcha (even if it is manual).

However, maybe there are some other holdouts like myself who will find this plugin useful. In reality, this will do nothing for stopping spam — for that I am currently having good luck with Akismet. However, I like that in displaying the captcha it costs the spammers presumably something to crack it so they can even attempt to submit the comment.

Comcast self-setup with Netgear routers

I just got a Zoom 5341 modem to replace a very sketchy old Motorola model and so far it seems to work fine.

However, the Comcast self-install switch process was not seamless. After you plug your new modem in the process starts fine, capturing your session and prompting you for your account. However at the next step it prompts you to download the Windows or Mac only client, directing you to an address http://cdn/....

It is at this point you can get stuck if you're behind a Netgear router (I have a WNDR3700) and probably others. The simple problem is that, for whatever reason, the factory firmware in this router does not pass on the domain search suffix through its inbuilt DHCP client. Without that, cdn doesn't resolve to anything, so you can't download the self-help tool and you're effectively stuck at a "can not find this page" screen. If you google at this point you can find many people who have hit this problem, and various information from erroneous suggestions that you've been hacked (?) to most people just giving up and moving into Comcast phone support hell.

Assuming you now don't have internet access, you can complete the process with a quick swizzle. Somebody might be able to correct me on this, but I don't think you want to run the self-install tool from an actual computer plugged directly into the modem if you have a router in the picture, because the wrong MAC address will get registered. So, your best solution is to turn everything off, plug your computer directly into the modem, turn it on, get an address, download the tool, turn everything off, plug the router back-in to the modem and then run the self install tool. At this point, everything just worked fine for me.

Netgear should probably fix this by correctly passing through the domain search suffixes in their routers. Comcast should probably fix this by doing some sort of geo-ip lookup to give clients a fully-qualified address in that webpage to download the tool even if their router is broken (or really, does downloading that file really require a content-delivery network?). You can probably fix this by running Openwrt on your router before you start.

Otherwise, the Zoom modem and the Netgear WNDR3700 seem to make a very nice combination which I would reccommend.

Adding a javascript tracker to your Docbook document

If you convert a docbook document in a chunked HTML form, it would presumably be very useful for you to place one of the various available javascript based trackers on each of your pages to see what is most popular, linked to etc.

I'm assuming you've already got to the point where you have your DSSSL stylesheet going (see something like my prior entry).

It doesn't appear you can get arbitrary output into the <head> tag -- the %html-header-tags% variable is really just designed for META tags and there doesn't appear to be anything else in the standard stylesheets to override.

So the trick is to use $html-body-start$. But you have to be a little bit tricky to actually get your javascript to slip past the SGML parser. After several attempts and a bit of googling I finally ended up with the following for a Google Analytics tracker, using string-append to get the javascript comment filtering in:

(define ($html-body-start$)
  (make element gi: "script"
  attributes: '(("language" "javascript")
                ("type" "text/javascript"))
            (make formatting-instruction
              data: (string-append "<" "!--
 var _gaq = _gaq || [];
  _gaq.push(['_setAccount', 'UA-XXXXXXXXXX-1']);
  _gaq.push(['_trackPageview']);

  (function() {
    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
  })();
// --" ">"))))

easygeotag.info

I think I've just about finished my Thanksgiving project easygeotag.info.

easygeotag.info

I'm a little bit obsessive about geotagging my photos and while I know there are many photo management solutions out there that can do it in various ways, I generally find it quicker and easier to use exiv2 and simple shell scripts to embed the location info directly into my files.

I've tried a number of things that have never worked out better than simply using Google Maps and panning around to find locations — I even bought a GPS tracker which would supposedly automatically tag my photos; assuming of course it could ever get a GPS lock, it hadn't run out of batteries and corrupted its filesystem, you had all times in sync and could figure out the various timezone issues, daylight savings changes, etc etc. I always feel safer having all my metadata embedded in the actual files just incase Yahoo ever does a del.icio.us to Flickr (I use a little Python script with IPTC bindings for comments, which I then backup similarly obsessively locally and to Amazon S3).

The site is fairly simple in concept — it allows you to search for locations, easily extract the geotag info and provides the ability to save frequently used locations for easy reference.

Mostly it was an exercise for me to implement something after reading the excellent Javascript patterns with YUI3, Google App Engine and OpenID — all of which of which I managed to cram in.

Although the audience may be limited (maybe just to me :) I hope someone else finds it useful for managaing their memories! If you think this might be useful and would like the output in some other format, just let me know.

Symbol Versions and dependencies

The documentation on ld's symbol versioning syntax is a little bit vague on "dependencies", which it talks about but doesn't give many details on.

Let's construct a small example:

``$ cat foo.c``

#include <stdio.h>

#ifndef VERSION_2
void foo(int f) {
     printf("version 1 called\n");
}

#else
void foo_v1(int f) {
     printf("version 1 called\n");
}
__asm__(".symver foo_v1,foo@VERSION_1");

void foo_v2(int f) {
     printf("version 2 called\n");
}
/* i.e. foo_v2 is really foo@VERSION_2
 * @@ means this is the default version
 */
__asm__(".symver foo_v2,foo@@VERSION_2");

#endif

``$ cat 1.ver``

VERSION_1 {
      global:
      foo;
      local:
        *;
};

``$ cat 2.ver``

VERSION_1 {
      local:
        *;
};

VERSION_2 {
      foo;
} VERSION_1;

``$ cat main.c``

#include <stdio.h>

void foo(int);

int main(void) {
    foo(100);
    return 0;
}

``$ cat Makefile``

all: v1 v2

libfoo.so.1 : foo.c
        gcc -shared -fPIC -o libfoo.so.1 -Wl,--soname='libfoo.so.1' -Wl,--version-script=1.ver foo.c

libfoo.so.2 : foo.c
        gcc -shared -fPIC -DVERSION_2 -o libfoo.so.2 -Wl,--soname='libfoo.so.2' -Wl,--version-script=2.ver foo.c

v1: main.c libfoo.so.1
    ln -sf libfoo.so.1 libfoo.so
    gcc -Wall -o v1 -lfoo -L. -Wl,-rpath=. main.c

v2: main.c libfoo.so.2
    ln -sf libfoo.so.2 libfoo.so
    gcc -Wall -o v2 -lfoo -L. -Wl,-rpath=. main.c

.PHONY: clean
clean:
    rm -f libfoo* v1 v2

``$ ./v1``

version 1 called

``$ ./v2``

version 2 called

In words, we create two libraries; a version 1 and a version 2, where we provide a new version of foo in the version 2 library. The soname is set in the libraries, so v1 and v2 can distinguish the correct library to use.

In the updated 2.ver version, we say that VERSION_2 depends on VERSION_1. So, the question is, what does this mean? Does it have any effect?

We can examine the version descriptors in the library and see that there is indeed a relationship recorded there.

``$ readelf --version-info ./libfoo.so.2``

[...]
Version definition section '.gnu.version_d' contains 3 entries:
  Addr: 0x0000000000000264  Offset: 0x000264  Link: 5 (.dynstr)
  000000: Rev: 1  Flags: BASE   Index: 1  Cnt: 1  Name: libfoo.so.2
  0x001c: Rev: 1  Flags: none  Index: 2  Cnt: 1  Name: VERSION_1
  0x0038: Rev: 1  Flags: none  Index: 3  Cnt: 2  Name: VERSION_2
  0x0054: Parent 1: VERSION_1

Looking at the specification we can see that each version definition has a vd_aux field which is a linked list of, essentially, strings that give "the version or dependency name". This is a little vague for a specification, however it appears to mean that the first entry is the name of the version specification, and any following elements are your dependencies. At least, this is how readelf interprets it when it shows you the "Parent" field in the output above.

This implies something that the ld documentation doesn't mention; in that you may list multiple dependencies for a version node. That does work, and readelf will just report more parents if you try it.

So the question is, what does this dependency actually do? Well, as far as I can tell, nothing really. The dynamic loader doesn't look at the dependency information; and doesn't have any need to — it is looking to resolve something specific, foo@VERSION_2 for example, and doesn't really care that VERSION_1 even exists.

ld does enforce the dependency, in that if you specify a dependent node but leave it out or accidentally erase it, the link will fail. However, it doesn't really convey anything other than its intrinsic documenation value.

Logging POST requests with Apache

After getting a flood of spam, I became suspicious that there was an exploit in my blog software allowing easy robo-posts. Despite a code audit I couldn't see anything, and thus wanted to log the incoming POST requests before any local processing at all.

It took me a while to figure out how to do this, hopefully this helps someone else. Firstly install libapache-mod-security, then the magic incarnation is

SecRuleEngine On
SecAuditEngine on
SecAuditLog /var/log/apache2/website-audit.log
SecRequestBodyAccess on
SecAuditLogParts ABIFHZ

SecDefaultAction "nolog,noauditlog,allow,phase:2"

SecRule REQUEST_METHOD "^POST$" "chain,allow,phase:2"
SecRule REQUEST_URI ".*" "auditlog"

So, to break it down a little, the default action says to do nothing during phase 2 (when the body is available for inspection); the allow means that we're indicating that nothing further will happen in any of the remaining phases, so the module can shortcut through them. The two SecRules work together -- the first says that any POST requests should be tested by the next rule (i.e. the chained rule), which in this case says that any request should be sent to the audit log. After that, the similar allow/phase argument again says that nothing further is going to happen in any of the subsequent phases mod_security can work on. As per the parts between A and Z, we'll log the headers, the request body, the final response and trailer.

So, as it turns out, there is no exploit; it seems most likely there is an actual human behind the spam that gets through, because every time they take a guess it is correct. So I guess I'll take a glass-half-full kind of approach and rather than being annoyed at removing the spam, I'll just convince myself that I made a small donation from some spam overlord to one of their poor minions!