Unfortunately this morning I got hit by a bug where an updated library broke an existing program.
The first thing I noticed was that if I rebuilt the program in question against the new library, everything worked again. This sort of thing points to a (probably unintentional) ABI change.
The source code was large, so I needed to try an zero in on what was happening a bit more. I figured if this was an ABI change, it should show up in the assembly. Thus I created a dump of both binary images with objdump --disassemble.
I then ran that through tkdiff to see where I stood. This showed up about 1500 differences, but looking at them they were mostly
@@ -2138 +2138 @@ -4000000000006dd0: 01 98 81 03 38 24 [MII] addl r51=7264,r1 +4000000000006dd0: 01 98 01 03 28 24 [MII] addl r51=5184,r1
As you may know, on IA64 r1 is defined as the gp or global pointer register. Functions aren't just functions on IA64, they have a function descriptor which contains both the function address and a value for the global pointer. The add instruction can take up to a 22 bit operand, so by adding to the global pointer you can offset into a region of 4MB of memory (2:sup:22 = 4MB) directly. When gcc builds your program, it sets r1 to point to the .got section of your binary. Now between the start of the binary and the GOT there is a whole bunch of stuff, notably unwind info, which might push the offsets out. So we can pretty much ignore all of these when looking for the root of our problem.
So a bit more sed and grep gives you a much reduced list of changes, and one in particular jumps out ...
-4000000000051a2c: 04 00 10 90 mov r38=512 +4000000000051a2c: 24 00 08 90 mov r38=258
This is where the very handy addr2line comes into play. Running that over the binary gives us
ianw@lime:~/tmp/openssh-3.8.1p1/build-deb$ addr2line --exe ./ssh 4000000000051a2c ../../openbsd-compat/bsd-arc4random.c:60
Peeking at that code
static RC4_KEY rc4;
void arc4random_stir(void)
{
unsigned char rand_buf[SEED_SIZE];
60-->memset(&rc4, 0, sizeof(rc4));
if (RAND_bytes(rand_buf, sizeof(rand_buf)) <= 0
... blah blah ...
This looks a lot like the sizeof(RC4_KEY) has changed on us. If our library has a different idea about the size of things than we do, it's sure to be a recipe for disaster. A little test program confirms the hypothesis.
#include "openssl/rc4.h"
main(void)
{
printf("%d\n", sizeof(RC4_KEY));
}
-- 0.9.7e-3 --
ianw@lime:~/tmp$ ./test
258
-- 0.9.7g-1 --
ianw@lime:~/tmp$ ./test
512
Of course, the "what" is the easy bit. Finding out why the size is different is left as an exercise, and a reason why your projects should always keep a ChangeLog in excruciating detail.