technovelty

Although it's not particularly clear, a neat feature of xtightvnc is the ability to modify the pop-up menu that appears when you press F8 to send arbitrary keystrokes.

The trick is overriding some of the X11 resources of the file. If you have a look in /etc/X11/app-defaults/Vncviewer (make sure you have a recent package) you can see the default keybindings for the popup menu, which are a good template. Assuming you want to keep them all, you can add your own buttons by starting at button 9 and re-defining the total button count in your ~/.Xresources:

xtightvncviewer*popupButtonCount: 11

xtightvncviewer*popup*button9.label: Alt-F1
xtightvncviewer*popup*button9.translations: #override\n\
                                            <Btn1Down>,<Btn1Up>: \
                                            SendRFBEvent(keydown,Alt_L) \
                                            SendRFBEvent(keydown,F1) \
                                            SendRFBEvent(keyup, F1) \
                                            SendRFBEvent(keyup, Alt_L)

xtightvncviewer*popup*button10.label: Alt-F2
xtightvncviewer*popup*button10.translations: #override\n\
                                            <Btn1Down>,<Btn1Up>: \
                                            SendRFBEvent(keydown,Alt_L) \
                                            SendRFBEvent(keydown,F2) \
                                            SendRFBEvent(keyup, F2) \
                                            SendRFBEvent(keyup, Alt_L)

xtightvncviewer*popup*button11.label: Alt-F12
xtightvncviewer*popup*button11.translations: #override\n\
                                            <Btn1Down>,<Btn1Up>: \
                                            SendRFBEvent(keydown,Alt_L) \
                                            SendRFBEvent(keydown,F12) \

Make sure you merge this with xrdb -merge ~/.Xresources if you don't want to bother logging out and in. This way I can easily send Alt-F1, etc. through to the other side; useful for things like switching virtual terminals in VMware workstation running remotely. I imagine if you were insane you could do all sorts of other tricks too!

posted at: Thu, 22 May 2008 16:29 | in /linux/tips | permalink | add comment (0 others)

Conditional Operator Harmful ? yes : no

It seems existing versions of gcc don't warn when you use an assignment as a truth value as the first operand to the conditional operator. For example:

int fn(int i)
{
	return (i = 0x20) ? 0x40 : 0x80;
}

Presumably you meant ==, but unfortunately that does not warn with gcc (maybe it will one day).

We can use the tree dumping mechanisms (previously mentioned here) to confirm our problem. For the following function gcc creates:

fn (i)
{
  int D.1541;
  int iftmp.0;

  i = 32;
  if (1)
    {
      iftmp.0 = 64;
    }
  else
    {
      iftmp.0 = 128;
    }
  D.1541 = iftmp.0;
  return D.1541;
}

As you can see, this isn't the result we wanted. This can be quite nasty if you use the conditional operator in a hidden fashion; for example behind an assert. A common idiom is:

#define ASSERT(x) (x) ? : fail()

/* programmer now subsitutes a == for */
ASSERT(x = y);

You'll currently get no warning you've slipped up and used the assert wrong. Moral of the story: a quick audit of your code might turn up some surprising uses! However, in this case, the Intel compiler picks this one up:

$ icc -c test.c
test.c(3): warning #187: use of "=" where "==" may have been intended
  	   return (i = 0x20) ? 0x40 : 0x80;

Although it's often not trivial to move to another compiler, as a rule I would recommend trying alternative compilers for your code as it's a great sanity check.

posted at: Sat, 26 Apr 2008 08:29 | in /code/c | permalink | add comment (1 others)

Tips and tricks for your E3 Visa

Having recently been through the process, I've managed to come out the other-side with some some advice for anyone taking this on.

To start out, the link for making appointments at the Australian Consulate is VisaPoint. I found it was remarkably well hidden.
Upon entering the country, it's quite likely the immigration officer won't be that familiar with the E3 Visa. I'd recommend you watch them and very politely correct anything you think wrong, in my case my Visa was incorrectly entered as E3-Dependent along with my wife (proving that the system doesn't actually keep referential integrity between an E3 and an E3D holders).
You can not get paid before receiving a social security number. You can not apply for a social security number within about 10 days of entering the country, as there is a lag between immigration and social security systems. Once you apply it will take a few weeks for the card, but you can probably go back to the office with sufficient ID within two days and receive a print-out of the number. You can call the office you filed with first and they can tell you if it's worth coming in to try it. In the absolute worst case, if things screw-up, you could be in for a long wait. Your company may be able to loan you money if things get desperate.
If anything really stuffs up you'll have to enter the US immigration services, which is no fun. If this happens to you in the Bay Area go straight to the San Francisco office; the San Jose office doesn't have deferred inspections so you'll probably just have to go there anyway.
Your dependent on the E3D Visa will need to fill out an I-765 work authorisation form. This will take up to 3 months to process; you can find out the approximate queue time here. The form doesn't appear to have been updated for E3D; I would strongly recommend getting a lawyer to file it for you so you don't stuff it up. Budget about $400. After this has been processed, the dependent will then need to file for a SSN before they can be paid.
Tax is a black-art. In California expect to pay something like 20-25% federal, 10% state and up to 7% more for Medicare and social security. Expect another $80/month for health insurance.
You can get a bank account without a SSN easily. A good way to transfer money is OzForex, but watch out for fees on the US side (you may have to do wire-transfers which cost about $10 a go, though some banks have bill-pay). The country still runs on cheques for some reason.
You can rent without a SSN, but it will cost you. Budget for 2 months extra rent as deposits and paying ahead.
A car lease is probably a good way to go. Finance will usually look at your visa and only give you a two-year loan, and with no credit that will be at a very high interest rate. In California you can get a 2 year lease on a base model new car (sticker price ~$17,000) drive-away for a down-payment of about $2500 and then $250/month plus taxes. Insurance will be about $120/month on-top of that. The beauty is you just give it back when you go home. For long holiday trips SUV rental may contribute to global warming, but is a very comfortable way to see the country.
In California you'll need to book in for a driving test to get a license; you can't do this in a rental car but some local driving schools offer a short introduction course and then let you use their car for the test.
If you're living in temporary accommodation without great facilities Dream Dinners is a good way to easily get some meals together.
In the Bay Area, Yelp is an awesome resource.
Comcast make Telstra look like a well-run, efficient and customer focused organisation!

Feel free to drop me a line if you're thinking of taking an opportunity and have any questions.

posted at: Thu, 10 Apr 2008 22:41 | in /personal | permalink | add comment (1 others)

Grace Wifi Radio Review

I recently purchased a Grace ITC-IR1000 Wifi Radio from Beach Audio.

There seem to be a number of competing products available at the moment, and it seems they are all based on the Reciva internet radio platform. This appears to be an ARM based system running Linux (checkout the GPL compliance area).

It has built-in wireless, so connects up directly to your 802.11 network. It supports WEP and WPA; entering the WPA key is a little like entering your high score on an arcade game with a built-in scrolling alphabet. It supports all sorts of codecs, including the most important Real, WMA and MP3.

Once on the network, it contacts Reciva service (free) to update and download channel lists. Once there, you can select stations filtered by region (Europe, Oceania, etc) then country. There are literally thousands to choose from, and users can contribute streams which are filtered by Reciva and added to the database periodically. An excellent feature is the ability to log-in to the website and both build favourite lists and add your own personal streams. You then register your radio's serial number with the website and on the next channel update you get access to your lists.

It also offers you the opportunity to connect to a local PC to stream music from folders there; I haven't looked into this feature at all.

Overall, it's a great device. The audio quality is great, and it seems to have at least a little sub-woofer built in. It has some nice features; 5 of the buttons on the front are quick access preset buttons, just hold it down when you are listening to the station and it remembers it. It supports a wide range of streams, the interface is simple to understand, it keeps the clock via NTP and directly connecting to the wireless is great.

My ideal application is for a bedside clock radio, so I can listen to .au radio as I go to sleep and wake up. It has an inbuilt back-lit clock, sleep timer and alarm system, but they all take a bit of fiddling. The buttons are small and black, which doesn't help in the dark, and most access is done via the big scroll wheel which you press to make a selection (think iPod). However, the resistance of the rubber pads is less than that the pressure required to push the button in, so it really takes two hands. This means when you want to setup an alarm and put on the sleep timer as you go to bed there is a bit of fiddling through menus, etc. Also, as it is playing the clock seems to go away. Initially I had a problem with firewall ports, but it doesn't seem a big issue on the forums so it must have just been me.

I also wonder what happens when Reciva go out of business, because all the tuning happens via their website. Something similar happened when 3com Audrey support was dropped; it then became a matter of faking DNS replies. Sniffing the protocol might be fun :)

All in all, although it sounds a little silly, if you know what you want it for this is a great little system. I imagine any expat will get their money's worth listening to home radio stations. If you're only interested in local radio, I'd still suggest the best thing is still probably (surprise, surprise) a radio. It leaves a little to be desired as a bedside radio but is otherwise a very nice toy!

posted at: Fri, 21 Mar 2008 22:43 | in /toys | permalink | add comment (0 others)

Setting up an iPhone via prepay

Just a quick note for Google on setting up a newly purchased iPhone on an AT&T GoPhone pre-pay plan, rather than the contract plan. Although the plans kind of suck, you might like to do this if you don't have a social security number or credit history.

Several forums mention using an all-9's SSN or an all 1's SSN in the signup field. As of 4th March, 2008, these are not working. You need a different SSN which can be provided by AT&T (a hint, it has a lot of 1's and a 4, although I get the feeling they change this as too many people discover it). This will have the effect of failing the credit check, and even though you have to go through and agree to all the contract terms it will then present you with prepay options.

Hopefully that saves some people time setting up their new iPhone!

posted at: Tue, 04 Mar 2008 21:58 | in /toys | permalink | add comment (0 others)

Forwarding VNC via ssh, or finding the PID of a running ssh

I found a few people asking how to automate scripts using ssh with the -f option but not many solutions. Th -f option makes the ssh process fork into the background once started, but since it doesn't give you it's PID it becomes hard to kill it once the script ends.

The solution is to setup a control channel with -M. This allows you to communicate to the ssh process and close it down when your process finishes. For example, here is how I open a VNC connection to my desktop at work, which involves going through a corporate ssh firewall box.

ssh -M -S /tmp/vncssh-firewall.ctl -L 2022:desktop.internal.company.com:22 -f -N username@firewall.company.com
ssh -M -S /tmp/vncssh-localhost.ctl -N -p 2022 -L 5901:localhost:5901 -f -N username@localhost
xtightvncviewer -encodings tight localhost:1
ssh -S /tmp/vncssh-firewall.ctl -O exit localhost
ssh -S /tmp/vncssh-localhost.ctl -O exit firewall.company.com

posted at: Tue, 04 Mar 2008 21:51 | in /linux/tips | permalink | add comment (0 others)

How not to move an iTunes library

I now have several tips on how not to move an iTunes library from a Windows machine to a Mac. The presmise is simple; firstly request iTunes consolidate the library on the source machine and then move the c:\Documents and Settings\username\My Music\iTunes folder to ~/Music/iTunes on the Mac.

So, what I would reccommend not to do:

Simply copy the iTunes directory from an NTFS partition to a FAT formatted media. Somewhere between there and OSX I found special and unicode characters in the filenames got lost. Theoretically maybe this should work, but it didn't.
Attempt to be "unixy" and use a Windows tar utility to a FAT disk. FAT only supports files less than 4GiB.
Re-format the external media as NTFS and then use a Windows tar utility. Untarring on OSX resulted in even more screwed up filenames.
Moving via samba/NFS

What I would recommend is using the 7-Zip utility to create an uncompressed archive of the directory on a NTFS formatted disk (for large file sizes) and then unarchiving it on OSX at the other end. This reliably handled all the crazy file names and the migrated library works perfectly.

posted at: Wed, 06 Feb 2008 10:23 | in /music | permalink | add comment (0 others)

CDBS and .install files

Last night, after dropping a package from a control file, I was wondering why my package.install file for the remaining package seemed to be ignored (upstream installs a bunch of stuff that I don't want installed in the Debian package; the symptom was all that junk was making its way into the package). Turns out it comes down to the following logic in CDBS:

ifeq ($(words $(DEB_ALL_PACKAGES)),1)
        DEB_DESTDIR = $(CURDIR)/debian/$(strip $(DEB_ALL_PACKAGES))/
else
        DEB_DESTDIR = $(CURDIR)/debian/tmp/
endif

i.e. if there is only one package, then by default install in debian/package. Therefore whatever make install does is what you end up with in your package. Although I can see the reasoning behind this, it wasn't what I wanted since I need the package installed into a temporary location (i.e. debian/tmp) which then uses a .install file to pull out only those files I want. The solution is simple, override DEB_DESTDIR to $(CURDIR)/debian/tmp/ in rules.

I hope this saves someone the half hour or so I spent investigating why my install file was "corrupt"!

posted at: Tue, 05 Feb 2008 15:30 | in /linux/debian | permalink | add comment (0 others)

Compare and Swap with PIC

Our Dear Leader Sam Hocevar has previously blogged about PIC and inline ASM. Today I came across a sort of extension to this problem.

Consider the following code, which implements a double word compare and swap using the x86 cmpxchg8b instruction (for a bonus you can lock it to make it atomic).

#include <stdio.h>

typedef struct double_word_t {
	int a;
	int b;
} double_word;

/* atomically compare old and mem, if they are the same then copy new
   back to mem */
int compare_and_swap(double_word *mem,
		     double_word old,
		     double_word new) {

	char result;
	__asm__ __volatile__("lock; cmpxchg8b %0; setz %1;"
			     : "=m"(*mem), "=q"(result)
			     : "m"(*mem), "d" (old.b), "a" (old.a),
			       "c" (new.b), "b" (new.a)
			     : "memory");
	return (int)result;
}

int main(void)
{

	double_word w = {.a = 0, .b = 0};
	double_word old = {.a = 17, .b = 42};
	double_word new = {.a = 12, .b = 13};

	/* old != w, therefore nothing happens */
	compare_and_swap(&w, old, new);
	printf("Should fail -> (%d,%d)\n", w.a, w.b);

	/* old == w, therefore w = new */
	old.a = 0; old.b = 0;
	compare_and_swap(&w, old, new);
	printf("Should work  -> (%d,%d)\n", w.a, w.b);

	return 0;
}

This type of CAS can be used to implement lock-free algorithms (I've previously blogged about that sort of thing).

The problem is that the cmpxchg8b uses the ebx register, i.e. pseudo code looks like:

if(EDX:EAX == Destination) {
    ZF = 1;
    Destination = ECX:EBX;
}
else {
    ZF = 0;
    EDX:EAX = Destination;
}

PIC code reserves ebx for internal use, so if you try to compile that with -fPIC you will get an error about not being able to allocate ebx.

A first attempt to create a PIC friendly version would simply save and restore ebx and not gcc anything about it, something like:

  __asm__ __volatile__("pushl %%ebx;"   /* save ebx used for PIC GOT ptr */
		       "movl %6,%%ebx;" /* move new_val2 to %ebx */
		       "lock; cmpxchg8b %0; setz %1;"
		       "pop %%ebx;"     /* restore %ebx */
	    	       : "=m"(*mem), "=q"(result)
		       : "m"(*mem), "d" (old.b), "a" (old.a),
		         "c" (new.b), "m" (new.a) : "memory");

Unfortunately, this isn't a generic solution. It works fine with the PIC case, because gcc will not allocate ebx for anything else. But in the non-PIC case, there is a chance that ebx will be used for addr. This would cause a probably fairly tricky bug to track down!

The solution is to use the #if __PIC__ directive to either tell gcc you're clobbering ebx in the non-PIC case, or just keep two versions around; one that saves and restores ebx for PIC and one that doesn't.

posted at: Fri, 01 Feb 2008 21:03 | in /code/arch | permalink | add comment (0 others)

A cool laptop

Or, a modern guide to CPU frequency scaling with Linux. Having setup my laptop long ago, I had a strange hybrid of daemons running all trying to scale the frequency of my laptop periodically based on a myriad of different situations. Having decided to fix this, it appears the simplest approach is, as usual, the best.

The way of the future appears to be to let the kernel do all the work with the ondemand govener. The cpufrequtils Debian package will arrange this for you on boot and by default should just work, although you can of course tweak parameters like minimum speed to decrease to and so forth. You can use the cpufreq-set utility to fiddle with settings, or just go into /sys/devices/system/cpu/cpuN/cpufreq/ to tweak them by hand. The many available daemons (cpufreqd, powernowd, etc.) appear to be largely redundant and are probably best removed.

This dropped the temperature of my laptop about 10 degrees C and, apart from a much cooler lap, is so far imperceptible.

posted at: Tue, 29 Jan 2008 16:30 | in /linux/tips | permalink | add comment (0 others)

5 Second VPN

I was recently on a wireless network I trust even less than I usually trust wireless networks, so was looking for a way to ensure a little more security. I've previously setup a PPTP tunnel, but that server is on a boat heading to San Francisco so that was not an option. I have a linode with a generous bandwidth limit, so my first thought was to set that up and route through it with OpenVPN. It started getting a bit complicated and didn't work out-of-the-box with NetworkManager so I gave up.

Then I found that ssh has a really nifty -D option to implement a SOCKS5 proxy. Therefore all I needed to do was ssh -D localhost:8080 remote.box and then setup Firefox to use localhost:8080 as a SOCKS proxy server. But it gets better; I did wonder if my DNS requests were leaking onto the local network, which a quick packet sniff confirmed. It turns out with SOCKS5 all you need to is go to the Firefox about:config page and turn on network.proxy.socks_remote_dns and DNS is tunnelled too.

Since my mail already comes and goes via encrypted channels, this zero maintenance approach pretty much wraps up everything I need from a VPN solution!

posted at: Mon, 21 Jan 2008 11:04 | in /linux | permalink | add comment (0 others)

2GiB in Dell X1

Although the documenation doesn't mention it, I just tried a 2GiB SO-DIMM in my Dell X1 laptop and it works fine, although it does seem to lose the inbuilt 256MiB. The chip I got was a 2GiB PC2-5300 CL5 200 Pin SODIMM which has the bonus of also being sligtly faster than what I had before.

Old : 1196 MB in  2.00 seconds = 598.21 MB/sec
New : 1344 MB in  2.00 seconds = 671.89 MB/sec

To install, you only need undo the screws marked with "K" and then flip the keyboard up (crappy photo). It required a bit of force to seat the DIMM properly, at first I thought it wasn't working but a bit of a push and it was fine. If I could afford a solid state drive, I rekon I could rename my new memory enhanced laptop a "Dell Air"!

posted at: Thu, 17 Jan 2008 13:46 | in /toys | permalink | add comment (0 others)

24 Technobabble

I was watching 24 the other day and (without giving anything away) Edgar Stiles had to "patch a system call" to save the world. After chiding his assistant for giving him the wrong opcode for a "load A, conditional" the following briefly appears on the screen.

It's actually a pretty good piece of technobabble. The register names and use of 0x80 suggest whoever wrote this has at least some familiarity with x86 assembly. It would have been slightly better if they called the "microshell" a "microkernel" and put the system call in eax before calling the interrupt, but if you believe Jack's amazing ability to get cell phone reception then it's only a small issue!

posted at: Thu, 03 Jan 2008 11:22 | in /humor | permalink | add comment (0 others)

Playing with the x86 PMU

A discussion about commas lead to an excuse to have a play with the IA-32 processor performance managment unit (PMU). To start, take two versions of a program to count the number of commas in a text file — one in C and one in Python. The C one runs faster on the input data set of ~60MiB of random data, but why?

The CPU performance monitors give are the key to getting some idea of where the programs spend their time. I like to use perfmon2 because it's what I know, but Oprofile can do it too. All the available events for IA-32 are described in the manual; I currently of no better way of finding out about them than just reading it. On Itanium I reccommend Caliper which, for the most common situations, does most of the work for you and presents it in a nice report. Intel's Vtune also does a similar thing.

The first thing to investigate is if the CPU is getting enough instructions to keep busy. The IFU_MEM_STALL metric is a good place to start as it is triggered when the instruction fetch pipeline is stalled, presumably waiting on either the ITLB or the trace buffer (Icache).

$ pfmon -e CPU_CLK_UNHALTED,IFU_MEM_STALL ./comma < ./randomcommas
340559375 CPU_CLK_UNHALTED
   192115 IFU_MEM_STALL
$ pfmon -e CPU_CLK_UNHALTED,IFU_MEM_STALL python -Sc "import sys; print sum(l.count(',') for l in sys.stdin)" < ./randomcommas
1287100
4571257047 CPU_CLK_UNHALTED
 71981750 IFU_MEM_STALL

That works out to 0.05% of total cycles for the C version and 1.5% for the Python version, neither of which sets off immediate warning bells. If it did, we could start drilling down to things like the L2_IFETCH and ITLB_MISS events, or the BR_* branch events to try and see why the CPU is having to wait to get its next instruction.

Next it is useful to find the CPI (cycles per instruction). This is calculated by the ratio of retired instructions against the number of CPU cycles; since a superscalar machine can issue more than one instruction per cycle this should ideally be much greater than 1 (for example, an Itanium can execute up to 6 instructions each cycle).

$ pfmon -e INST_RETIRED,CPU_CLK_UNHALTED ./comma < ./randomcommas
542953593 INST_RETIRED
340612036 CPU_CLK_UNHALTED
$ pfmon -e INST_RETIRED,CPU_CLK_UNHALTED python -Sc "import sys; print sum(l.count(',') for l in sys.stdin)" < ./randomcommas
1194455205 INST_RETIRED
4569931735 CPU_CLK_UNHALTED

This works out at a CPI of 1.59 for the C version and 0.26 for the Python version. The Python version is clearly spending a lot of time waiting, because it isn't even able to issue one instruction every cycle.

At this point it seems the CPU has enough instructions to do, but it is sitting around waiting to get through those instructions. This suggests the waiting is related to getting data from the cache.

The load and store requests from the L2 cache are accounted to the L2_LD and L2_ST events respectively. These have the ability to mask out cache lines in different states of the MESI protocol, but for this we don't care so just ask pfmon to show us everything.

$ pfmon -e L2_LD:M:E:S:I,L2_ST:M:E:S:I ./comma < randomcommas 
102505 L2_LD:M:E:S:I
   167 L2_ST:M:E:S:I
$ pfmon -e L2_LD:M:E:S:I,L2_ST:M:E:S:I python -Sc "import sys; print sum(l.count(',') for l in sys.stdin)" < ./randomcommas
3278774 L2_LD:M:E:S:I
  10457 L2_ST:M:E:S:I

This shows us that the Python version does quite a few more stores than the C counterpart. Considering this program should simply be reading the input stream and counting the number of commas, we do not expect much store traffic at all. This suggests the Python version is doing some extra copying, for whatever reason (maybe some Python expert can pinpoint it?).

We can drill down a bit more into the memory related latencies. The DCU_LINES_IN event gives the total number of lines allocated in the cache. Another event, DCU_MISS_OUTSTANDING, gives a weighted measure of the cycles spent waiting for a cache line to be brought in. Each cycle spent waiting is weighted by the number of outstanding cache misses (I think the Pentium M I'm using can have up to 4 cache miss requests outstanding at once) and has some caveats, but can be considered a rough estimate of the time spent waiting for a cache line to be brought in. Therefore divding DCU_MISS_OUTSTANDING by DCU_LINES_IN gives us an approximate metric of how long a cache miss takes.

$ pfmon -e DCU_MISS_OUTSTANDING,DCU_LINES_IN ./comma < randomcommas 
769736 DCU_MISS_OUTSTANDING
102387 DCU_LINES_IN

$ pfmon -e DCU_MISS_OUTSTANDING,DCU_LINES_IN python -Sc "import sys; print sum(l.count(',') for l in sys.stdin)" < ./randomcommas
99810150 DCU_MISS_OUTSTANDING
 4240179 DCU_LINES_IN

So that works out to 7.5 cycles for the C version and 23 cycles for the Python version. This seems to strongly suggest that it is memory traffic that is weighing the Python version down.

That is only the initial phase; the results give a high level idea of why one program is running slower than the other. The initial analysis phase generally consists of taking the ratios of certain events to try and get some idea in your head of what the program is doing. Then comes the really hard work; drilling down to figure out how to fix it!

Some useful references:

posted at: Fri, 21 Dec 2007 11:25 | in /code/arch | permalink | add comment (2 others)