in your .Xdefaults file add a line
emacs*font:6x12
You can replace the 6x12 with pretty much anything you see when clicking shift-left mouse. Make sure you merge your changes with xrdb -merge .Xdefaults.
The problem with adding something like (set-face-font 'default "6x10") to .emacs is that if you start up emacs with -nw for no window mode you'll get errors about not being able to setup the font.