General Advice
We'll focus on getting good results when recording from a large space like 1920x1200 / 1440x900 to a smaller device. Typically this is for web presentation which right now tops out at 1280x720 (another good example is the iPhone at 480x320).
The tips here are intended for a workflow where you know you'll want to edit, add titling, do voiceovers. In other words, put the footage through any other application before finally compressing it into H264.
Throughout the entire editing process you want to keep the quality of the footage as hight as possible. This leads us nicely to rule #1:
Rule #1: Record using Apple Intermediate Codec.
Try not to have too large a difference between the source recording size and the output format. It's technically possible to record a large area (e.g: 30" LCD Panel) and have that scaled down to show on an iPhone, but it'll look this:
I don't know about you - but I can't read what song is playing :-) In the above case, if I wanted to show the whole screen that's fine, but if explaining some particular UI element I should zoom in.
Rule #2: If recording full screen don't scale down too much; or put in appropriate zooms when editing the footage.
Never compress twice: This is related to rule #1, but it's worth repeating. If you've got to put the footage through multiple programs (iMovie / FCP -> iDVD for example), don't recompress the footage if you can avoid. Have the footage output again in Apple Intermediate Codec.
Finally, after you're happy that your editing is complete, you should compress into H264, using multi pass mode. Either Stomp or QT Pro can do this (Stomp is easier, we think, but we're biased!). If you're going to scale down your footage, do it at this last stage and not before.
Rule #3: Scale down when compressing, not before.