Capture "smoothness" can be tricky and depends on a number of factors:
1) The output size is the movie. That is, what size is it (width x height) that you're creating?
2) The compression method in use (H264, Apple Intermediate, JPEG etc)
3) The frames per second of the capture (and whether variable length frame durations are enabled)
4) Speed of machine
The first two questions are the most important: What is the output size and compression? Let me provide an example.
If I have a MacBook Pro, and try to capture full screen (1440x900) using H264 at 30fps, the MacBook Pro isn't going to be able to do it. H264 is simply too resource (CPU) intensive. As a result, you'll get frame skip and the movie may look jerky.
The solution is to record using:
Snow Leopard Users: Apple Intermediate (AIC) or Apple Animation (AA)
Lion Users: JPEG.
and then convert the movie to H264 later using either Stomp or QT Pro. Both of these use far less in the way of CPU resources, and allow for faster (and thus smoother) captures.
Note: These 'fater' codecs use minimal compression (When set to "Lossless" with Apple Animation) and thus will create larger files. Please ensure you have enough disk space if you are capturing long segments. You can check in your iShowU HD preferences to verify or change the path of your storage to a higher capacity drive.
About capture size / output size (iShowU HD / HD Pro specific)
Note that the capture size itself doesn't make much difference. It's the final movie size (called Output Size, in iShowU HD) that matters. The larger the size, the more load is put onto the CPU. Capturing full screen to 640 x 480 is pretty easy for almost any modern machine. Capturing the same full screen to 1440x900 is about four times the amount of work.
Recommended suggestions:
1) Try using Apple Intermediate (You must be running Snow Leopard and have iLife installed for this to appear) or JPEG for video.
2) Try reducing the output size dimensions of the movie.
3) Turn on Variable size frame durations, and turn off Low CPU mode. (HD Pro Version Only)
4) Turn on Low CPU mode (compression will be performed after capture) (HD Pro Version Only)
Those last two options - why would they help?
iShowU HD (and HD Pro) are both compatible with Final Cut. What this means in practice is that they write video frames at exactly the right points in time, and guarantee to do so. However in some cases this can yield bad performance (e.g: capturing a heavy scene in Second Life).
You can tell because the movie itself says 30fps but playback appears to be only 10fps. Or worse.
The real solution is to relax the frame duplication constraint and allow iShowU HD full freedom to skip frames when it needs to. This by the way is what iShowU Classic does. With variable frame durations enabled, HD can drop a frame and expand the duration of the next frame to fill the gap. In this way total CPU and disk load is reduced, giving a smoother capture.
Lion Users: The way that variable frame durations is done in Lion is different. If enabled, HD will capture using variable duration mode, and then after it's done, it'll recompress the video (using the same codec settings) to turn it from variable into non-variable.
The other option is to enable Low CPU mode, so that you turn off compression entirely and let HD write out temporary frames to disk. When you're done recording, HD will compress the frames. This is a tradeoff between slowing the computer down while recording or slowing it down at the end. By turning off the compressor while recording we free up computer while recording, thus resulting in faster, smoother captures.