(NOTE: These instructions are for Windows XP SP3. From what I can tell, it works pretty much the same in Vista and 7. For all you Mac lovers out there, lucky you, you don't get this problem.)
First, right click your desktop. Click “Properties”, then click the “Settings” tab. Now click the “Advanced” button, then click the “Troubleshoot” tab. Drag the slider there to the notch just the other side of the middle, and apply everything and start up Minecraft. Viola!
To get even better performance (for a slow computer) try using OptiFine with your Minecraft installation. You can get it here: http://www.minecraftforum.net/topic/249637-11-optifine-hd-a4-fps-boost-hd-textures-aa-af/