Mine are more funny than annoying, though.

Sometimes, while I'm using the mouse, the cursor will (just as you described) take off in some random direction. It's only for a split second though, I've got control again almost instantly. It's never happened anywhere terribly important (dragging and dropping, for instance), so I'm not really annoyed by it.

The other one is when I let the mouse go to type, and the cursor starts to vibrate a little bit. It looks pretty hilarious on screen, like the arrow is cold or something.

Microsoft Optical Wheel mouse here. Everything else works okay, and like you, I use it on the desk surface, rather than a mousepad. I thought these things were supposed to eliminate the need for a mousepad?