Is this on Linux?
If you have Pulse Audio running this can happen. Especially on older versions.
This is the same bug as the stuttering audio output.
First update your distro to make sure you have the latest version so that you have the latest version of Pulse Audio. This will reduce this bug significantly.
Additionally make sure that Pulse-Audio is run at high priority: at least nice level -10 and at least 5 higher priority than your X11 driver. (If your X11 is running -7, run PulseAudio at -12). Remember the further negative the nice level the higher the priority, the further positive the lower the priority. You will need to use root to change the priority of the PulseAudio daemon.