Yes, audio interface buffers and plug-in delay compensation are the two largest contributors. Fortunately, these are things the user has control over. Except for WASAPI shared which is fixed by the OS, the audio interface buffers are set in preferences or, in some cases, software provided by the interface manufacturer. For 44.1/48K sample rates most find 128 is the largest usable buffer size although some can work with 256. Plug-in delay compensation is determined by the plug-ins on th