I've also been having similar problems. Unresponsiveness on my end isn't as atrocious (I get delays in single seconds, mostly). For a test case, see my post "ssh slowdowns under X".