Is it possible that you’re somehow saturating VPS throughput? (which might be less available than expected if being QoS throttled or have chatty neighbors)
My overall guess is that these symptoms have something to do with thwarted UDP hole punching (pinging), bandwidth saturation and/or NAT routing issues.
I first saw this kind of disconnect sporadically while running domain/assignment and accessing over local WiFi (using two Ubuntu machines), but never found an effective way to reproduce (or to trace the interwoven networking code paths).
About a month ago both machines began exhibiting the symptoms regularly though – somehow correlated with multi-avatar situations. Like the weekly meetups began 99.9% consistently booting me out – I could initially get in and hear people talking for a brief while, but then in about a minute get disconnected with “server going away” log entry. At the time since nobody else seemed affected I figured this was just my slow broadband connection (15MB/1MB) failing to keep up with the increased number of avatars.