Bandwidth Requirements per Connection?


#1

Has anybody crunched the numbers on how much server bandwidth is needed per avatar?

What about hardware specs for 250, or even 500 avatar capacity?


#2

I dont think anyone has even tried anywhere close to 250 or 500. But generally speaking most of the reference for large amount of folks would be the one from back when they did the 110+ user test back in March. we managed to hit 150 before stability issues, but it should give good info on the hardware specs.

https://blog.highfidelity.com/scaling-in-a-single-virtual-reality-space-74e36e2f0ad4
Specifically the domain was split into multiple separate nodes

Lots of the Bandwidth talk from 15-16 is a bit outdated, but I’ve usually estimated with the following information I’ve got from various experiments:

This is however a number I have -mostly- pulled out of my hat and may not reflect current info:

Avatar Position Data + Joint Count * Joint Data Size per Second + Audio Mixer Data size

So around 2.3-5mb per client node (depending on how many joints the avatar has, and if user is talking, and what frame rate they have in use, and how many clients are already in the domain) on domain, even more if assets are loaded from the ATP. Forexample, here is @Judas’s screenshot of his data download Note that this value does not include asset downloads from external urls.

You can also control the amount maximum a client can use per domain using the Show Advanced > Avatar mixer > Per-Node Bandwidth, which by default caps to 5 mbps per client. You could downscale this as well, but not sure on audio quality after.

Again though might be outdated now.


#3

I imagine disabling voice chat would help a lot, especially if it’s a club where music is playing.

Hrmm that’s not super informative being that its AWS and not a actual server with specs you can do math on. I find it a little strange Highfidelity hasn’t bothered to do any math in regards to server specs or bandwidth. You would think they would want to try to optimize these things.


#4

I remember seeing some stats somewhere of the 110+ stress test but I can’t seem to find it. Audio worked nicely but other parts begun to struggle (avatar mixer iirc)

You can get a general idea from the specific info available on AWS:

So looking at the HiFi stuff:

6 x c4.large -> 2 v cores of Intel Xeon E5-2666 v3, and 3.75 GiB ram, 500 Mbps connection -> everything but
c4.xlarge -> 4 v cores of Intel Xeon E5-2666 v3, 7.5 GiB , 750 Mbps -> Entity server
c4.4xlarge -> 16 v core of Intel Xeon E5-2666 v3 And 30 GB and 2 Gbps -> audio mixer

End result would, physically be something of:

2x Intel Xeon E5-2666 v3, and 64 GiB ram and a 5-6 Gbps connection

So generally combined an entire unit for 120+ users. And thats a generous estimate as since some I feel aren’t really needed during the test (asset server can be dropped if assets are on s3, scripted avatars still goes through avatar mixer, and entity script server probably doesn’t need to be as beefy.) Mostly focus should be on entity server, avatar mixer and most of the bandwidth and computing in the audio mixer.


#5

I did tests with 100, 150 and (attempted) 200 simulated avatars. I’ll try to find post where I talked about that later, but, tldr; version. I used a 16 processor digital ocean droplet for avatar mixer, another 16 processor for audio mixer then an 8 processor for entity server and, finally, a 4 processor for DS and remaining AC tasks. AC script runners were 3 of my local machines and another DO 8 core machine (took a lot of CPU to sim all the avatars). Bandwidth is fairly easy to figure - open your stats in interface / then click stats overlay to expand then add up your numbers. It’s gotten a tiny bit better since my last quoted numbers so it’s best to look at current number vs what I’ve previously quoted.

Nothing, and I mean nothing, is simple/easy/reliable (yet) past 150 avatars. The best I did with simulation was 178 when I ran out of CPU on avatar mixer and it began crashing.


#6

Looking at it right now it shows from Interface a rate of 0.07 Mbps incoming and 0.18 Mbps outgoing in a domain with only myself present.

That is probably the least required per agent… rates come up a bit depending on audio and avatar-mixer (number of people running HMD vs desktop and what they’re doing). It would probably be best to run some tests with a network stats monitor on your server side to verify those numbers are the real story. Also, if using ATP, you’d have to factor in bursts of ATP traffic.


#7

Are you positive about this? I clicked your AWS link and seen this:
c4.4xlarge 16 30 EBS-Only 2,000

The 2,000 refers to Mbps.

I understand you did some math, but if AWS for the C4 offered caps at a 2.0Gbps fiber connection; are you sure TERA-bits is the metric you need for the bottom line?

I don’t know whom (if anyone) can realistically push that kind of data from one machine. (Except maybe CERN) :stuck_out_tongue: https://home.cern/


#8

Woops, for some reason had typed T instead of G :smiley: for both cases. (Mah Derp)