Why is Sound not a 1st class citizen?


#1

In VR, sound MUST be a 1st class citizen. It’s not just a frill: it’s crucial to the atmosphere, mood and sense of immersion, and in many cases it directs the user’s behavior, e.g. to attract attention to an animated object, or for narration and voice guidance. The HiFi API has this ability, but it is not presented well in the authoring process.

Computer software has historically given short shrift to audio. In the 80s/90s, users would often just turn their sound off so as not to annoy the guy in the next cubicle. Apple was the only company that recognized sound and music as a killer app, and look where they are now. In the modern games world audio has become increasingly important. Top gamers rely heavily on the voices of their buddies, not to mention sound and music to cue them to events in the game–not surprising, as composers are using messages from the game engine to trigger their sounds. And in VR it serves an even more crucial purpose. The top VR games are doing a great job with sound; Facebook and Google and Magic Leap have dozens of employees in their audio departments.

Given the pedigree of the HiFI founders (RealAudio etc) they have worked very hard on real time avatar dialog. It’s clear from the API that they take sound very seriously. There’s a great opportunity for HiFi to be the exemplary social VR platform and to do a better job with audio UI and environmental sound! But it is not currently presented well to everyday content creators…

-When you hit the ‘create’ tab there are buttons for many classes of entity—cube, sphere, light, particle etc—but none for Sound or MIDI.

-in the Entity List there’s a pull-down menu for most entity types, but not for Sound or MIDI

-sound entities should have their own Properties tab interface, with easy access to the following parameters:
-URL for .WAV or .MP3 or .MID
-Volume
-Spatialized? yes/no
-loop? yes/no
-propagation factor (how much the volume diminishes based on user’s distance)
-occlusion factor (how much the volume and tone diminish based on solids in line of sight, e.g. walls of a building. This does not need to be acoustically/mathematically correct!)

-If you’re interested in audio streaming, reverb, domain audio environment etc you have to dig deep into the documentation. These functions need to be more accessible in the UI.

-people often complain that sound or music loops get repetitive and annoying. It would not be hard to mitigate this by allowing for playlists. For example, load 10 MP3s and randomize them each time a user visits. Ok, you could achieve this using a script, but how many programmers can be bothered? I have seen a total of ONE attempt at good adaptive sound in HiFi, and that’s in Beyond. But with good UI, creators could be encouraged to take their domain’s sound more seriously.

Great sound and music are low-hanging fruit. Let’s tell HiFI what we’d like to see improved. I’d be keen to hear other developers’ views?


#2

Im still holding out for a return to vinyl


#3

Also, maybe see if some of these audio properties could be plug to the Web entity.
Web entities play region wide… would be good if they could play base on distance/volume.


#4

I had to re-read this a few times but I think I can provide some pushes towards the right direction.

In a way, sound emission is already a thing in High Fidelity, as noted in the API. My own avatar uses this for playing footsteps when my feet hit the ground rather than on a timer and a VERY subtle heartbeat that plays all the time and can only be heard at VERY VERY close range.

That being said, after really thinking it over, I can somewhat understand the usefulness of a sound entity type, where the properties of it are based on the already existing sound player, which can be found in the marketplace. The item, in question, meets about half the requirements you specified (minus MID support, but more on that later) but that’s mostly due to the later requirements not being in place.

Part of the reason propagation factor isn’t fully supported is due to that being a domain setting, which has its own form of propagation factor effective for all sound mixing. On one hand, this helps universalize sound so that my footsteps sound and play the same as my own voice would, which are coming from points very near each other. On the other hand, it would be nice to be able to specify sound like a light, where you could have situations where audio is near consistant in a given radius, begins to fade past it, until fully diminished when out of the full radius. To an extent, this could be done with the current API, but if I remember correctly, tinkering with sound position and rotation, things that would need to be done to mimic such an effect, results in the sound emitter to restart due to a bug. I think it’s been addressed for a future update or the team is aware of it, but I haven’t tinkered with it too much since.

Not to mention, I’ve actually had this as a feature request for awhile now (psst, devs, look up “Feature Request: ability to find perceived audio volume”).

Occlusion factor would be interesting and I think clever use of zones could even make it work out, but I’m pretty sure there’s easier ways of having it automated so a user doesn’t need to play “I want to be a map maker.”

Anyway, going back to the entity idea: I think the bigger issue is that the current way of specifying data in an object is pretty good for programmers, but confusing as heck for non-coders, which is why things like the currently in place sound player isn’t doing too much. Part of it would be due to having to find a way to make the Create app GUI flexible enough to allow for custom parameter specifications, but I think having something like that in the long run would end up allowing what you want to happen to become more possible without reinventing the wheel while adding additional flexibility for programmers and creators in other tools.

This is where I want to being up MIDI again. While I’m not quite sure how useful it would be to pass a MID file as an audio file source, I could totally see cases where you could specify the MIDI device you are trying to mirror on a virtual incarnation, where the two would match the actions of each other, which has been demonstrated a lot lately (passing MIDI from a VR user’s virtual keyboard to the synth of another user in another location, sending MIDI from an instrument to the virtual one to cause an action, etc). Having the above optimizations could mean making this far easier since the user would have to only worry about a simple GUI and not a complicated “Time to bootcamp myself through JSON.” The key thing is to find a way to make this universal so that no matter the create app or system, the expectation of where to look is the same.

Regarding audio streaming, reverb, and what not, I completely agree. In fact, most audio stuff now mentioned in the server’s settings aren’t even correct. This was brought up ages ago but I guess it’s worth bumping again, since the reverb effect is pretty well done when it works.

Regarding the playlist script… actually, this one does exist, but where it’s used is very weird. In The Spot, the radio playing near the grill is actually a mono mp3 file that is over a half hour long (it used to be a 175MB uncompressed WAV file). The irony is that the burgers nearby? Those are actually using a playlist script for 3 different sound files that randomly shuffle which one is played… at least last I inspected it. I just tried to find the scripts in the GitHub, but alas, they aren’t there.

I guess that brings up another thing: a good chunk of stuff that typically gets requested in High Fidelity does already exist, but isn’t exactly broadcast too well. Things like the sound player, playlist player, spawn spacer, web shortcuts, marketplace item shortcuts, nametags, and even something as simple as chat are all available for free, but are held back by needing access to community knowledge. Sure, you could have a backup file of an item that has what you need, but that gets old pretty fast and isn’t easy to send from person to person. So while I’d love to hand you the playlist script, there isn’t an easy way of doing it (that and I’m not sure if High Fidelity wants it public or not, despite having other items publicly available in their GitHub).

So let’s TL:DR this entire thing:

  • Most things are in the API and do require a script to be made
  • While a sound entity would be an interesting addition, improving the ability for creations to be customized without needing programming knowledge is a must
    • Yes, I’m even going to call JSON programming in this case
  • MIDI support does exist, but the above listed improvements would help make it even better
    • Except MID file support. Honestly not sure how that could be used unless you wanted local/server effects that used MID file actions (light controls?) to be translatable to other objects as a form of an always running automation system
  • Propagation Factor would be awesome and would be even nicer if we could perceive the current audio levels post PF in scripting
  • Audio stuff in the server settings needs some love
  • Most things that have been requested feature wise do exist, but the ability to find them isn’t the best.

Overall, I agree that creator tool usability improvements are a must, be it for people trying to tinker with sound or any other metaverse aspect. Most of the issues I see here could be quelmed if the ability to interface with pre-existing scripts had easier access without needing to know what JSON is, while the ability to recall scripts would also help out in knowing if something already exists or not. This would effectively mean that custom entity types, in theory, could be made on the fly. Need a sound emitter? Just have a good sound emitter script on standby. Need a playlist emitter? Make like Glade and plug it in.


#5

I do nothing in this platform, because I am of meagre skills, but I still have memories of the earliest days of this platform, and, unless i am wrong, sound was one of the driving goals of HiFi. @philip used to speak about sound quality, and concerts across the world without latency.

Maybe I am addressing something else, but it does seem as if the emphasis on sound has dimmed.


#6

Great thoughts. I think a core idea here is to add a sound entity type - I will look into that.


#7

I totally agree, I’ve been digging through the documentation to do Audio related stuff in VR that mimics RL situations, and I can see that most of what I’m aiming for is doable, but me re-tooling my programming skills is holding me back.


#8

As long as the ability to still use scripted sound emission is a thing without needing to rez out special sound emitting entities, I’m totally fine with it. If anything, I could see it being interesting to take the sound entity properties as an easier way to set up scripting parameters.


#9

I wanted a nice easy way to play music everyone could use no extra complexity. For a while it seemed like we were getting that.
Oh well lol


#10

I agree, I’d still like the option to do scripting as needed, but I’d like (for example) the ability in the create app to create a room, type in the dimensions, surface materials of the walls, ceiling, and floors. and have all the sound entity parameters propagate into the sound zone of that room.

Hope that makes sense, I’m actually doing that now IRL for a tour coming up in a neat software program called MAPP XT.


#11

And I believe it should be added to the Zone menu.


#12

The issue is that audio doesn’t work so well in squares (even square waves). The best I can think of are light entities since a good chunk of how they work could be viewed in audio property form as well.


#13

Agreed, I like the light type entities idea, but you could just as easily use a sphere shape, as an Audio Zone.

The way the Audio Zones are done at the moment on the server is as a cube shape, so I figured it would be easiest to start from there.