Rendering visual and audio information differently for different users


#1

Hi, I’m new to HiFi and would like to use it for a research project I’m working on.

For my project, I need to:

  1. Send “social cues” (head direction, location, is user speaking) of each user/avatar to an external server to analyze the social interaction in real time.
  2. Manipulate the way avatars are rendered separately for each participant, based on information from an external server. For example, by “fading out” (reducing oppacity / blurring / other visual manipulations) avatar X to participant Y but not for all participants, or making everyone see the rest of the avatars as looking directly to them. Similar ideas will be applied to the audio: participant X could hear participant Y but not participant Z while participants Y and Z hear everybody.

Although I’m not quite sure how, reading through the scripting docs it seems that the 1st task will be possible to achieve with HiFi. However, I don’t know if the 2nd one is feasible or not.

Does anyone have any experience with something similar?

Is it possible to manipulate other users avatars (hiding / muting / apply more involved rendering manipulation like changing head direction and blurring) in client side scripts?

I will surely use the next few days to find out by myself, but any clue from experienced users will help a lot!

Thanks,
Tom


#2

Under the people menu item, you can select users to selectively mute or ignore. I haven’t looked at those scripts to see how they work but it sounds related to what you want to do.

The users API defines Users.ignore and Users.personalMute
https://wiki.highfidelity.com/wiki/Users_API

I’m not 100% sure but I think the file that controls that UI is pal.js


#3

Wow! Thanks a lot, the Users API looks promising. It looks like hiding / muting users individually wouldn’t be a problem. Still, I can’t see how to manipulate head direction individually, to make two different users see a 3rd user head direction differently.
I will start playing with this tomorrow and update.


#4

Rereading the documentation, it seems that the 1st task will be trivial using the MyAvatar API in a client script.

@Alcofribas, the interface you show is indeed pal.js. However, it seems that this script loads and call functions from Pal.qml and doesn’t rely only on the Users API. For example, changing the audio volume of another user is impossible with the Users API. pal.js does that by calling Pal.qml that (as far as I understand) have access to lower-level details.

This arises another question. Is it possible to create qml files that manipulate avatars. Then load them and call them from JS without having the qml be part of the build of HiFi? If so, and in case I can’t find a solution for my needs in the JS API, I will be able to write the low level avatars manipulations in qml and add them to client / assignment scripts.

I’m thinking about taking this direction because I can’t find any other way to manipulate the audio and visuals of another avatar separately for different users. In terms of visuals, we can only manipulate our own avatar using the MyAvatar API, and the Users API supports muting and hiding (ignoring) other users but in a symmetric way (if I mute someone I won’t hear him and he won’t hear me), which is not what I want.

Any other suggestion?


#5

Yes. You can load .qml files on-the-fly. No, not ALL low-level manipulations are able to be performed there as far as I know…


#6

A script you might want to look into is the bubble script:

It basically creates an area around the avatar that if the avatar is in it, they will not see the other user, nor will the other user them. They will also not hear each other, but everyone outside it will see and hear both of them.

However currently you cant really control the placement of this area, as its only around the avatar it originates in. Technicaly speaking the client may need some modifications for this to work in a controlled manner as you spec, but it is somewhat already in existance.


#7

Thanks for all the suggestions. It seems that bubble.js calls Users.toggleIgnoreRadius / Users.enableIgnoreRadius / etc. Not exactly something I can work with. In addition, the most “involved” manipulation I saw with qml files is in Pal.qml, with relatively simple audio levels manipulations. I think that HiFi won’t work for my needs :frowning:.

On the other hand, I discovered a really interesting project with supportive community, so thanks for that!


#8

Keep an eye on it though! It is constantly in flux

They just added support for Overlays to have bones, allowing creations like the Doppleganger which allows you to mirror your own avatar (that only you will see), and adding support for Overlay Avatars which only show for who runs the script, among stuff :slight_smile: This combined with local audio playback and clever scripting could allow for some interesting usecases

Nothing however is out of the box, and this is atm a tinker’s playground.