New - in-app menus for HMDs


#1

The latest builds of Interface contain a new mechanism intended to ease the use of the application in a head mounted display: menus rendered inside the 3D view. This new menu view can be activated by tapping the ALT key.

This will enable users to access functionality via the menu that previously would have been otherwise inaccessible to HMD wearers. Of course right now quite a large number of the menu items still open dialogs that are not rendered inside the HMD, which will have to be addressed moving forward by moving more of the user interface into the in-app overlay. Currently only the address bar and login dialogs have been ported over in this fashion, but more components will follow. In time, users should hopefully be able to create and modify content, interact with the marketplace and alter user preferences without having to take off a headset or switch into non-VR mode.

Right now this menu is available regardless of whether you’re in an HMD or not, though it could move to an HMD only feature in the future.


#2

Using the Alt key for this function (along with using Alt drag to copy selection when in edit) make it very hard to use Alt drag for camera positioning.

While I understand the desire to seamlessly support HMDs, unless there is a better way to move the camera quickly for positioning things in edit or setting up the camera for screen shots, I think you should be able to turn both these functions off. Or better yet, use the Space Bar for the menus instead of the Alt key might be easier to find when you can’t see the keyboard.


#3

I assume this is with QML also? The integration looks really cool in general. I was looking to see if you can create dialogs from script at the moment. It all seems pretty hard-coded into C++ for that though.
Will have to try and get my head round this all some more later.


#4

I wasn’t aware of the use of the alt-drag feature to reposition the camera. Right now any key click other than alt will disable the menu from appearing when alt is released. I will modify the code so that a mouse clicks (as well as wheel and touch events) will inhibit the menu in the same way.

Using Space Bar might be problematic. How does a user bring up the menu if they have a QML dialog up on the screen with a text field focused? Right now these boxes consist of the address bar and the login dialog, but this number is going to grow.


#5

Yes, the new menu is QML based. Basically any UI we want to be able to render inside of the 3D scene needs to be QML based in order to be responsive, because Qt provides a mechanism for rendering QML directly to an OpenGL texture for use by the application. Any other method of rendering UI would require us to copy the UI image each frame (or at least each frame that the UI surface changes in some way) from the CPU to the GPU, which introduces latency and needlessly consumes bandwidth.

Right now there is a mechanism to create a new window from a script and point it at a URL. You can see it at work in edit.js:

var MARKETPLACE_URL = "https://metaverse.highfidelity.com/marketplace";
var marketplaceWindow = new WebWindow('Marketplace', MARKETPLACE_URL, 900, 700, false);

So… click the ‘market’ icon on the screen and a new desktop window appears with the market page in it.

However, right now, there’s no mechanism I know of to allow a script to create or point to a QML source and have it show up in a window. Technically this isn’t really all that hard.

However, what’s a bit harder is figuring out how to create a bridge between the QML engine and the JS engine running in the app. Presumably if you create a QML UI, you want to actually be able to do something in that UI that could then be acted upon by the script. There has to be some kind of callback mechanism whereby QML dialogs can send data back to the scripts that created them.

Additionally, we’re trying to evaluate whether this is in fact the right direction to go for user developed UI. Not everyone out there who wants to create some kind of UI from a script wants to learn QML. On the other hand, it we provide mechanisms for loading a static HTML document (that could contain some arbitrary JS in which to create the UI), as well as QML, and provide both of them a similar mechanism for triggering callbacks to the original script, that should probably cover all the bases.


#6

@Jherico said
"Right now this menu is available regardless of whether you’re in an HMD
or not, though it could move to an HMD only feature in the future."

I wonder what would make anyone think we want this feature when not in HMD?

Seems to me it is only required in HMD so lets keep it in HMD and away from non HMD (in the not too distant future), please.


#7

I can imagine that someone might prefer consistency of UI between VR and non-VR modes. However, I’m sure that it’s something that could be exposed in the user preferences.


#8

I think QML looks feature rich, editing by hand could be a bit tedious, it seemed like you would typically use QT Creator to author the files visually.
Some form of encapsulation would be preferable, but I’m keen to start using it as is.