How are supposed to be used Overlay 3d of type cube and why I cannot see them?


#1

Scripter newbie here. I tried to grok the Overlays at white belt level so I was able to run and understand how it works the attached script#1. It works but left me with the question: how can I find all overlays not deleted? Is there a function to fetch all of them (2D overlays)?

Then I tried to build up some overlays with type 3d as suggested by the documentation as this link is suggesting https://docs.highfidelity.com/api-reference/namespaces/overlays#.deleteOverlay (see script #2) and I coudnt see any “cube” in front of my avatar or anywhere in the desktop interface, so I am quite puzzled where these cubes are supposed to be viewed are they only visibile in VR?. I am on MacOs now. The script says there is a cube but I cannot spot on my interface.
So where should I be able to see them and what are they supposed to be used for?

script #1:

var overlay = Overlays.addOverlay("text", {
    text: "my text",
    font: { size: 24 },
    x: 200, y: 200,
    width: 500, height: 500,
});


Controller.mousePressEvent.connect(function (event) {
    var clickedOverlay = Overlays.getOverlayAtPoint({ x: event.x, y: event.y });
    print("Clicked: " + clickedOverlay + " " + " our overlay is " + overlay);
    if(clickedOverlay && clickedOverlay === overlay) {
        console.log("deleting overlay")
        Overlays.deleteOverlay(overlay);
        overlay = null;
    }
  });

script #2:

var overlay = Overlays.addOverlay("cube", {
    position: Vec3.sum(MyAvatar.position, Vec3.multiplyQbyV(MyAvatar.orientation, { x: 0, y: 0, z: -3 })),
    rotation: MyAvatar.orientation,
    dimensions: { x: 0.3, y: 0.3, z: 0.3 },
    solid: true
  });
  var type = Overlays.getOverlayType(overlay);
  print("Type: " + type);

#2

mmmhhh. I was “able” to see in desktop mode these white cubes which seem to accompany my avatar, but it is really difficult to interact with them, so this seems really a kind of 3D hud, which is almost impossible to use and even see in desktop mode.
Are there any interesting examples on how these are used? Maybe the tablet is done in that way when you are in VR mode, right?


#3

Overlays should be completely usable even in normal desktop model; Edit / Create App forexample uses them extensively, and the Tablet also uses it as reference if you have it active instead of the desktop UI.

If not in VR mode, you will have to however convert the “Viewport0Space” to World coordinates for Mouse Events.

For that you have the Camera.computePickRay api that takes 2d coordinates and converts them to a Raypicks (a type of raycast)

For HMD controllers you use the controller directions and rays that provide instead.


The following is an entity script you can attach to an entity that creates an overlay parented to the entity. Meaning everyone sees their own instances of the overlay, and see it in different colors after interacting with it.


(function () {
    return {
        preload: function (entityId) {
            
            var properties = Entities.getEntityProperties(entityId, ["position", "rotation"]);
            var context = this; // to keep context
            var overlayId = Overlay.addOverlay("cube", {  position: Vec3.sum(properties.position, Vec3.multiplyQByV(rotation, {x: 0, y: 3, z: 0})), 
                                                    rotation: rotation, 
                                                    solid: true, 
                                                    parentId: entityId,
                                                    dimensions: {x: 1, y: 1, z: 1} });
            
            // The Following is wrapped, to make sure we can keep context. Otherwise the connected event wont actually have any context to what "this" is.
            context._contextOverlayMousePressEvent = function(event){
                //  Convert Screen interaction into a pickray from the perspective of the camera,
                var pickRay = Camera.computePickRay(event.x, event.y); 
                /*
                    PickRay consists 
                        origin: vec3
                        direction: Unit vec3
                */
                var intersectionResult = Overlays.findRayIntersection(pickRay, [overlayId]); // findRayInteraction(PickRay,)
                if (intersectionResult.intersects) { // It Intersect with some event
                    return context._overlayMousePressEvent(event, intersectionResult);
                }
                return false;
            }      
            // When something is called inside a signal even, it is contextless. By using a wrapper like above,
            // we make sure it is called contextless, but the context remains from its parent definition
            Controller.mousePressEvent.connect(context._contextOverlayMousePressEvent);
        },
        // Not Predefined, this is something we simply call from _contextOverlayMousePressEvent, while maintaining this context.
        overlayMousePressEvent: function (event, intersectionResult){
            /* 
            intersectionResult https://github.com/highfidelity/hifi/blob/8201997a16e206fe54acd4a685a30172379764f8/interface/src/ui/overlays/Overlays.h#L58 contains 
                intersects: boolean
                overlayId: string
                distance: float
                parabolicDistance: float
                face: integer (not sure)
                surfaceNormal: vec3 (direction the face is at)
                intersection: vec3 (where in the face)
                extraInfo: object (not sure)
            */

            // Note that by default this triggers regardless of distance, so use the distance to filter interactions.
            Overlays.editOverlay(intersectionResult.overlayId, {color: {red: Math.random() * 255, green: Math.random() * 255, blue: Math.random() * 255}})
        },
        unload: function(entityId){
            // Good Practise to always unload controller stuff on entity destruction / script end.
            Controller.mousePressEvent.disconnect(this._contextOverlayMousePressEvent)
        }
    }
});

I havent tested this inworld as currently not home, so all of the above is by theory.

You also have a whole bunch of experimental ones which I dont think anyone has tried yet, so I wont list them here until I’ve gotten a crack at.

If you are feeling experimental you can find them here under signals, they seem like they would simplify this rodeo tons, but lets see.

For 3D Overlays you have

Overlays.findOverlays(vec3, float) similar to findEntities


#4

Yes, but I didn’t see anything similar to that for finding 2D overlays. The only thing I was able to find is the following which together with a mouse click can get many stray overlays…

var clickedOverlay = Overlays.getOverlayAtPoint({ x: event.x, y: event.y });

But I would have loved to return all the Overlays currently active, something like

Overlays.getAllOverlays()

but I was not able to find it, so I was concerned my Interface can go around polluted with dozens of hidden or invisible overlays :frowning:


#5

Usually that is solved by simply restarting the client:

You cant really distinguish what an overlay is even if it gave you all the information about the overlays available