Anybody find a way to stream image frames into a texture yet?


#1

… I think that’s the last puzzle piece ahead of having a way to bring our Android phones into Interface/VR with us.

Initially I’m just thinking about practical stuff like the ability to screen/answer calls and to check text messages without removing your HMD.

Within those use cases, consistency seems more important than frame rate – so even a way to stream framebuffer data in at 10fps could work initially (for tasks like swipe-to-unlock, navigation and pecking at the onscreen keyboard).


#2

I don’t think we can gyt. Especially since we cant set textures to specific materials in object within the engine.

Only way I can think of is using shaders, or the the web page but even so you can apply those to non primitives, and there is no known support for webrt in the qtbrowser


#3

Ahh well – that was essentially my assessment too, but ooooooh would it be cool to stream this data in reliably.

FWIW it is possible to cheat temporarily by using two Image3D overlays and buffer swapping (when you reset the overlay url it disappears completely for a moment, but using two alternatively achieves semi-stable results). Too bad this technique seems to a) rapidly fill up your disk cache and b) rapidly trigger some kind of memory leak.

But still… easily one of the weirdest/coolest experiences yet walking up to my phone in VR!!!

I wasn’t wearing my HMD here while testing, but I imagine some day we’ll be able to override the front-facing camera and have it see a mug shot of your avatar (then you could simply use Facetime or Skype etc. from within VR also).

And the best part? In VR when your phone acts stubborn you have the very real seeming option of throwing it across the sky. :smile:


#4

That’s a problem with all overlays. It’s a shame they don’t plan to fix it! I get the impression that they don’t give a sh*t about memory problems.


#5

@Cracker_Hax Are you talking about script resources not being automatically cleaned up?


#6

Well, I am not sure because obviously there is memory leaking going on somewhere with overlays. If you make and destroy them eventually you cannot make them anymore and everything gets laggy. This is why scripts need to be automatically managed. You can’t really expect to waste thousands of users time tracking down memory leaks when it turns out it is not their problem in the first place.


#7

This is why scripts need to be automatically managed.

I think there are 2 parts here:

  1. Automatic cleanup when a script ends
  2. Automatic cleanup (garbage collection) while a script is running, for resources that are no longer referenced

Both are mostly non-existent. 2 is tricky because you don’t always want to destroy a resource even if it is no longer referenced (Audio injectors are a good example, where you wouldn’t want it to be destroyed before the clip finished). That said, I (and probably most other devs) would agree that we should at least fulfill 1. I have it on my todo list to add automatic cleanup to overlays so they don’t stick around if you don’t clean them up or if the script crashes (!) before you even have a chance to clean them up (if other arguments for automatic cleanup aren’t convincing enough, I think the crash case should at least be).

The problem described in this post really shouldn’t exist, and is a bug, although it sounds like it isn’t a resource that the script is leaking, but an issue with textures being leaked or not being removed from the texture cache when they should be.

@humbletim Sorry if I hijacked your thread a bit here, this is some really awesome work!


#8

I think they stick around even when you delete them. I find that when I am debugging an overlay script I have to rerun it a lot and even cleaning them up “properly” they get to a point to where the overlays don’t even show up anymore.


#9

Interesting, do you happen to have a repro? I can look into it when I make the changes I mentioned.

Ryan


#10

@huffman – no worries.

Experimentally I count five different scripting behaviors: pre-entity, entity, include, load, AC. Some of those seem to require different cleanup rituals from the others. Scripts can also be attached indirectly to an Entity, which makes cleanup suddenly ambiguous. The automatic script reloader (inadvertently) introduced randomness to the equation.

So… automatic cleanup sounds great!

And beyond actual native leaks, I can suggest that a system-level JavaScript veneer works well for orchestrating this kind of cleanup. Essentially a .js file that gets baked into Interface like the current shaders and then bootstrapped into scripts automatically. Node.js does this to good effect. This also provides a means for maintaining backwards-compatibility with existing scripts as the C++ side changes.


#11

Not a particular one, no. It is just something I have noticed working with overlays a lot. It’s hard not to notice, really.