Suggestion: Javascript interface for Matrice Manipulation / Image Processing


#1

This may sound a bit crazy, and quite expensive in terms of performance, but hear me out:

A use case::

Have a script that generates voxel terrain from a file defined in the script. It then translates the image (with size limitations however, image should not be larger than N mb (use best case scenario here, they can get quite large in memory) and could only be done when interacted with) into a chunked voxel terrain:

For the script: Alpha channel could defined as height,
While colours them selves would define poly materials used (so FF FF FF / 17 = F0F0F0 in which the first significant bytes per color) are used to define the material used for the topmost voxel.

To do this,

  1. the scripter would need some way to load up an image in JS
  2. Store in into the object (do able)
  3. Convert to 4 Matrices (RGBA). (not doable natively, but there are libraries susch as Sylvester, but it doesnt match what is already available in client)
  4. Grab a specific section of each of the images (such as 32x32 area on top right)
  5. read each of the matrices

Other Use cases:

  1. Generate a texture
  2. Generate terrain
  3. Manipulate textures
  4. Manipulate terrain via custom “brushes”
  5. “Biome” sorta like variation Polyvoxels
  6. Nearly Limitless other stuff. Etc.

However instead of this, could just have a Image processing interface in the interface: This might allow the utilisation of the Graphics Card: it would call a defined callback that returns the 4 matrices.

Could also be awesome if we could use this same method to set textures onto objects, and maybe even animate them similar to sprite sheets: or even procedurally generate textures.


#2

I’m sure it use to do something like this. You could import a png image that coudl be voxelisied

https://alphas-new.highfidelity.io/uploads/highfidelity/187/fbec79a74d75b3ee.jpg


#3

Something of the sort, but the discussion is here to open up the scripting engine to also allow to do matrix math in JS, rather than just vectors or quaternions.

Metavoxels, if i have heard right that was actually using existing js interfaces specifically made for the voxels to read the image files and generate voxels.

Allowing matrix calculations to be done in JS would allow to do alot more than just selectively get specific areas of heightmaps.
Or even generate terrain maps or textures from algorithms, such as Perlin Noise.

Could also open up in generating textures them selves for objects (as soon as that is explained as well, that is applying textures / materials to objects instead of just loading defaults from model file.)


#4

Native support for procedural meshes, materials and textures would be awesome!!

For now I wonder if such a terrain system could be prototyped using existing parts?

Like stock Cubes for voxels and a WebWindow as the terrain preprocessor. WebWindow has access to HTML5 Canvas (which can be used to load images and get at raw pixel data) and then HiFi EventBridge could be used to broker processed voxel details back over to the HiFi side.

Also for math/matrix support you could try using my glm-js wrappers. Some glue code is currently required to use glm-js with Hifi, but in some future release I’ll probably make it compatible out-of-the-box.


#5

There actually already is a voxel system in place which is called metavoxels. It allows for chunks of 32^3 * 255 (presuming material) voxels. More here when they first popped in. https://alphas.highfidelity.io/t/voxels-are-back-ish-report-list-of-suggestions-for-devs/

the glm js wrappers seems to already go all the way to vectors and quaternions, as of the moment it doesn’t cover matrices.


#6

I think maybe you meant to say HiFi there? (since it already has Vec’s and Quat’s, why not simply add a Matrix class?) Also I’m assuming by matrices you mean stuff like mat4’s?

In any event glm-js supports the same style matrices as GLM, which coincidentally is what the HiFi C++ side standardized-upon some time ago. Maybe using the same standard on both sides can help find better optimization paths.

As a reference point for the various ways to ponder matrices in JavaScript, here’s a quick sampling of creating a translation matrix across several different libraries:


// glm-js (GLM style)
var m = glm.translate(glm.mat4(), glm.vec3(1,2,3));

// glm-js (how I would do it)
var m = glm.mat4();
m[3].xyz = [1,2,3]

// Sylvester
var m = $M([
  [1,0,0,0],
  [0,1,0,0],
  [0,0,1,0],
  [1,2,3,1]
]);

// gl-matrix
var m = mat4.create();
mat4.identity(m);
mat4.translate(m, [1, 2, 3]);

// tdl-fast
var m = new Float32Array(16);
tdl.fast.matrix4.translation(m, [1,2,3]);

// THREE.js
var m = new THREE.Matrix4();
m.makeTranslation(1,2,3);

//eof

#7

Yes, I did mean Hifi, and yes, basically open up the Mat4 objects for JS, similar to Vec3 and Quat.


#8

Ahh, OK – there isn’t a Mat4 class yet on the C++ side to open up, so I suppose it’d have to be specified.

The current Vec3 and Quat classes are thin wrappers around glm::vec3 and glm::quat, so maybe Mat4 is just a limited set of matrix operations around glm::mat4. Probably would need a Vec4 class too then.