Avatar Standard Compendium


#1

Note this a work in progress and open for anyones input as Hifi is also in Development.
After all, we should all agree on one most will be happy with, as having conflicting standards would conflict with everyones interest. Once this is in a ready format and the standard requirements have been met by High Fidelity, I will apply to have this added to documentation.

Remember that this is a recommendation to best cross-compatibility with everyone else content, who follows the standard. It is not written in stone, you do not have to follow it: but its best for the end users if it does.


Maya users can simply use the HumanIK skeleton as that has pretty much all but the optional stuff
If you using Blender, use

Instead

Proposal: Avatar Standards

Version 0.0.5.0

Last Updated April 7 2018

  • Updated to include Material Entity use and meeting a pillar of this Standard
  • Added more description on the naming convention

Lexicon

  • Avatar - The being which behaves as the selected representation of a person or npc
  • Mesh - Topology for the avatar model. Without this, the avatar is invisible
  • Bones - A Component of a Skeleton that defines a “limb” such as an arm, leg, etc that might have to be animated as a separated limb
  • Skeleton - Bones, Armature used by Avatar mesh and animations to animate to avatar the mesh
  • Rigging - Creating a structure of the Skeleton of the model, and the various constraints/tools which affect that skeleton
  • Weights - Per vertex definition which define how each bone in the skeleton affects the mesh.
  • Skinning - The act of painting weights for the skeleton.
  • UV mapping / unwrapping - Defining how the topology maps to a 2D texture
  • Shapekeys / Blendshapes - variations of the topology that act targets how the mesh is modified to create various “shapes”
  • FST - Faceshift Transform file - used to define the faceshift blendshape bindings
  • End Users - the users of the avatar
  • Avatar Base - this is the base avatar without any custom modifications.

Requirements of the Standard from High Fidelity

  1. FBX Processing for all 3D suites must behave similarly. Currently, Now Fully Realized.

  2. Scripts must be able to modify bone’s scale and translation as well as rotation. Currently rotation and translation is supported. Scale is supported through animation on entities, but remains untested via scripts on avatars.

  3. Avatar animations played should not override bone scales or translations set by scripts. instead of overriding, they should be additive. Currently, animations played back on avatars do not scale avatar, but do affect entities, there also isn’t away to do relative translation that is not effected by rotation or translation overrides

  4. Allow scripts both to lock joints from being animated by animations but also allow them to open up joints to mixing of animations and script set position

  5. Don’t override rotation if setting translation of a bone: Currently if you set the translation of a bone, you also set the bone to its rest position, instead of adding or mixing with the animation system.

  6. Scripts must be able to read what Blendshapes an avatar and entity mesh has available. Currently no such plans are in the process.

  7. Blendshapes on Clothing, Entities and Attachments should be supported.

  8. Allow Textures on Avatars be defined like they can be on Entities - With Material Entities this is now doable, only regulard attachments cannot have Material Entities, yet.


Pillars of the Standard:

  • Bones are used to animate the characters limbs, and define the scale variation of limbs.
  • Special-Bones are used to adjust the avatar’s shape.
  • Blendshapes animate the face
  • Naming conventions are similar if not same in all avatar skeletons.
  • Animations and Avatars not using this standard cannot claim to use this standard.
  • Scripts use this convention to create interfaces which end users can use to customize their avatar

Baseline Humanoid Avatar Skeleton:

The standard humanoid Skeleton of the avatar should follow HumanIK Skeleton / Mixamo.
This skeleton system will work with the input systems already in place in High Fidelity.
however with some modifications mostly to do with the eyes and additional spine / neck components:

Hips – Must always be present. This is checked when the avatar is loaded up.

  • LeftUpLeg
    • LeftLeg
  • – LeftFoot
  • – - LeftToe*
  • RightUpLeg
    • RightLeg
  • – RightFoot
  • – - RightToe*
  • Spine
    • Spine1
  • – Spine2
  • – -- Neck
  • – -- - Head
  • – -- – LeftEye
  • – -- – RightEye
  • – -- LeftShoulder
  • – -- - LeftArm
  • – -- – LeftForeArm
  • – -- – - LeftHand
  • – -- – -- LeftHandThumb1*
  • – -- – -- - LeftHandThumb2*
  • – -- – -- – LeftHandThumb3*
  • – -- – -- – - LeftHandThumb4*
  • – -- – -- LeftHandIndex1*
  • – -- – -- - LeftHandIndex2*
  • – -- – -- – LeftHandIndex3*
  • – -- – -- – - LeftHandIndex4*
  • – -- – -- LeftHandMiddle1*
  • – -- – — LeftHandMiddle2*
  • – -- – -- – LeftHandMiddle3*
  • – -- – -- – - LeftHandMiddle4*
  • – -- – -- LeftHandRing1*
  • – -- – -- - LeftHandRing2*
  • – -- – -- – LeftHandRing3*
  • – -- – -- – - LeftHandRing4*
  • – -- – -- LeftHandPinky1*
  • – -- – -- - LeftHandPinky2*
  • – -- – -- – LeftHandPinky3*
  • – -- – -- – - LeftHandPinky4*
  • – -- - RightShoulder - same as left, but prefixed with Right
    (68 standard bone nodes)

NOTE: Finger#4 is always at the tip of the Finger. Finger#1 is NOT the metacarpal, instead its the first joint between the Proximal and Intermediate

The above standard skeletal structure is for bipedal plantigrade humanoids, but it should also work with quadrupedal stances and digitigrade (as if walking on high heels foot elonged to the toes), and unguligrade (as if angled to toes, or same as digigrade without the to bone) foot structures, especially as the procedural animation scripts for walk scripts get better:

Do expect that animations following different skeletal shapes to make you move unnaturally if the animation is designed for a different foot structure, but you may always mask out foot and toe animations for better consistency.

Following this standard will allow end users to use their input devices to control their avatars arms and fingers (if they have any),

Bones marked with an # and * are optional. Bones marked with # should NOT be animated as they are used for customization of the avatar (eye width). Unfortunately, You can a only have 0 to 2 rotating eyes, so having a multiple eyes using this standard is not possible. (I Suggest engine should support for Eye# + LeftEye and RightEye which would work same as the current eye bones (look at targets)

On Extras:
  • If The avatar’s eye distance is adjustable, The avatar skeleton should have an “eye socket” bone to allow for unified bone translation and scaled for the position of the eyes and eyelids, without the eye bones affecting the eyelids them selves.

  • Animations should only effect the ones defined in the baseline standard

  • If making only the head of an avatar, you may start Neck from but must have a root before it.

The standard must be extendable, so custom bones are allowed: just note that for most of them you must create your own animations if deviating:

Custom Bones

If using custom bones in the avatar, make try to keep the bone amount less than 80 to save on transmission costs. Each bone animated will cost 1 KBps per transmission of either animation or script. Technically speaking you could still make a multi legged insect or winged feral dragon as an avatar, since you can simply remove the fingers and add more custom bones. Be creative! but do remember that this would deviate from the humanoid skeleton standard.

Custom Bones naming conventions:

We will adhere to a Capitalize naming convention so <Side or #> like with the HumanIK Skeleton. _End is reserved for generated bone ends.

So Forexample:

If you avatar has rotatable ears

LeftEar
RightEar

or if you avatar has multiple tails

Tail1 - First bone of the first tail
Tail# - # bone from the first tail
#Tail1 -  First bone of # Tail
#TailX - X bone from the # tail 

Scripted Bones:

Sometimes you may to have a custom script do something with an a custom avatar’s bone, we would then describe the bone as

myBone#

where as action would be the name of the script. In this case, the lowercase “my” would be the what ever unique word you want to use.

These are all bones that should not be animated by an animator.

However there should be a few reserved bone action names: sim and morph which should be available in avatars that want to follow a standard:

Sim Bones

Prefix “sim” would be for physics simulated bones: Forexample, clothing, hair, tails.

Forexample A full Cape that surrounds the avatar.

simBackCape1 - first bone of the cape, center back.
simBackCape# - # bone of cape, center back. etc
simFrontCape1 - first bone of the cape, center front.
simFrontCape# - # bone of cape, center front. etc
simLeftCape1 - first bone of the cape, Left,
simLeftCape# - # bone of cape, Left,
simRightCape1 - first bone of the cape, Right
simRightCape# - # bone of cape, Right

or a simulated tails (instead of animated through animations)

simTail1
simTail#
sim#Tail1
sim#Tail#

Morphing Bones

This is a new ademendum to this standard added on 2017.

Bones currently are completely adjustable via script, however there are some caveats that have been added to the Pillars.

These will be bones which the user should be able to adjust the position of, and as they are adjusted, allowing for customization of the avatar. They should always be parented to the nearest parent bone, and following the following convention

morph<Parent><CustomName>

Forexample, if I want to have an adjustable nose: I’d use

morphHeadNose or morphHeadMouth

A script can then detect these and filter them accordingly and allow for an interface that the user can use to adjust their looks. More details comming

(This part is added on May 14 2017. If it still hasnt been updated, bug @Menithal… he gets easilly distracted.)


Avatar Supported Blendshapes Reference:

Unlike Bones, _ usage will be quite extensive due to the nature of FaceShift standard.

Convention is: L = Left, R = Right. Exception is Jaw. (not sure where its comming from but just go with it)

Basics

  • EyeBlink_L: Blinking action for the Left eye
  • EyeBlink_R
  • JawOpen: Opening the Jaw

Advanced:

  • BrowsD_L: Corner of the brow going down
  • BrowsD_R
  • BrowsU_C Center of the Brow going up
  • BrowsU_L: Corner of the brow up
  • BrowsU_R
  • JawFwd: Jaw being pushed forward
  • JawLeft: Jaw being rotated to the left slightly
  • JawRight
  • JawChew: The act of clenching the jaw at the end of a chew
  • MouthFrown_L: Corner of the mouth down to a frown,
  • MouthFrown_R
  • MouthSmile_L: Corner of the Mouth up to a smile
  • MouthSmile_R
  • MouthDimple_L: Cheek inward
  • MouthDimple_R
  • Sneer: Act of Sneering
  • Puff: Puffing out the cheeks
  • LipsFunnel: Funneling of the lips, similar to pucker, but open.
  • LipsPucker: Puckering of the lips
  • MouthLeft: Mouth oriented to the left side
  • MouthRight:
  • LipsStretch_L: Stretching of the lip on the left side
  • LipsStretch_R

Subtle Detail / Microexpressions:

These are blendshapes that not everyone will notice, but lend to character if done in expressive or subtle way (note some of the images may be reverse of, just use the correct side :slight_smile: )

Remember looking up some anatomy helps as well, as alot of face expressions effect more than just the part in question (for example, both frown and smile effect the neck too!)

  • CheekSquint_L: Cheek in a state of a squint
  • CheekSquint_R:
  • EyeSquint_L: Eye Lid in a squint
  • EyeSquint_R:
  • EyeDown_L: Position of the eyelids when the eye is looking down
  • EyeDown_R
  • EyeIn_L: Shape of the eyelid when an eye is looking towards the nose
  • EyeIn_R
  • EyeOut_L: Shape of the eyelid when an eye is looking out of face
  • EyeOut_R
  • EyeUp_L: Shape of the eyelid when the eye is looking up
  • EyeUp_R
  • LipsUpperClose: Upper Lips rolled in
  • LipsLowerClose: Lower Lips rolled in
  • LipsUpperUp: Upper Lips rolled out
  • LipsLowerDown: Lower Lips rolled out
  • LipsUpperOpen: Upper Lips opened up slightly
  • LipsLowerOpen
  • ChinLowerRaise: Lower Chin Slightly up
  • ChinUpperRaise

Avatar Customization Blendshapes:

####DEPRECATED in favor of bone customization as it is taking too long for custom blendshapes to be implemented. See Thread History for detail.

Optimization and Filesize

Content creators will have limited bandwidth on servers (read small print on any unlimited host plans) so optimization is important, for both the end users and content creators. The more polygons and larger textures you use, the more bandwidth you are using from your servers per load. Optimally Best to keep Avatar models under 20 mb.

UV Mapping:

You can only have a single UV map per mesh in the object.
Make sure you fill as much as possible of the textures without too much distortion.
If you can repeat textures (such as for cloths, etc) do so. All of these help the size of the avatar, and allows you to push more detail via polygons or textures.

Textures:

Compatibility between avatars and textures will be really difficult if not impossible, however the rule would be to try to keep total size of all the textures per avatar below 8 mb. They should be always smaller than 2048 squared, unless all the textures are in a single file. If using multiple texture files, then smaller the better, especially if you can make the textures smaller. Remember that you can get a lot more detail through roughness and bumpmapping, than just textures. It is suggested that you keep Albedo at a smaller size than your roughness for best detail through light reflection instead of color variation.

Avatar LOD Polygon targets:

Depricated, as of the moment, both avatars and entities do not use these information and LOD hasnt been in focus for a long time

High LOD < 30k polygons.
Medium LOD < 15k polygons.
Low LOD < 4k polygons
Lowest < 2k polygons

Generally a good rule of thumb for LOD levels would be a
1:2:8:16 ratio

Until LOD models are re-implemented into high fidelity, that is, target ones avatar between High and Medium LOD, but the lower the better.

to be continued…


WIP: MHX2 importer mods to help with makehuman to blender to hifi avatar creation
Animation with Heads
Problems getting custom avatar to work
Make human 1.1.0
In-world meeting Friday 2:30 - 4 PM at Playa
Q How democratic do you want highfidelity to be?
The Welcome Thread. Introduce Yourself!
Custom avatar appears, then disappears
How can I go about creating a custom avatar?
High Fidelity Blender Add-On - Version 1.0.4 Released
Custom NPC model
High Fidelity Blender Add-On - Version 1.0.4 Released
No avatar, no business
3D Avatar Modeling Critique Wanted
Re:Mixamo Avatars Heads up!
MakeHuman to Blender to High Fidelity ATP
Allow us to set Textures on Avatars like they work on Entities
#2

@ozan this is where we [Alphas] will make our stand on avatar’s.


#3

This is an unbelievable amount of work you have done, @Menithal. Thank you!


#4

Very nice work. Is a tail on a humaniod a consideration?


#5

That looks very impressive, good job!

Is this volunteer work? a proposal? Why is not HF doing this by defining all these as part of the software development, and hiring somebody to do it… Is this the same case as the “Text Chat” being left to the community? This is the first Alpha that I’m a part of, i assume i don’t understand this processes


#6

@VR_Architect - yes tails are considered in the spec however they are additional bones and should generally be procedurally animated as extra bones instead for base avatar as it would over complicate the animation process for people not using tails. Youd also run the problem of some individuals running multiple tails… So instead I just define the naming conventions for them as Tail# and Tail#_# for each tail, and bone in the tail. Because these are procedurally animated I’d rather leave that spec to the ones creating those scripts.

I’ll go into more detail on them as week nears its end, which also has a work list ticket that I’m looking into.

@ncsolar - as of the moment its a proposal for extension of the current standards, which I am opening up for discussion. I will be later move it into a worklist item to apply for small reward (if accepted and discussed by community)

Once its done , I’ll apply for migration to high fidelitys documentation. . Generally these would be done someone interior but due to the open source nature, HiFi uses external developers to assist with the development and community progress.

As you are able to create anything these standards are present only to make general content cross compatible with each other;

They already have some standards which use the mixamo skeleton and faceshift in their documentation but i am expanding it here to allow for customisation options that have not yet been scripted and get some optimisation to be with the standard as well, or trying to make a future proof standard.

its a decision of the content creator to follow these or not, but generally following is better for end users


#7

Updated spec with info on custom bones, foot structures bone purpose re-allocation, input control creativity, and the sort.


#8

typo - Front cape bone is duplicate of back cape bone info.


#9

Excellent! Can’t say I understand all of this entirely, but enough to know there’s an evolving, comprehensive, standard blueprint we can all learn… to hang our avatar creations upon in virtual space! AND with much room for considerable customization of positions/proportions to boot…[chuckles] - brilliant stuff, Many thanks @Menithal, I hope to understand/use all of it soon.

I have been retopologising my first attempt at making a working humanoid avatar from scratch as you know. I have some basic understanding of LOD and the three ideal levels you mentioned. I have been going back and forth with the figure trying to work out optimal levels of geometry and trying to keep them more dense where needed etc. it’s currently at about 6.5 k at lowest subdivision since I added more to face / lips and joint areas etc.

I see that an exported makehuman avatar is about 13k naked - which would almost suit your suggested middle level LOD target.

I get the impression from this that I need 3 meshes to work with in HiFi one version to be as low as 3k. Presumably that version would be the one used at some distance from the viewing avatar so the details don’t need to be as fine and somehow HiFi will play with all 3 LOD’s depending on how close someone is to the viewed mesh?

One thing to mention which I have found in experimenting in Zbrush is that when subdividing it is always the current poly level divided by 4 - so having a 3k low LOD figure, means subdividing again to get 12k [fine], but then another subdivision will make roughly 50k polys if dividing up with the same mesh…

I guess I can use my newly retopologised 6.5k humanoid, and divide that once to get the high LOD at about 26k - I just worry that somewhere I’ll end up in trouble [mismatching verts or something] if all three LOD versions aren’t derived from exactly the same 3k basemesh geo-wise… I might well be way off the mark here with my limited knowledge but it doesn’t matter - I will have a walking talking custom avatar soon I can feel it!

:slight_smile:


#10

Heya @Arch.Stanton:

Do note that as we are still in alpha, optimization is still somewhat of secondary target, but what I am proposing in the OP would be a standard aim for when HiFi is released.

So for now feel free to experiment, we need alot more information on what can and cant be done, try not to stress about the goals yet!

LOD models require quite a lot of work to get right, but a Mid LOD target (or even just a high one) is enough for now… Unless that is you are volunteering to create free content for the good of the community :slight_smile:

From what I have gathered the LOD levels are there for the client to try to lessen the performance hit for content. So if the client cant hit a certain frames per second, it will try to lower quality of the content that is the most detailed. If it cant find a lower LOD, it will either try to render a cardboard (for avatars) or simply not render the object at all!

I still suggest just trying out stuff before working on any final products There are still alot of stuff to work on that are missing for this standard to be of any use (customization scripts, features to enable them, etc). But i found that working on a standard concept now would be beneficial.

I also tend to have quite a stringent budget for polygons in modern standards, but thats because I try to have the foresight for potential amount of users “nearby” and due to bandwidth restrictions on host end.

I hope that helps!


#11

Thanks for the input! This gives us a good sense of your priorities.

Some questions and comments.

Re: Requirements of the Standard from High Fidelity

  • this all sounds good and is something we’ll need to discuss and prioritize with the engineering team.

Re: Standard Skeleton

  • can you say a bit more about where the current hierarchy is falling short?

Re: Custom Bones and Naming Conventions

  • This all sounds good and is something we’ll want to validate with the engineering team.

Re: Avatar Customization Blendshapes:

  • Sounds good. We’ll need to discuss with the engineering team.

Other:

  • I’ve made a note to integrate the lexicon and some of the general concepts in your post to our documentation. By the way, you can find that here:

http://docs.highfidelity.com/v1.0/docs/high-fidelity-avatar-standards

Let me know if you see any problems or have other additions you’d like to make.

-Ozan


#12

Heya @ozan,

The current skeleton is fine but as I discussed in the OP, it is lacking potential customization options regarding the head of an avatar.

if and when manipulation of translation and scale of bones is supported for avatars, and ability to set blendshapes via script, it would allow end users without any prior knowledge of 3D to modify their avatar shape and limb size to the limit of what the creator of the avatar has allowed would open up myriad of shapes and scripts would support.

I added extra bones for the eyes so that if there come a case where the eyes are scaled up or offset (by some degrees) you would also affect the eyes surrounding area on the main mesh (literally the eye sockets). This way the eyes wouldn’t pop out of the head, as usually they are the only things bound to the eye bones. These however should never be animated.

The additional spine bones and neck bone are only optional, but would allow for more curviture when someone is looking left or right or head up or down. Right now it feels like assigning lean on spine2 causes the entire body to rotate too much, and when looking with the head, it becomes an angle instead of a curve

I hope that explains the shortcommings


#13

Hi @Menithal,

Gotcha’. Sounds good.
So to restate your comments, a lot of what you’re describing is about support for skeletal customizations. Right?

From what I can see, that customization support would work with either skeletal hierarchy. Unless there’s a specific reason to modify the existing base hierarchy standard, I’d suggest that we add your new customization requests to it and move forward from there.

Sound good?

Ozan


#14

Yes,

In short, The EyeSockets only purpose is to allow for scaling/of and allow modification of the placement of the eyes them selves (instead of having to combine myriad of customization blend-shapes and eye offsets, considering that not all avatars are even human, and with human heads eye positions can vary as well (no one is perfectly symmetrical) ). Otherwise they are not necessary at all. From some tests the eyes behave normally even if there is a bone before them that is not the head…

Although, I’d rather suggest that all these customization options be able to be done through the JS scripts instead of the interface viewer, so there would be script API to do these modifications on. This is why I had the requirements include item #2, #4, #5.

In anycase, sounds good.


#15

This looks really good. Especially in such an early state.


#16

Bumping this post up. Updated some information on the original Post, and also to give it some more visibility again after nearly a year of it being out here on the Forums.


#17

Announcing a full (Human) Avatar skeleton available here, with all the rotations set properly to a degree (Updates pending):

Has the Above standard, Blendshapes and Rules of thumbs when using it as a base skeleton:

Rules of Thumb(s) for best cross compatability:

  • As much as possible, avoid rotating any of the bones
  • Only scale and translate bones along their normal axies to fit your model.
  • Finger1 are actually the tendons connected to the fingers. They shouldn’t bend as much as Fingers2-4

This Skeleton and Standard is Released is under Public Domain.

Can do the above because every bone is made from scratch and positioned by eye.

Later will create a script to create all the rigs for animations, but that will be provided via CC attribution.


#18


working well :slight_smile:


#19

Just a quick wonder for improvement. When you close your fists, do the fingers cross each other?


#20

Its just an auto weighted quick try, I had to scale parts up and down to fit the avatar, so its first fix but not far away