Yattaaa! [fixed] Glitchy leg rotations

#1

I’ve been struggling with my bvh importer for HiFi for ages now, as I couldn’t get the rotation order stuff quite right. Having strained my brain for days, I’ve finally got it all sorted out and produced a set of new harmonic animations that required no editing after conversion :slight_smile:

Open and Run from URL: https://s3-us-west-2.amazonaws.com/davedub/high-fidelity/walkBeta/walk.js

To check the fidelity, I converted some of the exact same animations used by the new rig animation stuff @howard.stearns has been working on so I could compare (walk, idle, walk backwards, side stepping). I also added a mocapped flying animation, just for fun. It’s interesting to compare the new FBX animations (Interface menu -> Developer -> Enable Rig Animations) and the new walk.js ones.

Just to recap, the animations I’ve produced are 100% procedural and contain the harmonics necessary reproduce the original keyframe data curves. I’ve basically adapted some audio engineering mathematics and applied it to character animation.

And it all works, beautifully!

Working beta here (can open and run from URL):
https://s3-us-west-2.amazonaws.com/davedub/high-fidelity/walkBeta/walk.js

Problem: always hovering / flying?

If you have any trouble with always hovering (i.e. not detecting the ground correctly), please try reloading the script whilst standing on a surface.

Problem: walking slowly?

I've recently seen the situation where the avatar won't move any faster than 0.4 m/s. The avatar speed is not something that is controlled by walk.js, and the cause is currently unknown. To work around, I've moved the avatar well above the surface, then let him drop back down onto it before reloading the script.

Dancing

I’ve also converted a couple of dances for experimentation, as the timing / synchronisation benefits of harmonic / procedural animation look set to enable dancing perfectly in time to the music.

Technical

- Production pipeline: By producing a bvh file without any pre-rotations and then applying to avatars using the new bind pose, all animations pretty much fit all avatars automatically - awesome! (This is provided the avatars have the correct joint orientations and has only really worked properly since the recent introduction of the new binding t-pose.)

  • The pipeline is: fbx -> bvh -> Murayama JS bvh converter -> walkTools JS harmonics converter.

  • The next big step is implementing IK in JS so I can start sticking the feet to the ground, adjusting for uneven terrain and at some point enable avatar to avatar interactions.

As always, any feedback, comments, insights or observations are most welcome.

  • davedub
#2

Did i just understood correctly is the fact that these are indeed caused by the pre/post rotations that I guessed on earlier? Also what did you mean by new binding T-pose, ?

Will have to give this a shot later tonight

#3

@Menithal

Pre / post rotations (and rotation orders)

The pre-rotations in the fbx animation files had to be removed before conversion, as they pertained to the animation file's skeleton, and not the final avatar's. So, by exporting bvh files without pre-rotations, I got some very strange looking bvh files that work great when applied to an avi in HiFi - because the HiFi avatar has it's own pre-rotations, already applied. Funny how simple it all seems in retrospect!

Example bvh file without pre-rotations here:
https://s3-us-west-2.amazonaws.com/davedub/high-fidelity/bvh/standard_walk_no_pre_rotations.bvh

I didn’t find any animation or avatar fbx files that had post-rotations. My main mistake was thinking that the Euler rotations in a bvh file could be converted to pitch/yaw/roll angles by simply adding vectors in the correct order. Once I employed a couple of THREE.js objects to do a proper conversion via quaternions, I got my ‘yattaa’ moment!

T-Pose

@howard.stearns mentioned the T-Pose converted animations in this PR: https://github.com/highfidelity/hifi/pull/5667 I'd actually had the same idea, and was in the process of creating a re-targetting skeleton that also went to a T-Pose when it's rotations (but not pre-rotations) were zeroed when I read the PR notes - saved me a job! I don't know if this affects the requirements for the bind pose from a rigging point of view though - I'll leave that for you to work out!

walkTools

In case you or anyone else is interested, I've updated the public walkTools beta with the new bvh player and animations:

https://s3-us-west-2.amazonaws.com/davedub/high-fidelity/walkTools/walk.js

(click the walk.js settings tab, then click the ‘walkTools’ button to access all the tools)

I’ve not included the actual converter, as it is still a monolithic mess with dsp.js, a bvh parser and a sprinkling of THREE.js objects just pasted in. It also needs specific variables set each time it’s used, so I’m planning to get it more usable before I release it and start having to support it. The bvh player (walkToolsBVHPlayer) works fine though.

  • davedub
#4

Ok, found a weird problem. the old script have my feet normal

The new script that also load pretty slow have my feet in a weird angle, also with walking. This seems to happen when the male character is selected

#5

@Richardus.Raymaker - Thank you very much for the feedback.

Foot positions

I'd like to investigate this further - which avatar are you using there? If it's one that is not on the marketplace, can u send me a link so I can test?

Loading times

Regarding the loading times, yes, I have to agree. There are a number of optimisations I can and will put in before I get to the PR stage that should significantly reduce the loading time.
  • davedub
#6

It where with my own made avatar. But just tested. the Zach avatar seems todo the same with the feet.

Another hint, it would be nice to keep the foot step sound turned off by default, because many mabye don’t know how to tun it off. andit’s very annoying with voice and meetings. Until the day we can block or change the sound volume.

#7

I tried this with my own avatar and it has gone… abit in an odd tangent (with “mixamo prerotations”).
looks

7b88c012bd5b8b0b.jpg606×797 305 KB

And without it

3cc5ac0801651545.jpg606×797 320 KB

While my actual Skeleton is oriented with:

89ef72967a1499b6.png822×865 44.2 KB

and exported with the settings I have reported in my blender thread.

In the older version the orientations offsets felt alot more sense. Will have to look into it.

I have also repeated the issues with my Tests:

Control - http://www.norteclabs.com/HF/labs/tests/skinning_test_mixamo_autorig.fst?V11 - Test avatar running on Mixamo Autorig mesh. Control “normal Avatar”
Reexported - http://www.norteclabs.com/HF/labs/tests/skinning_test_mixamo_autorig_blender_re-exportYX.fst?V1 - Test avatar running on Mixamo Autorig mesh. Rexported from Blender with defaults

5b89058d03e40592.jpg373×680 129 KB

Bones were created in mixamo and re-exported from blender, and using mixamo-prerotation applying.

Will have to check if my world rotation switch causes this that I used to avoid having to rotate the model -90 degrees in the world axis.

#8

@Menithal - I’ve been giving this some thought. The problem is almost certainly the fact that Blender fbx exports do not define pre-rotations. Blender, I think, combines the pre-rotations with the actual animation data on fbx import. This causes problems later, as there is no way to know what the original pre-rotations were for that avi once it’s been exported from Blender.

In HiFi, avatars with the correct pre-rotations defined in the fbx will always hit a t-pose when the joint angles are set to zero. i.e. the pre-rotations define a reference pose to work from when animating, and so enables fitting all animations to all avatars (providing joint orientations are correct of course

So, in order that all animations basically fit all avis, pre-rotations are essential. But Blender doesn’t preserve them!

Possible solutions:

  1. Adapt the Blender fbx importer / exporter to retain the pre-rotations. Probably a big job, but is the best solution. The pre-rotations could be merged with the animation data whilst modelling, as long as they are removed and defined separately on fbx export.
  2. Define a set of fixed pre-rotations for Blender exported avis and subtract them when Interface imports an avi. This would be a hack, and would never yeild ideal results, as the true pre-rotation values will vary from avi to avi.

This is all based on stuff I’ve worked out on my own recently - I hope it all makes sense!

Footnote: I have not checked if Blender exported avis show the exact same problems with the new Rig Animations built into Interface. Am not at home today, but will try to verify this later. I’m pretty sure will hit the same lack of pre-rotation issues though.

#9

I have been actually diving into the FBX exporter for quite some time (slowly), but been brushing up on my quaternions and matrice math: the importer has the pre-post rotation differences already included as with mixamo avatars imported into blender without pre/post behave exactly the same as blender avatars on export.

For now I have been absolutely stuck on the math: what Id like to know how I can build the pre-rotations and post rotations out of quaternions (Blenders actual native rotation format). That way I could make a fix for Blender FBX Export and solve this darn issue from Blender’s side once and for all. I am guessing it is actually just the Pre rotation, rotation and post rotation are built from the columns of the 3x3 Rotation Matrix: I havent had the time to try it out though.

#10

Just to confirm I understand what you’re saying, you’ve been working on separating the pre-rotations from the animation data on export in Blender? If so, awesome, this is just what’s needed!

I can only offer what I hope is helpful information:

  1. When viewed in MB, Mixamo the pre-rotations are listed as having XYZ rotation order, so they are most probably Euler rotations and so need to be represented as such in any calculations. The actual rotation order should be present in the source fbx also…

  2. When I converted Euler rotations to pitch/yaw/roll angles, I did so via the THREE.js Euler and Quaternion objects. Perhaps there is a Blender / python equivalent to THREE.js’s super useful Euler object? Here’s what worked for me using HiFi JS + THREE.js:

    // convert Euler rotations to angles using THREE.js objects
    function eulerRotationsToAngles(rotations, rotationOrder) {
        var euler = new THREE.Euler(
            filter.degToRad(rotations.x),
            filter.degToRad(rotations.y),
            filter.degToRad(rotations.z), 
            rotationOrder);
        var quaternion = new THREE.Quaternion();
        quaternion.setFromEuler(euler);
        var quat = {
            x: quaternion.x,
            y: quaternion.y,
            z: quaternion.z,
            w: quaternion.w
        };
        return Quat.safeEulerAngles(quat); 		
    }
  1. This can only work if the pre-rotations found on Blender fbx import are available to you at export time. It seems to me such a pity that Blender combines the pre-rotations and animation data in the first place…

I hope this helps - but I too am brushing up on my 3D math - university was many years ago!

  • davedub
#11

Blender exported avis

I'm hoping @Menithal will succeed with the Blender fbx exporter work. If not, I'm not sure how we'll get around the Blender (lack of) pre-rotations issues. But we will get there in the end!

Mixamo avis

Did you try comparing the idle animation to the one now available via the new rig animation stuff? (Interface menu -> Developer -> Enable Rig Animations, make sure walk.js is NOT loaded) - the new idle animation does have foot positions similar to the picture you posted...

Footstep sounds

[quote="Richardus.Raymaker, post:6, topic:7101, full:true"] Another hint, it would be nice to keep the foot step sound turned off by default, because many mabye don't know how to tun it off. andit's very annoying with voice and meetings. Until the day we can block or change the sound volume. [/quote] Agreed! But Phillip is very keen on them - from past experience, if they are working ok, he will want them turned on by default if a PR is to go through. I have been turning them off in beta versions, but in doing so I run the risk of forgetting to turn them on again at PR time, so I've been leaving them on recently. I will try turning the volume down in the next PR, as they really should not be as intrusive as they are right now.
  • davedub
#12

Blender has the mathutils python library to handle all the grunt work: so basically its as simple as

<Mathutils.Quaternion var>.toEuler( <axis> )

What I was planning to do is basically (by theory that Ive read)

rotationMatrix = <Quaternion of object>.toMatrix()
alpha = rotationMatrix[0]
beta =  rotationMatrix[1]
gamma = rotationMatrix[2]

Apply the vector components from rotation matrix as euler rotations to Pre/Post/Rotation in file.

And see where that leadds me. Unfortunately the pre/post/rotations are not that well documented.

#13

May the force be with you - and enjoy your ‘yattaa!’ moment when u get it all sorted out! :smile:

  • davedub
#14

Sorry, but the Enable rig Animation seems the problem, it’s showing the wrong position for my feet.and i stopped all scripts complete.

About the sounds, if the are much softer it can work.

#15

@Richardus.Raymaker - ah - what you are seeing there is the new idle animation that does have quite splayed out feet. If, when you load the rig animations, the animation is the same as the walk.js versions, then it’s all working as expected.

The avatar you’re using may need it’s pre-rotations adjusted if the feet are more splayed than on other avis. We need two standard avatars now, one male, one female, so we can start to iron out these remaining little discrepancies.

  • davedub
#16

@davedub The problem is that the feet keep splayed out when you walk with the male in th enew walk.js. in some way it’s not switching from idle animation to walk.

Not sure about pre-rotations. , made it in mixamo that rigged it with idle animation. after that i have packed it in high fidelity so i have fst file. Here is the fst file, without the object name.

name = -
type = body+head
scale = 1
filename = -
texdir = -
joint = jointHead = HeadTop_End
joint = jointRoot = Hips
joint = jointLeftHand = LeftHand
joint = jointNeck = Neck
joint = jointRightHand = RightHand
joint = jointEyeRight = RightEye
joint = jointLean = Spine
joint = jointEyeLeft = LeftEye
freeJoint = LeftArm
freeJoint = LeftForeArm
freeJoint = RightArm
freeJoint = RightForeArm
bs = JawFwd = JawForeward = 1
bs = ChinLowerRaise = Jaw_Up = 1
bs = MouthDimple_L = Smile_Left = 0.25
bs = MouthSmile_L = Smile_Left = 1
bs = EyeBlink_L = Blink_Left = 1
bs = EyeOpen_R = EyesWide_Right = 1
bs = LipsLowerDown = LowerLipDown_Right = 0.7
bs = LipsLowerDown = LowerLipDown_Left = 0.7
bs = EyeOpen_L = EyesWide_Left = 1
bs = JawLeft = JawRotateY_Left = 0.5
bs = JawOpen = MouthOpen = 0.7
bs = BrowsD_L = BrowsDown_Left = 1
bs = Sneer = Squint_Right = 0.5
bs = Sneer = Squint_Left = 0.5
bs = Sneer = NoseScrunch_Right = 0.75
bs = Sneer = NoseScrunch_Left = 0.75
bs = BrowsU_L = BrowsUp_Left = 1
bs = JawRight = Jaw_Right = 1
bs = MouthSmile_R = Smile_Right = 1
bs = LipsLowerClose = LowerLipIn = 1
bs = EyeSquint_R = Squint_Right = 1
bs = MouthLeft = Midmouth_Left = 1
bs = EyeSquint_L = Squint_Left = 1
bs = BrowsU_C = BrowsUp_Right = 1
bs = BrowsU_C = BrowsUp_Left = 1
bs = BrowsD_R = BrowsDown_Right = 1
bs = ChinUpperRaise = UpperLipUp_Right = 0.5
bs = ChinUpperRaise = UpperLipUp_Left = 0.5
bs = LipsPucker = MouthNarrow_Right = 1
bs = LipsPucker = MouthNarrow_Left = 1
bs = LipsUpperOpen = UpperLipOut = 1
bs = MouthRight = Midmouth_Right = 1
bs = EyeBlink_R = Blink_Right = 1
bs = MouthDimple_R = Smile_Right = 0.25
bs = MouthFrown_R = Frown_Right = 1
bs = MouthFrown_L = Frown_Left = 1
bs = Puff = CheekPuff_Right = 1
bs = Puff = CheekPuff_Left = 1
bs = LipsFunnel = TongueUp = 1
bs = LipsFunnel = MouthWhistle_NarrowAdjust_Right = 0.5
bs = LipsFunnel = MouthWhistle_NarrowAdjust_Left = 0.5
bs = LipsFunnel = MouthNarrow_Right = 1
bs = LipsFunnel = MouthNarrow_Left = 1
bs = LipsFunnel = Jaw_Down = 0.36
bs = LipsFunnel = JawForeward = 0.39
bs = LipsUpperClose = UpperLipIn = 1
bs = LipsLowerOpen = LowerLipOut = 1
bs = LipsUpperUp = UpperLipUp_Right = 0.7
bs = LipsUpperUp = UpperLipUp_Left = 0.7
bs = BrowsU_R = BrowsUp_Right = 1
jointIndex = RightHandRing1 = 28
jointIndex = Head = 69
jointIndex = RightHandMiddle4 = 43
jointIndex = Hips = 0
jointIndex = Bottoms = 3
jointIndex = Shoes = 4
jointIndex = RightUpLeg = 7
jointIndex = LeftHandThumb1 = 60
jointIndex = LeftHandPinky4 = 51
jointIndex = RightArm = 21
jointIndex = RightHandPinky2 = 25
jointIndex = RightFoot = 9
jointIndex = Spine2 = 19
jointIndex = RightHandIndex1 = 32
jointIndex = RightHandPinky4 = 27
jointIndex = LeftHandMiddle1 = 64
jointIndex = RightHandIndex2 = 33
jointIndex = LeftShoulder = 44
jointIndex = Neck = 68
jointIndex = LeftForeArm = 46
jointIndex = RightLeg = 8
jointIndex = LeftHandThumb3 = 62
jointIndex = LeftHandPinky3 = 50
jointIndex = RightShoulder = 20
jointIndex = RightHandMiddle1 = 40
jointIndex = RightHandPinky3 = 26
jointIndex = RightToeBase = 10
jointIndex = LeftEye = 70
jointIndex = RightHandIndex3 = 34
jointIndex = RightToe_End = 11
jointIndex = Spine1 = 18
jointIndex = LeftToeBase = 15
jointIndex = RightHandIndex4 = 35
jointIndex = LeftHandThumb4 = 63
jointIndex = RightHandMiddle2 = 41
jointIndex = LeftHandMiddle4 = 67
jointIndex = LeftHandPinky2 = 49
jointIndex = Tops = 1
jointIndex = RightForeArm = 22
jointIndex = RightHandPinky1 = 24
jointIndex = Spine = 17
jointIndex = Eyes = 6
jointIndex = LeftHandThumb2 = 61
jointIndex = RightHandThumb3 = 38
jointIndex = LeftHandIndex4 = 59
jointIndex = RightHandThumb1 = 36
jointIndex = LeftArm = 45
jointIndex = LeftUpLeg = 12
jointIndex = Body = 5
jointIndex = RightEye = 71
jointIndex = LeftHandMiddle3 = 66
jointIndex = RightHandRing3 = 30
jointIndex = RightHandMiddle3 = 42
jointIndex = RightHandThumb2 = 37
jointIndex = LeftHandRing4 = 55
jointIndex = RightHand = 23
jointIndex = HeadTop_End = 72
jointIndex = LeftHandPinky1 = 48
jointIndex = LeftHandRing2 = 53
jointIndex = LeftHandRing3 = 54
jointIndex = LeftHandRing1 = 52
jointIndex = Eyelashes = 2
jointIndex = LeftFoot = 14
jointIndex = LeftLeg = 13
jointIndex = RightHandThumb4 = 39
jointIndex = RightHandRing4 = 31
jointIndex = LeftHand = 47
jointIndex = LeftHandMiddle2 = 65
jointIndex = LeftToe_End = 16
jointIndex = LeftHandIndex2 = 57
jointIndex = LeftHandIndex1 = 56
jointIndex = RightHandRing2 = 29
jointIndex = LeftHandIndex3 = 58