Next Level Character Animation

#1

TL;DR

~ Novel character animation system discovered and investigated
~ Government innovation promotion secured to reduce costs and facilitate legalities
~ Company formed to develop usable version of animation system
~ Extensive hardware for developing animation system implementation put in place
~ Collaborative opportunities offered
~ Current task: Animation software implementation well underway but not yet complete

Introduction

Where to start, so many ideas! From the beginning:

Back in April 2017, a researcher called Dr Daniel Holden presented a paper at SIGGRAPH describing the ‘Phase-Functioned Neural Network for Character Control’ or PFNN for short. The technique uses a specialised neural network to animate a character and requires very little memory, is fast to compute at runtime, and generates high quality motion in many complex situations[1]. I was highly impressed with the results:

In fact I was so impressed that I decided to commit to developing the system further to make it accessible to platforms like High Fidelity and Unity. Once up and running, I plan to train new neural networks with data I have captured myself and sell said trained neural networks.
In retrospect, I may have under-estimated how difficult reverse engineering someone’s PhD project would be! Although I feel I’m getting close, I’m not there yet. I forked a fork of the PFNN demo adapted for Unity and managed to get some rotations on a simple FBX character. This is a bit of a cheat, as I used the global rotations coming directly out of the PFNN which is not really the correct way to do it. Nevertheless, it works. Code here: https://github.com/DaveDubUK/PFNN-Demo

So how does this relate to character animation in High Fidelity? After all, in the case of an HMD interface, the hand and head positions are driven by the HMD and hand controllers. Movement over any significant distance is via teleport. So why do we need a character animation system for avatars at all? Here’s some thoughts I have been mulling over since my early HiFi animation projects back in 2015:

  1. First, not all users are wearing an HMD. Taking on board Philip’s comments in his recent blog post[2], this situation is unlikely to change in the near future. In which case, the current HiFi model is still relevant. By replacing this model with a PFNN, the complexity of animation graphs would be removed and the processing / memory overheads for producing compelling animation decimated. Not to mention the hugely increased realism!

  2. NPCs. The PFNN is absolutely ideal for driving NPCs.

  3. Even in the case of a user interfacing via an HMD, there is still a role for a neural network driven animation system to ‘fill the gaps’. For an example, check out Philip and Peter Diamandis locomotion as they enter the stage at the beginning of this virtual talk:

There are also some other very exciting developments in the neural network space (again from Daniel Holden) that may provide even more elegant solutions in the future, but I mustn’t get ahead of myself! See the movie below (imagine replacing the mocap data with HMD and controller data and extend):

Of course, the above idea is a way off yet, but I think it demonstrates neural networks form the future for character animation.

The formation of a plan

In order to bring these sorts of techniques to market, I identified what I’d need to put in place:

  1. Somewhere to work where the cost of living is low, especially during the R&D phase of the project
  2. A motion capture studio
  3. A lot more knowledge of neural networks in general, Python (including the NN libraries like Numpy, Theano, Scipy etc) and 3D maths (always a struggle!)

The first point is already covered; I live on a tiny tropical island in the Gulf of Thailand, literally in the jungle. Even though we have high speed Internet, supermarkets, a hospital, post office etc, everyday living costs like food, electricity, Internet and rent are still pretty cheap. The island has a fast growing IT professional community that’s keen to promote the island as a ‘Silicon Paradise’.

However, I also needed to have a visa and work permit. The best option here was to get approval for my project from the Thai Government’s Board of Investment, as this would give me a number of privileges; the ability to apply for working visas for visiting professionals, no corporate tax for three years, refund of import duties for any equipment I import and a ‘Smart Visa’ that allows me to live and work unhindered for two years plus quite a lot more.

For the second item, I visited Qualisys at their main office in Göteborg, Sweden back in April 2018. In the past, they have been mainly focused on medical applications of motion capture, but are currently expanding into virtual reality and other entertainment applications. We had a lot to talk about! I decided to go with them and signed a great deal on a 50-camera motion capture system that would enable capture of up to six actors simultaneously, enable live performance streaming and of course the generation of very high quality animation data.

The third item is still ongoing. I do love a challenge and have made progress, but I’m not there yet! There have been some amusing results from my attempts to re-train a PFNN for a HiFi avatar’s armature though, so I may as well share one here, give you all a bit of a laugh:

Enacting the plan

I rented a small office space and kitted it out with aircon, electricity stabilisers, UPS etc and furnished it. I then applied to the Board of Investment and National Innovation Agency and was recently granted my Smart Visa (this entire process took twenty months).

I imported my Qualisys 50-camera mocap system a couple of months back and have set up a test studio in my office:

I have come to an arrangement with a local dance studio owner whereby I get to rent the space as and when needed. This works very well economically, as motion capture studios tend to have a lot of downtime:

I am currently focused almost entirely on getting the PFNN to play nicely with FBX character models…

What’s next?

Using the output of the original PFNN to drive an arbitrary 3D character in Unity and HiFi is the first step in getting PFNN solutions out there and usable. After that, training a PFNN using the original animation data set re-targeted to a HiFi character is the next goal. The initial research phase will be complete when a PFNN is trained using data I have produced using my new system.

Eventually, I’d like to provide a system that allows users to specify percentages of characteristics (e.g. male, female, shy, confident, happy, solemn, sprightly, stoic, light footed, heavy footed, clumsy, elegant, tired, lively) before automatically generating their own, personalised PFNN. This would involve mixing different animation sequences from different actors as input when training a PFNN. As in RL, this would give avatars their own unique gait and character, thus incentivising people to protect their uniqueness by taking reasonable steps towards preventing others copying their personal animation neural network.

I have previously collaborated with a couple of HiFi users (thanks again to @humbletim and @fluffyjenkins for previous input!) and would like to do so more:

I’d like to work with a high-end avatar creator with the goal of creating PFNNs specific to the avatar’s armature. There may be a market for bespoke avatars, complete with their own, unique animation neural network. Previous experience in SL has shown me that there are people out there willing to spend a LOT of money on high quality digital representations of themselves.

I’m open to the idea of working visits - if anyone has a suitable project they want to work on, we can talk about my offering office space and access to equipment (50-camera mocap studio, 2 x Vive Pro (one wireless), Emotiv EEG). With my BOI privileges, I am in a position to help out with visas etc. I’m not looking to make money here, I am looking to help promote the creation of high-end digital goods. Given that future development of High Fidelity could possibly fall to the open source community, I’d be happy to make the same offer to people working on the core code of High Fidelity. However, spaces are limited.

If anyone has a interesting project that needs some mocap data, I am at the moment open to offering mocap services at significantly reduced prices - maybe even cost price if the project was educational or there were extenuating circumstances. What constitutes ‘interesting’ would be decided on a project by project basis…

But the very first step will be to capture some regular animation data for a female avatar and put it out there for free to HiFi users very shortly. This gives me practice using my new mocap system and promotes my new company, so it’s worth my while…

Footnote

Sadly, my plans are coming to fruition just as Philip published his recent blog post[2]. Whilst this is a disappointing development, I am not put off - I have a hope that HiFi’s development will be continued by the open source community (I envisage perhaps in a different, cut down form that focuses on high-end graphics, networking and easy importing of scenes from platforms like Unity and doesn’t concern itself with in-world build tools, marketplace, currency, avatar ids etc - more akin an html, browser and server standard for a 3D Internet and less like SL - but this an entirely separate conversation).

Neural Network based animation solutions would be suitable for porting to platforms like Unity and Unreal, so this presents a viable alternative. However, my real passion is for animating avatars in virtual worlds.

And finally, there may be a space for neural network driven animation systems in the collaborative work project Philip outlined in his blog post.

Summary

Well, that’s the big plan, halfway enacted. The boring practicalities are out of the way, the infrastructure is in place; only the fun tech stuff remains!

Any thoughts, comments, suggestions or criticisms welcome…

[1] Phase-Functioned Neural Networks for Character Control. Dr Daniel Holden. http://theorangeduck.com/page/phase-functioned-neural-networks-character-control
[2] Toward A Digital World. Philip Rosedale. https://www.highfidelity.com/blog/toward-a-digital-world

13 Likes
#2

Looks amazing.
I have a few questions
Have you had it working in hifi?
Can it handle hand controllers ? ie can i walk and wave my arms around.
Can or could it intergrate with other forms of animations. Can i sit on a chair?
Most importantly did you try out the dance pole? :slight_smile:

#3

I’ve recompiled the original project successfully and have also sucessfully projected the (global) rotations onto an FBX character in Unity, but I’ve not managed to fully implement the PFNN (i.e. with local rotations ) onto an FBX character yet. I’m close, so close!

@humbletim, @FluffyJenkins and I had a crack at getting something up and running in HiFi a few months back, but had no luck - in retrospect at least partially due the weird, non-standard armature that the original project’s character uses.

The handling of hand controllers would be outside my control, as is the ability to sit on a chair etc. The necessary functionality would need to be provided by the HiFi js API. I’ve not looked at the HiFi js API in detail for a while now, so I don’t know if we have the necessary fine-grained control and ability to blend the various animation sources (e.g. HMD / hand controller positions and rotations, PFNN, regular animation playback, IK, physics, streamed motion capture data, code driven procedural animation, etc).
I’m currently doing all my experimentation in Unity, as prototyping there is quicker and easier due to the maturity and usability of the platform.

I’ve not tried the dance pole, no; am far too old for that sort of malarky! But the lovely Clementina who owns the dance studio is a qualified gymnastic pole dance teacher, is awesome to watch perform and has agreed to be my first properly mocapped performer - so the set of promotional female animations (idle, walk, run, turn, fly, etc) I mention above will be of her. :slight_smile:

1 Like