DDE webcam face tracking


#1

Windows build 2282 / OSX build 2151 includes DDE webcam face recognition.

Enable it with Developer > Avatar > Face Tracking > DDE Face Regression.

If tracking gets confused, reset it with Developer > Avatar > Face Tracking> Reset DDE Tracking, or Ctrl+’ keyboard shortcut.

Enabling it or starting up Interface with it enabled will start up a DDE process (dde.exe in Windows) and your webcam will be automatically turned on. Disabling it or shutting down Interface will stop the DDE process.


#2

I’ve just been trying it out and it’s very interesting I did have to use admin mode starting the loading interface to get it working.
additional You could use the voice as a trigger for moving the mouth when it’s too fast to track
Not too sure about the camera moving around when you’re in first person mode
Nice to see the shoulders working can’t wait for the improvements with speaking :slight_smile: Great job People


#3

Regarding having to run Interface in admin mode on Windows … Is anyone else having this problem?

I don’t have to but then my user account is a member of the Administrators group (Windows > Control Panel > User Accounts, it says “Administrator” in the text under my account’s icon).


#4

Any Mac equivalent?


#5

@SterlingWright Yes, Max OSX Interface build 2151 and later includes DDE.


#6

Thats really fun, great work :smiley:


#8

After signing out of Interface, the camera on my Mac remains on.
What to do about this?


#9

@SterlingWright Your webcam staying on sounds like the DDE executable/process might still be running. (It should be automatically terminated when you turn DDE off or exit Interface.) … I don’t know how to do it on OSX but if you can find DDE in a list of running processes and terminate it, your webcam should turn off.


#10

We discovered that one should turn off DDE in the interface/developer tools before exiting interface. But, really, the connection ought to be broken automatically upon exit.


#11

I have some issues with getting the DDE working on my system. The camera does turn on and the dde client is visible in task manager, but nothing happens on any avatar. Using a 64 bit windows with a Microsoft HD Livecam. Derp was just driver issues…

It works pretty well.

Like @SterlingWright mentioned, Could the mic input affect the jaw movement with some more priority (forexample, add to it)? as of the moment the tracking overrides the voice input effect on the jawOpen Shape. Considering the quality of some cameras might not even pick up on the mouth opening when talking.

Otherwise, this is very fun to play around with :smiley:


#12

My messing with the dde cam
Firstly it doesn’t work if your camera is borked. I started my cam in skype and it was all grey. So I spent some time getting the picture looking nice with the advanced settings a balanced looking image not too dark and not too light. I also moved a picture off the wall behind me that the light was shining off.
To get the dde camera to work at all I had to run interface as administrator. then it seemed pretty good.
I think people may want to mess around with the blendshape numbers in the fst file to make the mouth tracking more emphasized.
Also when i close interface the dde process doesn’t stop so you may want to kill that after in task manager
otherwise stop dde in interface before you log out.

What this needs is a little gui so the blendshapes in the fst can be adjusted with sliders in world so we can customize the avas to get the best out of dde


#14

Oh, and I read this about using the Root in Mac OSX, but it did not make a difference to my camera operation. https://support.apple.com/en-us/HT204012


#15

Well gee, that was easy.


#16

Honestly, I am not sure I want to go kicking around in the root of my main computer system just for this camera. There should be a way to use this without having to do that, don’t you think? I may just hold out until I have the FaceShift setup.

Really, I am beginning to develop the opinion that one really needs to have a machine that is just dedicated to HiFi, so that one can install the stack manager and open up root access and so forth without having to worry about prying eyes and fingers making their way in.


#17

Thanks very much for all the testing and feedback. Addressing the bugs …

DDE not exiting when you shut down Interface: It should. And does on some people’s systems. There’s a pull request waiting to be merged that should hopefully make this happen more reliably on everyone’s systems.

Having to run as administrator: This is only sometimes needed on Windows. But it shouldn’t be needed. @MichelleLeckrone , @Judas : What Windows version are you using? And 32- or 64-bit? Thanks!


#18

65 bit that’s one extra


#19

Hey I did some more testing today and I have some more comments>

  • As mentioned earlier, The voice and jaw connection to be tweaked.
  • If the camera loses track of the face, (or a certain percentage of the face is not detectable) the tracking should “loosen up” and revert to original model position, and not try to attempt to match to at bounds to avoid the avatar from folding (in terms of looking into a very wrong direction). If it does this, there should be a warning pop-up. This would be applied until a face is available to be loaded.

Ill test the administration issue on the weekend.


#20

I unfortunately haven’t managed to reproduce the problem of having to start Interface as an administrator on Windows in order for DDE to work.

I’ve tried installing Interface on a Windows 7.1 64-bit PC from a standard (non-administrator) user account. (I had to enter an administrator password in order to install Interface.) Interface runs from this account and starts up DDE just fine.

Is it possible that anti-virus software is preventing DDE.exe starting up?

What could prove useful is details of any Windows event log entries about DDE.exe not starting up. …
In Windows’ Event Viewer (e.g., press the Start button if you have one and type “Event Viewer” in the “Search programs and files” box), have a look in the Application and System logs.


#21

Hey @leo ,

Can you use something like http://nsis.sourceforge.net/Builtin_NSISdl_plug-in to remotely download the DDE dependency if its not there yet? Will make the downloads alot smaller. Just for the offline installs you will need to have DDE next to your installer to make it work.

Thanks in advance,

Greetings,

Thoys


#22

ooh awesome mine pretty much ran out of the box except i have to stare at the screen now - oh tyranny