Hey there fellow High Fidelity avatars,
I am looking to find ways to enhance the overall cognition of NPCs (bots/scripts/pets/companions etc.) in High Fidelity… I would love to collaborate with a javascript programmer to import some preexisting libraries written in other languages so that they are compatible for use in High Fidelity. I would also like to have fun making humanoid bots and also some weird alien NPCs that can think deeply about their virtual environment (e.g. other avatars, NPCs, objects etc.) rather than simply respond to it in a predictable manner.
The following text is what I wrote to Liv and Ozan when they asked me to write a one-page summary of what I want to do in HF…
Current High-Fidelity NPC Situation: The High Fidelity NPCs and most NPCs in other social virtual worlds and video-games are aesthetically very interesting but cognitively, are just model-based reflex agents (finite-state machines). They simply react to environmental input with a (usually) predictably limited choice of behavioral responses. There is little sense that they are thinking deeply about the virtual environment, avatars and agents that they are interacting with. They simply react to perceptual triggers that transition them into a particular state (e.g. idle or chat-mode), and will always perform one of a limited amount if possible actions while in this state (until the next trigger). Sometimes, one has to intentionally interact with an NPC in order to get a response at all. For rudimentary applications, this is sufficient but not for more meaningful interactions.
Proposed enhancements: I wish to enhance the intelligence of these NPCs with cognitive architectures. There exist some code libraries available online for use and further development etc. The architecture I am most interested is CLARION.
https://sites.google.com/site/clarioncognitivearchitecture/
Cognitive architectures such as CLARION allow NPCs to think more deeply about whether or not to perform an action. These deeper thoughts can be designed around pre-defined goals, drives and motivations. The NPC can also monitor its own behavior. If designed properly, the NPCs will be seen as more deliberative and less reactive with an automated psychology closer to that of a human being.
Technical challenges: While available for use with flexible licences, many of these cognitive architecture libraries were coded in languages other than Javascript. The biggest challenge would be to convert these libraries into Javascript (or some compatible approximation) so that they can work with High Fidelity’s in-house scripting environment (with additional access to the Limitless integration). A personal challenge is that while I have previously hired programmers to help me design new NPC minds, I do not consider myself to be a programmer. I can occasionally read and modify code but I am not at a programming level where I can write code from scratch. I am able, however, to design the NPC’s mind at a higher-level from which the coder can code from. I am very familiar with cognitive architectures like CLARION and would know how to use these code libraries to enhance the cognitive functionality of a particular High Fidelity NPC.
Currently available resources: For greeter/guide NPCs, I can select from a variety of marketplace avatars. For my own NPCs, I can design the NPC bodies using either High Fidelity’s in-world tools or import from mesh (I am used to modeling in Second Life)…Limitless also already offers the capability to retrieve assets via a voice-command prompt. This would be very useful for providing voice-based perceptual input for the cognitively-enhanced NPC to think deeply about. Of course, High Fidelity gives me my own sandbox space to create, store and test these NPCs. There is also a great community of High Fidelity content-creators and programmers I also enjoy socializing with.
Proposed Test Cases for High Fidelity: To learn how to use Cognitive Architectures in High Fidelity, I would help design and test a greeter or guide companion with the aim of designing my own NPCs as I become more proficient in this kind of design process. This design process would include the development of goals, motivations, drives, intentions, salient perceptual information as well as ways in which the NPC can monitor its own mind for improvement and/or optimization. The architecture I wish to try out first would be CLARION. With CLARION in particular, the NPC can also gradually convert trial-and-error-knowledge into logical rules. I could also help specify the kinds of rule-learning possible within the High Fidelity environment.
Required resources: I would require a fairly dedicated Javascript programmer (either as a volunteer or for hire) to help me convert these cognitive architecture libraries for use in High Fidelity and also for quickly showing me how these coding libraries work in the new Javascript environment. I would hope that these capabilities would also be compatible with the Limitless voice-prompt integration.
Motivation: I am more interested in working with a programmer than with any financial reimbursement for me (if funding is an issue). For me, this is both a hobby and part of my ongoing academic research from which, some publications are likely to appear. In the meantime, I would be very happy to learn how to program High Fidelity NPCs the usual way (as finite-state machines). I will attend every University session I can in High Fidelity for this purpose.
