Trouble loading JSON data


#1

It’s been a while since I played around with HiFi. Quite a lot has changed and I’m playing catchup.

I want to load a large number of (locally stored) JSON data into my script but seem to have hit a wall. I must be missing something obvious. I’ve recreated my issue on a very small scale. Here’s a description of the methods I’ve tried so far:

JSON test data:

[ 1, 2, 3, 4, 5 ]

File structure:

Method 1: Using XMLHttpRequest (synchronous method)
This is the method I used to use a couple of years ago, but I can’t get it working now. The path has been tried with a preceding slash, a preceding dot slash and no preceding symbols. Test script:

var request = new XMLHttpRequest();
request.open("GET", "./testData.json", false); // false set for synchronous
request.send();
if (request.status == 200) {
    try {
        var file = JSON.parse(request.responseText);
        print("Got data: " + JSON.stringify(file, null, " "));
    } catch (e) {
        print('Error parsing JSON data: ' + e.toString());
    }
} else {
    print("Error " + request.status + " encountered whilst loading JSON file");
}

No joy:

[10/8 9:29:48] [testScript.js] INFO - Script Engine starting:testScript.js
[10/8 9:29:48] [testScript.js] Error 0 encountered whilst loading JSON file

Method 2: Using XMLHttpRequest (asynchronous method)
So I thought to try loading asynchronously instead. Test script:

var request = new XMLHttpRequest();
request.onreadystatechange = function() {

    print("\nonreadystatechange: readyState is " + request.readyState + 
    	  "\nonreadystatechange: status is " + request.status +
    	  "\nonreadystatechange: responseText is " + request.responseText);

    if (request.readyState == 4) {
    	if (request.status == 200) {
	        try {
	            var file = JSON.parse(request.responseText);
	            print("Got data: " + JSON.stringify(file, null, " "));
	        } catch (e) {
	            print('Error parsing JSON data: ' + e.toString());
	        }
	    }
    }
};  
request.onload = function() {

    print("\nonload: readyState is " + request.readyState + 
    	  "\nonload: status is " + request.status +
    	  "\nonload: responseText is " + request.responseText);

    if (request.readyState == 4) {
    	if (request.status == 200) {
	        try {
	            var file = JSON.parse(request.responseText);
	            print("Got data: " + JSON.stringify(file, null, " "));
	        } catch (e) {
	            print('Error parsing JSON data: ' + e.toString());
	        }
	    }
    }
}      
request.open("GET", "./testData.json", true); // true set for asynchronous
request.send();

No joy:

[10/8 9:3:7] [testScript.js] INFO - Script Engine starting:testScript.js
[10/8 9:3:7] [testScript.js] 
onreadystatechange: readyState is 1
onreadystatechange: status is 0
onreadystatechange: responseText is 
[10/8 9:3:7] [testScript.js] 
onreadystatechange: readyState is 4
onreadystatechange: status is 0
onreadystatechange: responseText is 

Method 3: Using ATP
I’ve not used the ATP server before. I only need the data on my local domain, so ATP would be a viable solution. I put the file on the ATP server:

Then try to load it into my script like this:

// Method 3: Using ATP
var path = "atp:/testData.json";
print("Attempting to obtain JSON file from ATP server at " + path);
Assets.downloadData(path, function (data) {
    print("Asset server data downloaded:" + data);
});

No data, no joy:

[10/8 9:46:51] [testScript.js] INFO - Script Engine starting:testScript.js
[10/8 9:46:51] [testScript.js] Attempting to obtain JSON file from ATP server at atp:/testData.json

I then tried using the SHA256 hash copied from the log:

[11/08 09:41:23] [DEBUG] [hifi.networking.asset_client] Attempting to upload "d:/Projects/HiFi/animation/test/testData.json" to asset-server.
[11/08 09:41:23] [DEBUG] [hifi.networking.asset_client] Successfully uploaded asset to asset-server - SHA256 hash is  "10165a66ac7396137a437e51d4892b0ca8f7534bae4a493eddd7ac2ffca8655a"

Like this:

var path = "atp:10165a66ac7396137a437e51d4892b0ca8f7534bae4a493eddd7ac2ffca8655a";
print("Attempting to obtain JSON file from ATP server at " + path);
Assets.downloadData(path, function (data) {
    print("Asset server data downloaded:" + data);
});

Finally!

[10/8 11:42:12] [testScript.js] INFO - Script Engine starting:testScript.js
[10/8 11:42:12] [testScript.js] Attempting to obtain JSON file from ATP server at atp:10165a66ac7396137a437e51d4892b0ca8f7534bae4a493eddd7ac2ffca8655a
[10/8 11:42:12] [testScript.js] Asset server data downloaded:[ 1, 2, 3, 4, 5 ]

However, I will need to load many JSON files, so manually uploading each to the ATP server and copying the hash from the log files for each will not be practical.

Method 4: Using script.include
As a last resort, I tried converting the data to a js var in a separate file and using Script.include (a bit hacky)

var testData = [ 1, 2, 3, 4, 5 ];

I then try to load the data from another script like this. I tried various relative path syntax (preceding ./ and /) and an absolute path for the include file.

Script.include("testData.js");
print("PFNN: Weights and biases loaded: " + testData.toString());

This one really confuses me - no joy:

[10/8 11:47:24] [testScript.js] INFO - Script Engine stopping:testScript.js
[10/8 11:47:24] [testScript.js] INFO - Script Engine starting:testScript.js
[10/8 11:47:24] [testScript.js] ERROR - [UncaughtException evaluate] Error: Can't find variable: testData in file:///d:/Projects/HiFi/animation/test/testScript.js:81
[Backtrace]
    <global>() at file:///d:/Projects/HiFi/animation/test/testScript.js:81

I expect I’m missing something quite obvious here. Any help would be greatly appreciated.

  • davedub

#2

Have you tried var json = Script.require(Script.resolvePath("filename.json")) ?
You can also use var json = Script.require(url);

For the others, you are referring to a local file on the filesystem, you may need to have Script.resolvePath

This is what I have been using for a while now, specifically since I made the clap script, which engine you can find here Script.require automatically converts the file into a Json type if json, and if js, it expects for a module.exports with the object.

Just note that there is a specific bug regarding joint rotations of the hands being overriden by the base line avatar, so it may cause some issues, but they are working on a fix for it.


#3

Script.resolvePath works for method 4. Thank you @Menithal

I’m still having trouble with the ATP method - but since I now have a working way to get the data in, I’ll stick with method 4 for now.

I’ve noticed the hands override bug, good to hear it’s being fixed.


#4

just note that Script.require does have some -aggressive- caching.


#5

@Menithal - I’m not sure what you mean by “aggressive caching” - would that be anything to do with this issue I discovered shortly after loading my files?

https://github.com/highfidelity/hifi/issues/11769 (Memory does not appear to be released on script reload)


#6

As in, once its loaded its shouldn’t be reloaded / doesnt update unless you cache bust it or, use one of the new flags I forget the name of which disables the caching…

Im guessing thats causing the memory leak as well, mainly it reloading the file into memory again…