Bulk ATP asset dump


#1

I need a way to quickly add over 7,000 animation files to my ATP.

Any idea how I can do that without uploading one at a time?

If you explain the hex and hash algorithm, I might be able to generate the filenames that map.json requires. Thanks in advance.

@Frizzlefry


#2

Hi @AlphaVersionD,

The asset server hashes each file using Sha256.
The “map.json” file stores a mapping from the file paths to the file hash:
"/path/to/my/file.fbx": "sha256-hash-of-the-file"
and the “files” contains a copy of each file where the name of each file is its hash (no extension)

Do you know how to write a bash script? this could be used to quickly generate all of that.
I can give you a hand if you need.


#3

Yeah! That would be great! Thanks @c.


#4

@AlphaVersionD can we chat in world so I can get a better idea of what you need?


#5

Ok, I had some time so I just wrote a small script:

It’s pretty straightforward, you just need to have the script with an input and an output directory next to it:

+--hasher/
|  +--hasher.sh
|  +--input/
|  +--output/

Put your animations in the input folder with the desired structure:

+--hasher/
|  +--hasher.sh
|  +--input/
|  |  +--animations/
|  |  |  +--walk/
|  |  |  |  +--walk-1.fbx
|  |  |  |  +--walk-2.fbx
|  |  |  +--run/
|  |  |  |  +--run-1.fbx
|  |  |  |  +--run-2.fbx
|  +--output/

And then run the script.
It’ll generate this:

+--hasher/
|  +--hasher.sh
|  +--input/
|  |  +--animations/
|  |  |  +--walk/
|  |  |  |  +--walk-1.fbx
|  |  |  |  +--walk-2.fbx
|  |  |  +--run/
|  |  |  |  +--run-1.fbx
|  |  |  |  +--run-2.fbx
|  +--output/
|  |  +--files/
|  |  |  +--<some-hash-1>
|  |  |  +--<some-hash-2>
|  |  |  +--<some-hash-3>
|  |  |  +--...
|  |  +--mapping.txt 

Now you just need to port what’s in output to your asset server.
First, make sure to backup your server!!!
Then, make sure your server is shut down.
Add all the files in output/files to your asset server’s assignment-client/assets/files/ directory.
and copy the content of output/mapping.txt into assignment-client/assets/map.json.
This can be a bit tricky to do without messing up the file if you’re not familiar with JSON, hence why the backup is important.
mapping.txt will look something like that:

"/path/to/file-1.fbx": "<some-hash-1>",
"/path/to/file-2.fbx": "<some-hash-2>",
"/path/to/file-3.fbx": "<some-hash-3>",

and map.json will look something like that:

{
    "/path/to/file-a.fbx": "<some-hash-a>",
    "/path/to/file-b.fbx": "<some-hash-b>",
    "/path/to/file-c.fbx": "<some-hash-c>"
}

Note the last line has no comma at the end.

Merge the 2 files in order to get this in map.json:

{
    "/path/to/file-a.fbx": "<some-hash-a>",
    "/path/to/file-b.fbx": "<some-hash-b>",
    "/path/to/file-c.fbx": "<some-hash-c>",
"/path/to/file-1.fbx": "<some-hash-1>",
"/path/to/file-2.fbx": "<some-hash-2>",
"/path/to/file-3.fbx": "<some-hash-3>"
}

The indentation doesn’t matter.
Note that I have added a comma at the end of "/path/to/file-c.fbx": "<some-hash-c>"
And that I have removed the comma at the end of "/path/to/file-3.fbx": "<some-hash-3>",
Every line needs a comma except the last one.

Now you are done, restart you server and make sure everything shows up in your Asset Browser.
Ping me if you hit a bump.


#6

Way to go above and beyond! Sorry I couldn’t meet up yesterday. Time zones. But yes, this looks great THANKS!

I’ll need to perform a few simple file renames to match your convention so that I can leverage the directory builder within; nice touch! This could take a day or two for me to finalize, but I’ll definitely report back here with results.


#7

No worries, yeah, definitely let me know.

FYI, native bulk upload, move, etc is on the roadmap but I have no idea when we’ll get to it, so hopefully this solves the issues for extreme cases like that in the meantime.