Asset Transfer Protocol - This is what's coming


#1

Hi All,

We have been talking about the Asset Transfer Protocol in our alpha meetings. The Pull Request has now been submitted. Those that feel confident testing PR’s can go and check it out: https://github.com/highfidelity/hifi/pull/5694

More details below:

Summary of the features added by this PR:

  • UDT layer in networking library for reliable congestion and rate controlled sending
  • new assignment-client type “asset-server” (type 3) - disabled in all domains by default
  • AssetClient/AssetResourceRequest to download/upload ATP files from Interface
  • reworking of resource request system to seamlessly handle HTTP/file/ATP requests

Asset Transfer Protocol (ATP)

ATP is a new protocol High Fidelity clients and servers can use to send assets (meshes, sounds, images, scripts) to each other over their existing hole-punched UDP connections. It is meant to eventually replace the serving of existing HTTP/TCP assets because of the following advantages :

  • no requirement for content creator to own a separate HTTP webserver or S3 bucket to serve assets
  • fairer than TCP to other concurrent connections
  • less aggresive than TCP at backing off when loss occurs (and quicker to recover from a one-off loss event)
  • easy direct LAN serving of files to locally connected clients
  • client downstream saturation by concurrent requesting of file blocks from multiple asset-servers (future improvement)

ATP URLS are of the form atp:$HASH.$EXT. Here’s an example:

atp:89fcfdc5a74f3c78fa277ec4edf8fef367224bdc102dbe9794eb4788f537eac7.fbx

Currently, when Interface is given an ATP URL (ex: for scripts, entities, or avatars) it confirms with the asset-server in its current domain that a file with the given hash and extension exists, and then it sends a request for the asset-server to send it the entire file. The asset-server dispatches a message that contains the data of the file via a UDT connection.

For clients, much of the ATP request/response is hidden in the AssetResourceRequest and AssetClient classes and networking layer. Callers that currently use HTTP downloads should expect an almost identical interface for success and failure for ATP downloads.

The asset-server will initially not be a default assigned type in domains. This is to allow for further testing of the new system in High Fidelity domains with many remote clients before suggesting widespread adoption in place of HTTP assets.

One major shortcoming of ATP in its current state is that it cannot handle files that contain links to other files via relative paths (meshes are the biggest offender here). This is due to the fact that there is currently no ability to reference a file in ATP by name. We have some potential solutions to this that we will explore in the coming weeks.

Preliminary support for asset upload is enabled in interface via a Menu option at Developer->Assets. If the upload finishes successfully, Interface presents a dialog with a copiable ATP URL. If it fails, it shows an error dialog that explains why the upload failed. Ideally this would have been done via QML but a QML-friendly file dialog is not readily available. We will likely need to add one for global use in Interface, at which point the asset upload flow should be moved over as well.

UDT

The reliable ordered sending required by the Asset Transfer Protocol is acheived by the addition of the UDP-based data transfer (UDT) protocol. We have only very slightly modified the behaviour of the fourth official version of the UDT protocol, created by Dr. Yunhong Gu and Dr. Robert L. Grossman.

Nodelist/LimitedNodeList UDP datagrams now flow through udt::Socket instead of directly through QUdpSocket. This allows the udt::Socket to determine if it needs simply to send the packet off, or hand it off to a matching udt::Connection.

The entry point for our implemenation of the UDT protocol is udt::Connection. At any time that a node is receiving or sending data to a specific socket (address and port), they will have a single connection open to that socket. While sending, the udt::Connection will spawn a udt::SendQueue on a separate thread (this is required so the thread can sleep for the packet send period and maintain the correct send rate). It is likely in the future that it will make more sense to create one udt::Connection per request-response, as this will allow concurrent data flows between the same client/server that share the same available bandwidth.

The reciever in a UDT connection sends NAK packets to ask the sender to re-send anything that was lost. In order to reduce the amount of packet loss and therefore re-sent packets, UDT uses a number of techniques to estimate the available bandwidth and current congestion for each connection. It attempts to use as much of the link as possible while being very fair to other protocols also using the same network connection.

The bulk of the UDT magic can be found in the udt::DefaultCC class. udt::DefaultCC is a subclass of udt::CongestionControl, a base class that UDT connections can leverage (via its subclasses) to adopt using any number of congestion control styles. The default implementation udt::DefaultCC is our slightly modified version of the congestion control technique presented in Dr. Gu’s and Dr. Grossman’s UDT papers.

Currently UDT can handle the following types of packets:

  • unreliable, unordered
    • this goes direct from udt::Socket to the wire
  • reliable, unordered
    • sent via the udt::SendQueue of a udt::Connection
  • reliable, ordered
    • sent via the udt::SendQueue of a udt::Connection, re-constructed by the udt::Connection on receiver

The obvious missing combination is unreliable, but ordered. Support for this will be added shortly - the sender will send a block of data that the receiver knows it should drop as lost as soon as it loses any of the packets in the message.

WWCD

With @Atlante45 in France we truly couldn’t have completed this work without the help of Clerb.

Much of our design is driven by the simple question “What would Clerb do?” - the answer is usually just to move it.


#2

First a few questions:

  • Where and how do you set the file path.
  • Where do you store the assets ?

It need to have the option to store assets on different partition.


#3

Is there some arcane magics to having this option show? Build 3066 Windows has no such option and I do have an ATP server running on stack side.


#4

If HTTP/FTP support ultimately goes away then … all content files must be hosted on an assert server assignment client somewhere.

Doesn’t this just swap the problem of “content creators” (by this I assume you mean “domain providers”) needing an HTTP/FTP host for assets to the problem of domain providers needing either a capable domain server and fat upstream pipe, or needing an ATP host for assets?


#5

If I have understood this correctly, at present meshes that contain references to textures within the file cannot use this mechansim @chris ?


#6

My hope is ATP will be part of a suite along with HTTP etc vs the only way. As to the rest the sticky issues persist.


#7

Wait, I thought it was discussed that the HTTP/FTP support is not going to fully go away, but relegated to an alternative method of content delivery?


#8

That is what i understand.


#9

Just to be clear. This is a Pull Request, it is not released to the main code base yet. https://github.com/highfidelity/hifi/pull/5694

  • @Richardus.Raymaker , in the first phase the assets will be stored on a single machine. Future versions they will be distributed
  • @OmegaHeron , it is not released yet. It still needs code review and testing. It is just a Pull Request.
  • @ctrlaltdavid , there are no plans to take away HTTP support for hosting assets.
  • @Ai_Austin , for this fist pass the texture will need to be embedded. We will solve for referenced textures in the future.

This PR is going to take a while to review, just want you all to be able to see what it is all about.


#10

Re HTTP, there’s a phrase, “It is meant to eventually replace the serving of existing HTTP/TCP assets” in the post the caught my attention.


#11

Thanks @chris, if it would run on VPS or dedicated server it’s indeed fine your way. Sometimes you need to get used to small changes. For home users i think web storage for assets is better solution. But it where a bits cared to see how much bandwidth 11 avatars used. shame i forgot to check cpu usage.

Other question, If object are deleted from the domain. the asset get deleted from the storage too ?

Is there a check that when you place the same asset in the domain it’s only stored once in the asset storage ?


#12

@ctrlaltdavid I think that is in reference to our own High Fidelity assets.


#13

@chris - Hooray for the return of asshat servers! In seriousness though, I have to ask the ‘elephant in the room’ question:

  • Does the ATP have any built in asset / IP rights protection? If not, are there plans to include protection for content creator’s IP rights in the future?

#14

Am pretty excited about ATP, need to test it on localhost.

But also makes it more intressting to move Hifi domain to a vps server, because bandwidth is not a big problem and costs are around the same. But that are only plans for now.

The asset server i would say give much more possibilities for protection too.


#15

@davedub this PR does not cover anything around IP protection.


#16

Ok, i tried to test it. Interface-dev installed. it start. And after that am complete lost. i have read soemwhere that you need to enable it in interface.exe. but like always when you need it you cannot find it back.

Other thing, it looks you need to start assignment client, but how or where or what ?

Other question: once you have uploaded some object. and you relog. where can you find the url back for the object or select it from uhmm inventory ?

Lost … need to look later again.

The option to enable assets is greyed out in the sandbox. Is there some hifi domain we can test it. or instructions how to activate the assigment client.


#17

Done a bit thinking, ATP is for me the same as what opensim use but BETTER :smile:
It only mean that we need after the ATP base code is working some extra tab on the stack-manager page where we can view all assets on the domain with coordinate and owner, and option to return it. Because when you store assets it need to be possible to easy spot and delete bad assets and remove it. or in case people complain the option to delete it.

This is especially the case when there are more creators and i hope at some point "personal zones’ different name for parcels.


#18

It’s noted before. But cannot find the topic back.
Asset upload is not working on windows. tried it just with build 3162
What’s the status for the fix ?


#19

@chris , any progress in fixing the above error for ATP ?


#20

Supposedly fixed by;