Cloud Domain - unable to start asset browser


#1

Hi HiFi Team,

I’ve setup today a cloud domain here via the highfidelity.com webside, and ist running. Just when I try to open the asset browser, I get an error message. Is seems it is not able to communicate with asset server. And I can’t close the message popup, it pop-up again and again… any help?

Thanks

Skimi


#2

And just to repeat, it can’t be closed anymore. Once opened the asset browser window, the error message popup x times.

Would really appreciate help in this case.

Thanks


#3

I had a similar thing happen with my DO cloud domain. @DrFran and I both made our’s at the same time except her’s is a $10 one with a placename located on the east coast US and mine is a $5 one with no place name on the west coast. Her’s seems to work fine, the default content shows up and things work as expected.

Mine is unusable. The default content does not appear, just a few cubes, a red platform and some particle mist. Eventually the unloaded default content times out and green outline boxes appear. Looking at the server nodes in domain settings, I can see the asset server blink off and on again, crashing and restarting.

Loading an externally hosted (S3) model seemed to work OK so I thought I would look at ATP assets and tried to open the asset browser window like @Skimi did above. I got the same error message window which then I could never close.


#4

Hi @Twa_Hinkle, yeah, that’s the same for me. I also had the unloaded stuff and red floor. And yes, I also have the $5 cloud domain.

But without the asset server it makes no sense to use it, cause if the stuff is loaded from Amazon server, the URL is visible in the logs and everything can be copied on the fly :confused:


#5

the fix for this is referenced above, you don’t want to build on a $5 server. i’ve had this happen a lot with the digital ocean images but always only on the $5. unfortunately i built on a 20 that couldn’t be scaled down and had to reset root pw, transfer models.json.gz over, delete the image, start a new image.

build in sandbox, export to cloud, never touch again until update (which doesn’t seem to be working anyway)…


#6

My point was, to a new user here, having nothing load and no clue, they would probably just quit and never come back.

I have other domains here, local host, EC2 and other machines. Most content hosted on S3 or other places. I was just trying the minimum DO config to see what it’s limitations were. And it is true, even hours later, the update still doesn’t work.


#7

The update works for me. The asset browser still doesn’t, even after update. I just wonder, is anyone of the devs looking at this problem? Will it be fixed? Or is it the asset server a ‘feature’, it is not included for the $5?


#8

The Asset Server and Asset Browser should function on all of the droplet configurations. It does seem like it might be crashing making it unusable.

We’re looking into this now - I will provide an update here on a fix for the issue once found.


#9

It they’re trying to run this on 1 cpu/512MB droplets and it’s attempting to bake any content beyond just tiny bits it’s going to die. I brought this up some time back in another thread when I warned some of the Linux people who ran on minimal hardware of impact ATP+baking could have. Given a simple content set ATP server usually survives baking cycle, but, it gets mighty mighty tight on a 1 CPU machine with 512MB, especially if no swap is enabled which is what D.O. prefers as they don’t want you burning up SSDs with swap and would prefer you buy more RAM. :slight_smile:

Disclaimer: Having not looked at the various configs on the templated D.O. servers I have no idea what the setup is, but, I was easily able to crash ATP on a 1CPU/512MB droplet using a stock compile and minimal hardware.


#10

We have identified a texture in the preloaded content that is particularly large and needs a large amount of memory to bake. That was, as @OmegaHeron suspects, causing us to run out of memory on the 512MB droplets. After looking into the issue we still believe that, with some necessary memory improvements, the 512mb droplet is viable for a typical 2-3 user experience.

I would like to outline how we’ve addressed and will continue to address this issue:

  1. For Cloud Domains launched after 4PM PST today (November 29th), the preloaded content is patched so that the bake of the crashing asset is already failed and will not be attempted.
  2. We will be reaching out to the users affected by the issue and offering to fix the issue manually for existing Cloud Domains. If you are affected by the issue and have a Cloud Domain you have yet to configure, you also have the option of deleting your current Cloud Domain and creating a new one (which will have the patch).
  3. We are implementing a general fix in the Asset Server so that it will not crash if the baking of an asset crashes for any reason. This means that in the future the Asset Server will simply catch crashes when baking assets and mark the bake for that asset as failed.
  4. We are reviewing our memory usage across all server components to reduce it as much as possible. This should allow for the 512mb droplet to handle more complex content (ex: baking larger textures).

#11

I tried my 512 DO domain again earlier today and it still did not load (before 4PM). Which is strange because one time when I logged into it last week everything was there, but then a day or so later it was gone again but probably it was gone from cache so I can understand why.

I will try updating the server now and see if that fixes it. I’d rather not have to delete the instance and start from scratch if I can help it. Thanks for the help.


#12

@Twa_Hinkle updating will not fix the issue - if you’d like to avoid deleting the instance we will be able to manually patch the Asset Server files on your droplet.

I will be sending out emails soon to the affected users to confirm how you’d like to proceed - if you can respond to that email that you’d like it manually fixed we’ll get your Asset Server back online shortly after that.


#13

hmm… oops, I was trying to avoid saying that the only reason I didn’t want to start from scratch was because I am lazy and did not want to redo my bookmarks and save the new URL etc. I have nothing on that domain yet, so nothing to lose. If I can’t get it to work I’ll respond to the email.

Thanks again.


#14

Yaay! Deleting the old one and starting from scratch has fixed the problems. Thanks for your help.


#15

Glad to hear it! Thank you for your assistance in reporting and testing the fix.


#16

I want to be sure that the explanation I posted is not misconstrued as an attempt downplay the issue outlined by @OmegaHeron.

We expect the majority of assets to bake successfully on the 512mb droplets. There can be assets with large textures requiring a large allocation of memory that may fail on the 512mb droplet.

We believe that steps 3 and 4 are an effective approach to make the 512mb droplet usable when the Asset Server contains assets with large textures. In that case, the Asset Server will mark the bake failed and the original asset will be downloaded by users visiting the domain.


#17

Hi @b,

thanks for looking into this Problem with the assets. I’ve killed my Cloud Domain this morning and installed a new Cloud Domain droplet. After a relog the same Problem appears for me.


#18

@Skimi - I’m sorry to hear that. It looks like you launched the second Cloud Domain before we had a fix for the issue out.

If it’s easy for you to delete and re-launch again, the issue is fixed now for new Cloud Domains.

If you’d like to keep your current Cloud Domain, we can manually fix it for you. Expect an email from us shortly outlining the options.


#19

yeah, now it works, thanks @b :sunglasses: And, I want say - it’s really cool what you guys have done with the droplets. It’s so easy to get an own domain in the cloud now. And no more hassle with updates. Great work! :+1:


#20

Great! Glad to hear it’s working now.

Thank you for your patience in dealing with the issue and your feedback in general on the new Cloud Domains feature!