Error while restoring a content backup


#1

Hi there,

before I’ve delted the broken cloud domain some weeks ago, I’ve made some backups of the server settings and content. The restore of the server settings works, just the restore of the content backup, I get for all three backups an error while uploading. the files are between 100 and 350 MB. Is there maybe a limit for the backup files? Maybe someone knows something? @b?

Thanks so much!


How to restore a downloaded backup on linux system
#2

Okay, no answer so far… sad

For me it seems, a larger backup is impossible to restore to the cloud domains. I found a working sandbox on my PC and made a fresh backup from the sandbox. Also around 300mb. If I try to restore it with the content settings page on the cloud server, after some time just a ERROR message pop up.

Even if no one has an answer, maybe it’s an important information for someone using seriously the cloud servers here on HF. You can make backups and feel safe - until you need one to restore.

Would be great, if a dev could answer here. Thanks


#3

This is a bit worrying to me because I’m going to be shutting down my domain by the end of the month. I have a content backup saved, which I plan to use to bring the server back someday in the future. But this problem makes me think I might not be able to.

I wonder if this is a problem specific to your server, or to the location of your server? Have you tried creating a different cloud domain (on a different location - ie. New York, San Fran) and restoring to that one? I’ve had problems with the San Francisco server, and had to switch it to New York to get it working. Some of them seem to work less well than others.


#4

Hi @Theanine, thanks for your reply. Yes, I’ve tried a server here at my location Frankfurt/Germany, San Francisco and also New York. It’s on all servers the same error:

Okay, maybe the backups are not broken, and it’s just something with upload of them. maybe a timeout or something else someone can fix. Maybe it never was tested with large backup files - don’t know.


#5

Hello -
Looking at that error message it seems like the problem might be with the archive. We will look into this next week. If you don’t mind sending us the archive we could start by verifying the archive is not corrupted. Thanks!


#6

Hi @divia, thank you! I’ll send you a message with the links!


#7

Thanks I received it. Will get back to you when I find out more.


#8

Hi @Skimi,

I have tested restoring your archives locally and was able to.
Taking a closer look at you error message, there is definitely an issue during the upload phase.
We have tested large files, but not on small connexions.
We’ll need a bit more information to figure out what exactly is going wrong.

Are you familiar with your browser’s DevTools?
If not, here are some instructions, the dev tools are pretty similar between most browsers.

  • Got to your domain content settings page
  • Right click on the page
  • Select the “Inspect” option
  • A new panel should open in your window
  • Select the “Console” tab in that panel.
  • On the web page, upload your archive
  • You might see some errors and warnings appearing in the console you just opened, forward those to me.

#9

Hi @c, thank you for taking a look to this. I’ve made two vids for you. In short form, I get always a 400 (Bad Request) error message, instantly after login to the server. During the upload process there is no output in the console:

Failed to load resource: the server responded with a status of 400 (Bad Request)

After a second try, I got this:

jquery-2.1.4.min.js:4 GET http://206.189.66.204:40100/api/domains/18c9a70b-7749-42d1-b98c-430501181656 400 (Bad Request)
send @ jquery-2.1.4.min.js:4
ajax @ jquery-2.1.4.min.js:4
getDomainFromAPI @ shared.js:188
reloadDomainInfo @ settings.js:765
Settings.afterReloadActions @ settings.js:37
(anonymous) @ base-settings.js:122
j @ jquery-2.1.4.min.js:2
fireWith @ jquery-2.1.4.min.js:2
x @ jquery-2.1.4.min.js:4
(anonymous) @ jquery-2.1.4.min.js:4
load (async)
send @ jquery-2.1.4.min.js:4
ajax @ jquery-2.1.4.min.js:4
n.(anonymous function) @ jquery-2.1.4.min.js:4
getJSON @ jquery-2.1.4.min.js:4
reloadSettings @ base-settings.js:106
(anonymous) @ base-settings.js:173
j @ jquery-2.1.4.min.js:2
fireWith @ jquery-2.1.4.min.js:2
ready @ jquery-2.1.4.min.js:2
I @ jquery-2.1.4.min.js:2

here are the vids, just for the case:


#10

Just to keep everyone else in the loop, I looked into this further with Skimi and then on my own.
I narrowed it down to the fact that the server runs out of RAM while trying to download and process the archive.

This is especially more likely to happen right after the server was setup since it might be baking the original assets for the first few minutes.
There are a few things we can do to reduce the memory footprint that we’ll be looking into.


#11

@Skimi good news, I have a PR up that should fix the issue you’ve been seeing for servers with limited memory:

It probably will not make it to RC68, but it will be in RC69.

As I was working on this issue, I discovered a separate one where multi-gigabyte archives will fail instantly due to the library we use to transfer the file not handling files that big.
A bug has been filed separately and we’ll get to it as soon as possible.


#12

Great news, thank you @c Clement. At moment I’m on vacation in dutch. Will try it when I’m back home.


#13

Hi @c,

it is still not possible to restore a backup file to the cloud domain. I can restore them all on my local sandbox, the files are not corrupted. At last I thought, maybe it’s because my ISP and the low connection I have 16/3 MBit, but today I’ve changed to 400/40 Mbit and a different ISP - but still the same problem.

I’ve Setup a fresh cloud domain, took the latest backup from my local sandbox - and fail to upload it.

I wonder, nobody else has this problem? Is the problem related to the country I am from? My machine is with the latest Windows 10 - no extra firewalls or so, no additional blocker from the net, just pure Windows.

I am a little desperate now, and I don’t really want setup a domain always by hand from scratch again. Help :worried:


#14

Its disappointing that your Hifi backup restore fails, I have no answer for that, but if you are using Digital Ocean cloud servers then I suggest in future taking image snapshots from time to time.

They have this as a regular service, they restore instantly (well one or two clicks and a few minutes) but it restores everything exactly as it was not just the models file but all the settings and config too, and is very reliable. Digital Ocean will store them for a dollar or 2 a month (depending on size).
They wont work on your sandbox but you have the hifi backup for that.
I know it doesnt help you in this case but may be good for future consideration.


#15

Hi @Skimi,

Can you tell me how big that archive file is and how big your Digital Ocean instance is?

- Clément


#16

Hi @c, thanks for getting back here

the backup file has 197 MB

the Digital Ocean cloud domain is the 2GB RAM, 50 GB space thing for 2-3 people. But I also tried it with a larger cloud domain, like the 4 GB RAM and 80 GB space, it fails the same way.

Update: For testing, I’ve tried to restore an old 20 MB backup, that works. So maybe it has something to do with the file size?


#17

And you are able to restore that exact backup on a local sandbox?

(The previous issue we had was that when we decompress the backup and upload the assets to the asset server we ended up with multiple copy of the assets in RAM and running out of it, but 2GB should be more than enough for a backup that size)


#18

Yeah, I’ve done that several times now. I can restore it on a local sandbox, and get an exact what I am expecting. That works smooth.


#19

Do you know if it fails during the upload or the restore?

Do you ever see the “Restore in progress” loading bar?
If you could send me a screenshot of the error (if any) that would be useful too.

If you wouldn’t mind sending me a copy of the backup in PM too that would help me debug the issue directly.


#20

Nope, I never have seen a restore in progress bar at the cloud domain. It’s only the upload window, followed with the error message, after ca. 10 sec.

I also send you a PM with the link to OneDrive.