Backups not saving?

Adrian · 2015-02-11 06:14:28 UTC

I have been doing some testing (nothing new)
things went wrong, stuff disappeared (nothing new)

But when I went for a backup to restore an older copy, the backups are way out of date.
Here is a screenshot, look closely at the dates, the red arrows point to the working .svo it has todays date,
and the last half hour.svo backup, which shows a date 3 weeks ago, apparently the half hourly ones arent working, they are all 3 weeks old.
The weekly backups seem to be working by the dates, the daily ones are over 2 weeks old.

I would expect to be able to go back to a backup half an hour ago or one hour ago, but there none there.

Adrian · 2015-04-22 04:08:09 UTC

Sorry but ALL this is still an issue, I cop a one two punch from time to time.
Stack Manager still eats content regularly, and backups are unreliable. The result is permanent damage to the domain content. (Windows) And left my domain unrestorable (I had to rebuild).
This is confirmed with another user @Twa_Hinkle has experienced the same thing, the half hourly backups stopped working a month ago and the daily backups are inconsistent.
Twa notes that he made his own rule for 12 hourly and that works ok.

Something is wrong with the auto backup system, still.
I suggest all users on windows make manual copies of the models.svo file and put it on your HD somewhere safe, do this every time you make serious changes until its fixed. So that if the stack manager eats something, you can restore.

I also had an issue once recently where everything was gone. The models.svo content was replaced entirely with the content from the describe-settings.json file. This file resides in the same folder. That might be a hint as to whats going wrong.

chris · 2015-04-22 04:35:00 UTC

@Adrian that does not sound good. Few things.

A backup will only happen if there has been a change to your domain. If you make no changes, backups will not run. From you last backup were changes made?
We put in place some logging for situations where models disappear from your domain. It is a check box (on by default):

This will mean your server logs files will show anything that has happened, if you look at that do you see anything that does not look right?

Adrian · 2015-04-22 06:42:15 UTC

Thanks for your reply @chris I will check those logs.
Meantime please check my resources folder,
the current models.svo is 17K
the weekly backup is 17K
but none of the other backups are at 17K surely the last half hourly backup should be 17K , but its not its only 3K
and it is at least a week out of date. Also the daily backups havent changed since the 16th.

And yes I have been making changes which is what brought this to my attention again.

I think this proves the backup system isnt working properly.

Thanks

chris · 2015-04-22 16:45:32 UTC

@Adrian, check and send you logs. Also can you send/post your domain config file.

Adrian · 2015-04-23 07:34:39 UTC

@chris I found these errors in my log file:

[DEBUG] [04/22 19:05:51] [entity-server] persist operation calling backup...
[DEBUG] [04/22 19:05:51] [entity-server] backup operation wantBackup: true
[DEBUG] [04/22 19:05:51] [entity-server] Checking [ "Half Hourly Rolling" ] - Time since last backup [ 768950912563 ]  compared to backup interval [ 1800000000 ]...
[DEBUG] [04/22 19:05:51] [entity-server] Time since last backup [ 768950912563 ] for rule [ "Half Hourly Rolling" ] exceeds backup interval [ 1800000000 ] doing backup now...
[DEBUG] [04/22 19:05:51] [entity-server] Rolling old backup versions for rule "Half Hourly Rolling" ...
[DEBUG] [04/22 19:05:51] [entity-server] rolling backup file  "resources/models.svo.backup.halfhourly.4" to "resources/models.svo.backup.halfhourly.5" ...
[DEBUG] [04/22 19:05:51] [entity-server] ERROR in rolling backup file  "resources/models.svo.backup.halfhourly.4" to "resources/models.svo.backup.halfhourly.5" ...
[DEBUG] [04/22 19:05:51] [entity-server] rolling backup file  "resources/models.svo.backup.halfhourly.3" to "resources/models.svo.backup.halfhourly.4" ...
[DEBUG] [04/22 19:05:51] [entity-server] ERROR in rolling backup file  "resources/models.svo.backup.halfhourly.3" to "resources/models.svo.backup.halfhourly.4" ...
[DEBUG] [04/22 19:05:51] [entity-server] rolling backup file  "resources/models.svo.backup.halfhourly.2" to "resources/models.svo.backup.halfhourly.3" ...
[DEBUG] [04/22 19:05:51] [entity-server] ERROR in rolling backup file  "resources/models.svo.backup.halfhourly.2" to "resources/models.svo.backup.halfhourly.3" ...
[DEBUG] [04/22 19:05:51] [entity-server] rolling backup file  "resources/models.svo.backup.halfhourly.1" to "resources/models.svo.backup.halfhourly.2" ...
[DEBUG] [04/22 19:05:51] [entity-server] ERROR in rolling backup file  "resources/models.svo.backup.halfhourly.1" to "resources/models.svo.backup.halfhourly.2" ...
[DEBUG] [04/22 19:05:51] [entity-server] Done rolling old backup versions...
[DEBUG] [04/22 19:05:51] [entity-server] backing up persist file  "resources/models.svo" to "resources/models.svo.backup.halfhourly.1" ...
[DEBUG] [04/22 19:05:51] [entity-server] ERROR in backing up persist file...
[DEBUG] [04/22 19:05:51] [entity-server] Checking [ "Daily Rolling" ] - Time since last backup [ 526458912563 ]  compared to backup interval [ 86400000000 ]...
[DEBUG] [04/22 19:05:51] [entity-server] Time since last backup [ 526458912563 ] for rule [ "Daily Rolling" ] exceeds backup interval [ 86400000000 ] doing backup now...
[DEBUG] [04/22 19:05:51] [entity-server] Rolling old backup versions for rule "Daily Rolling" ...
[DEBUG] [04/22 19:05:51] [entity-server] rolling backup file  "resources/models.svo.backup.daily.6" to "resources/models.svo.backup.daily.7" ...
[DEBUG] [04/22 19:05:51] [entity-server] ERROR in rolling backup file  "resources/models.svo.backup.daily.6" to "resources/models.svo.backup.daily.7" ...
[DEBUG] [04/22 19:05:51] [entity-server] rolling backup file  "resources/models.svo.backup.daily.5" to "resources/models.svo.backup.daily.6" ...
[DEBUG] [04/22 19:05:51] [entity-server] ERROR in rolling backup file  "resources/models.svo.backup.daily.5" to "resources/models.svo.backup.daily.6" ...
[DEBUG] [04/22 19:05:51] [entity-server] rolling backup file  "resources/models.svo.backup.daily.4" to "resources/models.svo.backup.daily.5" ...
[DEBUG] [04/22 19:05:51] [entity-server] ERROR in rolling backup file  "resources/models.svo.backup.daily.4" to "resources/models.svo.backup.daily.5" ...
[DEBUG] [04/22 19:05:51] [entity-server] rolling backup file  "resources/models.svo.backup.daily.3" to "resources/models.svo.backup.daily.4" ...
[DEBUG] [04/22 19:05:51] [entity-server] ERROR in rolling backup file  "resources/models.svo.backup.daily.3" to "resources/models.svo.backup.daily.4" ...
[DEBUG] [04/22 19:05:51] [entity-server] rolling backup file  "resources/models.svo.backup.daily.2" to "resources/models.svo.backup.daily.3" ...
[DEBUG] [04/22 19:05:51] [entity-server] ERROR in rolling backup file  "resources/models.svo.backup.daily.2" to "resources/models.svo.backup.daily.3" ...
[DEBUG] [04/22 19:05:51] [entity-server] rolling backup file  "resources/models.svo.backup.daily.1" to "resources/models.svo.backup.daily.2" ...
[DEBUG] [04/22 19:05:51] [entity-server] ERROR in rolling backup file  "resources/models.svo.backup.daily.1" to "resources/models.svo.backup.daily.2" ...
[DEBUG] [04/22 19:05:51] [entity-server] Done rolling old backup versions...
[DEBUG] [04/22 19:05:51] [entity-server] backing up persist file  "resources/models.svo" to "resources/models.svo.backup.daily.1" ...
[DEBUG] [04/22 19:05:51] [entity-server] ERROR in backing up persist file...
[DEBUG] [04/22 19:05:51] [entity-server] Checking [ "Weekly Rolling" ] - Time since last backup [ 193491032563 ]  compared to backup interval [ 604800000000 ]...
[DEBUG] [04/22 19:05:51] [entity-server] Backup not needed for this rule [ "Weekly Rolling" ]...
[DEBUG] [04/22 19:05:51] [entity-server] Checking [ "Thirty Day Rolling" ] - Time since last backup [ 675036942563 ]  compared to backup interval [ 2592000000000 ]...
[DEBUG] [04/22 19:05:51] [entity-server] Backup not needed for this rule [ "Thirty Day Rolling" ]...
[DEBUG] [04/22 19:05:51] [entity-server] persist operation DONE with backup...

Adrian · 2015-04-23 08:34:27 UTC

This is my json config file (password obscured, last 4 digits of the access token obscured)

{
    "metaverse": {
        "access_token": "e8069acca903b5092e2ac0566c57c49ecb86082918bfc562b407c4608611****",
        "automatic_networking": "full",
        "id": "f1531473-d070-4caf-823b-cc058ae968e8"
    },
    "security": {
        "allowed_editors": [
            "Adrian",
        ],
        "allowed_users": [
            "adrian"
        ],
        "http_password": "******",
        "http_username": "adrian"
    }
}

I must admit this looks somewhat incomplete, the last modified date on this file is 20-4-15

ZappoMan · 2015-04-28 20:40:26 UTC

Is it possible those files “resources/models.svo.backup.daily.2”… etc… that are reporting “ERROR” are unwritable? In other words is there some other process locking them? Do those files already exist? Do they not exist? Are you low on disk space?

Adrian · 2015-04-29 02:06:47 UTC

hey @ZappoMan,
I dont know why some files would be writable and some not, but I guess it is possible, this particular server is my home computer and the firewall always asks for access a few times everytime I start it, then asks again everytime someone comes into the domain, so if the firewall is involved in any way with the writing of the backups then it is possible.

Several programs appear in the list as wanting access, domain manager, stackmanager.exe assignment clients, which of these would be responsible for writing backups? Maybe I should set up a special rule.

ZappoMan · 2015-05-05 22:21:42 UTC

Note: last week we added some additional logging, which should display the OS level error message as to why we couldn’t copy/rename the file. You might want to check your log files to see if there is additional information.

Judas · 2015-06-04 11:08:32 UTC

A load of models vanished from my domain again, the last back up the stack manager made was may 11
may be worth checking what yours is doing.
lost it all

Adrian · 2015-06-04 13:02:47 UTC

Earth is backing up ok, so is adriania, my home server (australia) doesnt back up well at all, but it may be a firewall issue.

I also take manual backups of earth regularly because I have learned not to trust the system.

Judas · 2015-06-04 13:30:03 UTC

What happened was I noticed half the items at shop had vanished, so decided to use a back up to put it back, But of course i Built a bunch of new stuff in since then. I stopped the stack replaced the model svo with back up, which worked but i forgot about all the new buildings so thought id swap back.
went through the process again and it wiped the saved file leaving me with an empty domain. so I thought id cut my losses and put the 3 month old one back.
and kaboom also wiped.
which was nice

The models vanished from a locked domain.
So its a bug
I have nothing to submit other than empty svo’s
so I will have to start again.
Only the edit tools are a joke at the moment and theirs still presumably an un identified bug and Possibly another one related to restoring from back ups
I’m not brimming with enthusiasm as you can imagine at the moment.
I shall just do nothing until it happens to everyone else.

Its not like the windmill vanished from a domain that only chris had build perms in during last weeks meeting

Adrian · 2015-06-04 14:41:25 UTC

Yeah theres a tripple gotcha, its not until things go horribly wrong and you need a backup that you realise the backups werent working. So you go for an older backup and the machine eats it leaving you wishing you just lived with the loses. Sometimes when you put in a backup, as soon as you start the domain it rewrites the svo file, and I have seen a perfectly good models.svo turn into garble on a first run. It is most definitely a bug, a nasty one.

So another thing I have learned is copy out your models.svo and put a copy into service, dont put your only copy in service, because sure as shit thats when its going to fritz.

I feel for you, losing all that work, it leaves a big empty feeling.

It would be nice if the critical stuff was fixed.

Judas · 2015-06-04 14:49:42 UTC

I wish the svo was in plain text
1 model url location x,y,z rotation x,y,z scale xyz
then I could just cut paste between them from them.
course that would be too easy.

Menithal · 2015-06-04 14:52:03 UTC

That’s probably coming sooner than you think. We already got svo (json) import/ export in interface >_>

For now due to the domain issues I try to avoid doing too much domain related work

Judas · 2015-06-04 14:59:34 UTC

id rebuild but its like unworkability jumpy editing things

I had this dream to exhibit my students work in a domain and let people use the rift to view it.

Cant get the rift working on my win 8 tablet, its bugged to hell intermittently working (dk1 obsolete)
i got the Tuscanny demo working but cant get hifi to play with it now
it seems fine on my desktop but i’m not bloody taking that to work.
Even if the above 2 things were working
editing stuff in world as mentioned elsewhere is dire
i feel like I never used hifi before
so I’m going to as always print out stripey versions of their work on the laser printers.
and be a unimpressive thing in the corner again.

Maybe the future will be here next year.

Twa_Hinkle · 2015-06-05 04:56:16 UTC

My domain seems to be unusable… searching for backups looks like I don’t have any for months… wtf…

I thought this got fixed months ago.

And yes… editing stuff sucks… is worse than Judas’s vids… what is wrong with simple pan and orbit with toggle between blender (middle mouse), SL (alt/ctrl), or other method… cam control buttons suck, only ppl who don’t know better use them…

And is it just me that when I zoom and orbit/pan camera position to what I want to be seeing… all of a sudden with no input from me, not touching keyboard or mouse, reverts back to standard view?

Don’t mind me… I’m just pissed off because of undeletable line entities that someone left along with particles and other stuff… which is strange because supposedly only few have create rights there.

ericrius1 · 2015-06-05 15:00:23 UTC

Hi Twa_Hinkle,
I’ll look into the un-deletable line entities today. I was seeing this issue yesterday in sandbox, and I believe one of the reasons for it is we recently changed the way line entities work. So the lines people use to make now are not correctly conforming to the protocol- since before lines only took two positions (using position and dimension properties for two points) and now points take an array of points with a property called linePoints.

I would perhaps try running a script which deletes all your line entities as an assignment script. If you do that, let me know if it works!

Also what domain are you referring to?

Eric

Twa_Hinkle · 2015-06-07 06:19:49 UTC

It seems to be a moot point now. I shut down the domain for a day and when I restarted it all of a sudden it was reverted to the state from a month or more ago. Yes the line entities are gone but so is everything else that was added in the last month or more.

I did not restore the svo files… or anything else… it seems to have happened when the stack manager got updated. I could have salvaged some of the new stuff from what was there but now none of it exists.

I was not aware that updating stack manager could cause content to revert back to backed up version… and in this case… backups have not worked correctly for me for weeks.

Today I saw some of the purple lines that I think were undeleteable for me on another domain… not sure where… maybe earth… or one of the new ones.

For me this is just another bump in the road… and good thing is all alpha still. But the concept of deleting content when updating open source code sort of doesn’t seem right.