OMFG I think I am going to be sick

#1

High Fidelity just hit a new all time low for me.

Earth domain went unstable yesterday, this was due to the models.svo file blowing from 178K to over 25,000,000,000 overnight. (for some unknown reason)
Completely crashing every user that tries to connect.
I replaced the offending svo with a good working one and KABOOM, it all went south.

Earth disconnected itself from Hifi, and deleted all its own settings, yeah 70 odd users added to the allowed editors list gone in one fell swoop, all audio settings neutralized, passwords gone, sub locations gone (pass the bucket I’m gonna be sick)
And my touch panel script suddenly stopped working a few days ago for no apparent reason,
And my sandbox script suddenly stopped working a few days ago.
I have a pretty strong stomach for this kind of thing but even I have my limits, I have to rebuild everything, again, I am so sick of doing everything 3 and 4 times because hifi wont hold settings.

I spend most of my time trying to convince noobs that things arent as bad as they are, well it is that bad.

Even though I manually back up my domains regularly and work around the many issues regarding backups, it still found a way to screw me.

So over this. I need to lie down

Earth is offline until further notice, I am depressed until further notice.
Earth is not gone, I have backups of all your work and it will return soon if all goes well but for today I am over it.

#2

Take a break and come back later in a few days. Keep the logs around to share with the devs when they come back to their office. After all its been a weekend, and no ones usually at work. But as said in the other thread, yes the developers do really need to get more active responding to issues.

Earth is probably the best stress test we got right now for amount of assets and editors on domain and it will get bigger and more full of entities that it very hard to match. Any issues that that arise is because we are pushing it to the point bits begin to fall off or the engine just blowing up. It a matter of fixing these issues and they won’t pop up again (that sudden size increase is nuts)

Personally, the instability is the only reason I am only doing test builds and trying to save on the amount of time it takes to make something. HiFi is not ready and will not be ready for a year. I try absolutely not to burn my self out which is why I am online occasionally, not constantly.

#3

lol@stress test

its a subtle kinda rage this inspires innit on one hand ur logical brain goes “we shall rebuild” and then ur inner Cartman goes “screw you guys im going home…”

#4

Just noticed it my self that i hit it with the head saying stress test, it matches the case Both figuratively and literally (domain entities, and Adrian’s stress)

#5

That’s a tough one losing settings and other files. Having worked on projects at the alpha-ware stage at Apple and Microsoft, the experience for me here has not been that much different except for one aspect, and that is breaking the one maxim: Thou shalt never ever damage files or the file system. When that happens all work stops until the problem is understood and solved. I recommend prioritizing this problem to critical. So yes, Adrian, much sympathy from me. And yes, take a break.

I only visit earth briefly because the scene complexity cycling, the effect that simplifies the scene to just stars, has gotten really old for me. Have not found how to stop the sloshing, so I usually hang out in my domain.

#6

Yeah I will put in a bug report right away. Wait what? Bug report? Where?

Oh yeah there isnt one.

#7

Worklist, which is too damn difficult to use and goes with an assumption a third-party developer will look into it instead of the main developers.

As a side note, could you link up the log files after you have taken your break? between the time that the model.svo suddenly exploded in size and your config being lost.

After this, We now know there should be a daily backup of the config in (on windows)

Users\<user>\AppData\Local\High Fidelity\domain-server\High Fidelity\domain-server

Contains all config of the domain

as well as the damn model.svo file in (on windows)

Users\<user>\AppData\Local\High Fidelity\Stack Manager\resources

which still doesnt backup as it should… or atleast not updating “last modified” for files.

#8

Adrian, after all your hard work, this is quite disheartening. I hope that some questions start to get answered soon. I had this feeling that this was a summer weekend and most of the HF employees were off this weekend, and our dear Chris was, as they say, on holiday.

The BIG, BIG, NEXT BIG THING in health care is customer experience. I imagine it’s the same in VR. Our experience will be the difference between success and failure. We are not just testing the world, but the milieu as well.

Let’s put this on the agenda for Friday? Stability and backups and whatever else the more technologically savvy people think is behind this recent change.

If I were @Judas I would say something equally pithy and cheeky here, but…

#9
#10

@adrian, that S**ks. Now you finaly got all the lost bytes back in the models.svo. And now we don’t want it. Curious what’s inside the models.svo am thinking about a script that place lot’s of entities, how else can you bloat a file so quick.

Like @Menithal say, you can better backup the whole config dir. But it would be betetr if things like this don’t happen.

Now you know why am doing not so much in High Fidelity. To keep my mood at the green side.

#11

@Adrian, would you prefer we not re-build there at this moment or is it OK to throw some stuff down again? (sorry ya had to go through this!)

(Also, for anyone reading this, how to I go into fly mode? I can’t seem to find a page that tells me how to adjust my views and movements, thx! )

#12

Yep ok to resume building, domain is back up,
to go up press page up, you might have to hold it on to break gravity if you are on an entity, then just move around as normal.
To reset camera press ’ (apostrophe next to the enter key on windows)

#13

@Adrian a stretched box got away from me and went straight across the land. It won’t let me remove it. Help! thx!
@Menithal I will try that, thanks!

#14

I resized the long box to a 2m2m2m box, but if i am anywhere near that I keep crashing from strange reason when opening the edit script. Fixed that by restarting the edit script / restoring default scripts if you are crashing that is @Sublime

#15

Hi @Adrian sorry this happened to you. How did you get everything back up? Did you replace everything manually or some other way? Can you send over the large .svo? Also can you sent over your logs during that evening.

#16

@Sublime yep that happens to all of us, dodgy editing tools to blame, dont worry about it, we will have a cleanup one day.

@chris I rebuilt the domain, re-registered with metaverse, and reloaded all the users manually (see how much I love my residents :))
The svo file was just to big to do anything with, couldnt read it, couldnt download it so I just deleted it.
By the way this has happened to another of my domains, Adriania, the svo file blew out to some ridiculous size.

I have a feeling this was the cause of the loss of settings data, if the config file rewrites itself at each restart then my server only has 20G storage and this bloated svo file maxed me out and would no longer allow adding any more, so the file was not loaded? maybe?
Anyway I will send logs to you thanks Chris.

#17

me tink me lost de plot. pun intended

#18

Hmm. This could probably be tested by with a virtual host, limiting the environment size and making it near capacity.

It shouldn’t remove or rewrite the config file on restart if it cannot make a backup of it. Basically the stack manager should always prior restart be able to make a backup. If it can’t then it shouldn’t even try to remove the file.

#19

I have some manageable log files now.

I deleted the 22G one, and replaced a fresh backup, and put all the users in and manually saved the config file.
24 hours after restarting the models.svo has blown out to 2G,
so I saved the 2G svo file to send, and restarted and the svo file immediately went to 35M.

so I saved the 35M file and the associated logs so the files are smaller because it was a short run. I will send them.

The log files also blow out to unbelievable sizes, cant even open them without special tools.

#20

This is unacceptable as it makes it impossible to debug problems like this one. Another suggestion to the devs that there be a maximum log file size setting, and that the code close an active log file when it exceeds the size limit then open a new one. This is SOP for large or long-term systems.

Good news is we are learning where to apply needed chrome and polish.