Erratic FPS and Delay Performance in 4904 worse in 4970


#1

Continuing the discussion from Going out Staying in Judas's Blog:

This odd situation where walking or just turning in place ends up with odd delays, and too frame rates tanking at odd and frequent times is a bug. I ran the performance monitor and see that every delay and FPS tanking is correlated with interface doing disk activity. Would devs look into this problem please? It is affecting a lot of people including people with high end systems.

Though I experience this in many domains, I chose my conference domain for the best cse test since it has low content. Furthermore, the domain is local to me on a gigabit LAN. Just go there, walk to the middle of the main area, hold the left or right arrow key down to make the avatar spin in place, watch the delays happen. Every time there is a burst of disk activity you can see the avatar pause. Small bursts cause a large FPS drop. Longer disk activity bursts and the avatar stops moving.

Is there some synchronous disk I/O going on in Interface in the render pipeline??


4970 , Still flashing in HMD!
#2

I also see a ton of these messages in the logs:

[05/31 14:00:34] [WARNING] Long unsignaled sync  0x10  unsignaled for  4358485
[05/31 14:00:34] [WARNING] Long unsignaled sync  0x16  unsignaled for  4315970
[05/31 14:00:34] [WARNING] Long unsignaled sync  0x21  unsignaled for  4288985
[05/31 14:00:34] [WARNING] Long unsignaled sync  0x24  unsignaled for  4264656
[05/31 14:00:34] [WARNING] Long unsignaled sync  0x31  unsignaled for  4243603
[05/31 14:00:34] [WARNING] Long unsignaled sync  0x14  unsignaled for  4216981
[05/31 14:00:34] [WARNING] Long unsignaled sync  0x2b  unsignaled for  4196845
[05/31 14:00:34] [WARNING] Long unsignaled sync  0x6  unsignaled for  4171525
[05/31 14:00:34] [WARNING] Long unsignaled sync  0x20  unsignaled for  4151650

#3

i noticed tonight a frame drop with turning from 89 to 53 short. that where on playa in HMD.


#4

@Triplelexx and I ran into that old problem today where you can not move, you are stuck in a jump or flying pose and sometimes even relogging, clearing cache or moving to another domain does not fix.

One thing I did notice again was that the progress bar showed something that never finished loading. It was stuck at 10 or 20 percent. I seem to remember seeing something about this in one of the PRs that have been done since 4904 but because of the non-compatibility between the new ones and what most people run, I could not check to see if it fixed it.


#5

I’ll download the latest dev interface and stack manager build. I can try that on one of my local domains to see if things are fixed. So, I updated interface and the spiritmoving domain. It runs on the same computer as interface which makes it easy to eliminate other factors.

[edit]
results - I installed 4969 (latest dev build) both interface and my domain stack. Runs even worse than the current beta release. Spinning around oneself is slower. Walking around, slower. Collide with an object to walk atop it, nearly totally stoppage.

Reverted to 4904 - As before, far better than 4969, odd pauses when spinning in place. AT this point my guess is that interface is fetching model data from its disk cache and that is a synchronous process.

Will switch to previous beta to see how it fares.

My take on the abysmal performance of 4969 is that it seems as if it is getting much less efficient. It seems to be making more and more lesser systems useless, but since people with 980s are experiencing problems too, the performance problem really needs to be investigated.


#6

To be honest, I’ve been beating against this non-stop since I built a new desktop system to replace my 4 year old Alienware laptop. While the laptop was a bit anemic for GPU (GTX660M) it has an i7 CPU which turns out to be far more important for HF than I ever imagined. New desktop system is i5 based with GTX970. I decided to go i5 to keep my budget happy thinking I’d upgrade to i7 later as skylake i7s drop in price. Worked on paper.

Here - it’s been a non-stop torture device. Initially Interface ran 70% CPU (CPU NOT core) running default scripts doing absolutely nothing - not even connected to a domain. Turns out there was an issue with how script engine yields time (or in this case did not) that was fixed. Now I get low 30% use doing nothing. That’s a better thing, not good, but better.

Next torture event. What @Balpien.Hammerer is talking about. Assets coming in can totally obliterate performance - especially with ATP on LAN. Loading standard HF home scene with Interface/Stack all on same machine or Interface on Windows/Stack on Linux over LAN illuminate mechanism. Content begins to load - you try to move and you get a meltdown. Screen freezes, CPU use goes redline and, very often, even GPU driver crashes as it can’t get enough CPU time to execute its CPU required functions. Standard HF home scene is worst case scenario in that it has a lot of physics overhead and graphics overhead and tons of entities. Assembling all this into a rezzed scene is heavy. Too heavy if you dare try to move while it’s doing so.

Why isn’t this noticed and paid more attention to? Maybe because with an HMD on you don’t dare move until the scene is stable or you’ll lose your lunch. Or - they all have i7 CPUs where the problem is much less severe.

Reports of this have been made - including descriptions, proof of mechanism data etc. Hopefully it will soon get some attention. In the meantime, crazy as it may sound, I found that setting maximum ATP bandwidth (on stack) to something fairly low like under .5Mb/s or less helps a lot. It’s a stupid hack - but it’s made things usable when I’m forced to work with standard home scene. Of course that only helps if you’re using ATP.

I did my traces with xperf and Window performance analyzer - it really really shows what’s happening and confirms cause(s).


#7

I just downloaded the latest beta - 4970,and the results are:

  • Total standstill in Playa
  • Frame rate in 4904 at playa was 26-50, and now it is 0.2-2.0
  • In my very low content domains, conference and spiritmoving, I can barely move around smoothly

I no longer can be in any domain until this problem is fixed. I have been pushed out of HF by an even less efficient interface app. I thought mentioning these problems several times would have delayed the release as it is clearly fubared.

Oh well, let me know when it gets resolved.
Ciao!


#8

can you please send us your hifi-log.txt for a session of connecting to playa. Thanks. that would be beneficial in tracking this down.


#9

Logs Sent.

Perfmon shows heavy disk accesses when turning around. Disk I/O corresponds to FPS tanking and avatar turning pauses.


#10

Just got a note from one of the devs he is not going to look into this nasty performance because my GPU performance is too low.

One of the things I do is called canary testing where the code is run on edge limits hardware. It is the most effectve way of discovering overhead issues because performance problems can be masked on higher end equipment until the bugs that cause the overhead get institutionalized. By the time performance problems are discovered and acknowledged on high end hardware, so many other changes have piled on that it becomes quite difficult to fix things. This is particularly true in agile based development.

I just pointed out how the 970 and 980 systems are also experiencing that problem too. Menithal, Richardus as well as OmegaHeron have reported significant FPS drops both non-immersive and HMD. Using my lower end system, which BTW worked reasonably well until 4769, I can demonstrate the huge overhead bump added to that codebase. I also pointed out the correlation between disk I/O and those nasty FPS drops and movement pauses.

I urge the devs to fix the disk thrashing or maybe syncrhonous I/O which I pointed out and maybe this piece of bloated HF software will find a larger market.

Thanks, unfortunately, for nothing. Until then no HF for me. There’s no point in buying high end software that will get outrun by ever increasing software overhead.


#11

You know that’s not at all what I said. Feel free to share the actual text of my conversation with you.


#12

Stop with the myopia about my GPU. I said others with 980 system are experiencing this problem. Maybe an emphasis will brign some clarity:

I also pointed out the correlation between disk I/O and those nasty FPS drops and movement pauses. I urge the devs to fix the disk thrashing or maybe synchronous I/O which I pointed out and maybe this piece of bloated HF software will find a larger market.

The ball is in your court. Play ball!


#13

I have 2 primary rigs I use.

  1. Is my VR rig. It is using a 970 and the only place that I can cause it to choke ((using Rift / not Vive)) is Playa

  2. Is my home theater PC. This computer was constructed in 2007 with retail hardware and has gone through subsequent upgrades. It is running an Intel Q9650 and an ATI HD 7950. For a short time ((during a number of builds)) this machine simply couldn’t handle the wave shader and various other content heavy domains.

I’m writing this entry on machine 2, and I’m getting 60 fps with a fair cut of the render frame drawing the wave shader. My hardware knowledge is limited to my working with Arduino and Ethernet Shields so please don’t take this as an attack. I gather I don’t know my head from my arse when it comes to these deeper rooted issues. However, I consider this machine to be at or near the bare minimum of specifications and it still cuts the mustard. As for HDD I/O it is a 1.5 TB HDD on a SATA 2.0 ((just in case that info matters… i really don’t know)) I’m actually worried about upgrading to Windows 10 due to much of the testing others have conducted. It could mean the exact same machine no longer functions on par. I’ve had lots of folks I’ve tried to get on board; pun intended o.O and they simply couldn’t render anything.

I’m sorry if you wont be around for awhile. I was really hoping for that helicopter… :frowning: I’ll keep watching for it on the horizon.

But, it’s not all bubblegum and rainbows on deck my friend. I deal with crap too. Someone effectively destroyed proper use of my physics hull… AGAIN. It’s gonna be a long day…


#14

the fps goes up and down but i can spin around smoothly in my high tri domain (970)

the pause is the gif loop *


#15

Spin test. Modest Rig. Playa. < 20 fps.


#16

Done a new test after some vive update.

But this still looks like a big high fidelity bug.
Because my framerate on playa in hmd is now many time around 60fps with the stats window open and you look around to some content. i guess it’s the same with stats closed.

Desktop mode is still a stable 100fps. why the HMD is having so much more problems is not cl;ear the screen resolution is not so much different compared to my desktop (desktop: 1920x1080) (HTC Vive:2160 x 1200 , spec from internet)
I see the framedrops in desktop mode to. only in desktop mode it seems to recovery much better from it. and that need to be fixt in HMD. no flashing in desktop. Fix the Flashing in HMD !

and my i7-4790@4ghz with gf980 is really not budget system and need to run in HMD mode without framerate drop or screen flashing with hify. The framerate is dropping to at the same time.

some say my tree’s are to heavy. if i go walk to single tree the framerste is dropping too. the other tree’s i do not see at all, out of view range. looking at one of my tree’s create the same effect as looking around on playa. hify is made for only very low poor content in HMD mode. and for HMD you just need higher quality.

@Balpien.Hammerer , could you test you framerate at my domain simsquare ? especially with the tree’s.

@AlphaVersionD , what is your domain name. So i can check waht happens with my hmd on hour domain.

Curious how sansar is doing it with framerste.

If we could make screenshot in hmd with the stats it where easy to show. Does the log have option to registrate framerate drops ?


#17

@judas, your domain is special. that also kill sometimes my desktop frame rate :open_mouth: But playa is light compared to yours.


#18

It is called “open”.


#19

Thanks. render rate is a bit low. in HMD it where stuck around 30fps just turned off HMD restartedhify etc. and now in desktop mode the render rate is a 60-99fps.

Not seen to many flickring on open. mabye once. but not been there to long in HMD. It’s danrous place. already did fell overboard. :grin: did fell true a weak floor. :scream:

Global, it feels that the framerate works worse in the last version.


#20

AlphaD - I’ve filed a bug about this physics hull issue. We’re looking at it.