TThrottle and extensive pagefaulting

[Pepo]

There is one thing I wanted to ask a long time ago... observed on both Win XP SP3 and Win 7. When TTh is actively throttling BOINC applications, the process causes a lot of page faults. E.g. my currently not even 10 days running TTh has already had 1770 mil. page faults (2100/sec over the whole time, but has been throttling just a part of the time), I can often see up to 200 000 page faults during a second or two, followed by a silent pause.

I believe these are not the "hard page faults", caused by the swapping to/from HDD because of the lack of RAM, but "soft page faults", possibly caused by accessing memory protection boundaries, some types of system calls, etc. Hard to tell without any possibility to see symbolic call stacks. But the net effect is the application uses maybe 3% of its CPU life in user mode and 97% in the system calls. 1-2 CPU hours daily.

Any way to find out, what exactly is causing this? Could solving this allow lowering TTh's CPU footprint?

BTW, a long time ago I've already reported the Process Explorer's and Process Hacker's dev teams, that their applications have issues with such pagefaulting programs, because when their counters overflow 31 bits, the values are either displayed negative, or incorrectly positive, or even the broken displaying routines get blocked

[fred]

There is one thing I wanted to ask a long time ago... observed on both Win XP SP3 and Win 7. When TTh is actively throttling BOINC applications, the process causes a lot of page faults. E.g. my currently not even 10 days running TTh has already had 1770 mil. page faults (2100/sec over the whole time, but has been throttling just a part of the time), I can often see up to 200 000 page faults during a second or two, followed by a silent pause.

I believe these are not the "hard page faults", caused by the swapping to/from HDD because of the lack of RAM, but "soft page faults", possibly caused by accessing memory protection boundaries, some types of system calls, etc. Hard to tell without any possibility to see symbolic call stacks. But the net effect is the application uses maybe 3% of its CPU life in user mode and 97% in the system calls. 1-2 CPU hours daily.

Any way to find out, what exactly is causing this? Could solving this allow lowering TTh's CPU footprint?

BTW, a long time ago I've already reported the Process Explorer's and Process Hacker's dev teams, that their applications have issues with such pagefaulting programs, because when their counters overflow 31 bits, the values are either displayed negative, or incorrectly positive, or even the broken displaying routines get blocked

1) TThrottle version?
2) Is the system low on memory?
3) Do you see these page faults on the TThrottle process? How do you check this, what program do you use.
4) TThrottle uses almost no memory except for the graphics. You may switch them off in the 4th tab.
5) I checked this on several machines, but I hardly see any page faulting occur.
http://en.wikipedia.org/wiki/Page_fault

[Pepo]

...observed on both Win XP SP3 and Win 7. When TTh is actively throttling BOINC applications, the process causes a lot of page faults, I can often see up to 200 000 page faults during a second or two, followed by a silent pause.

I believe these are not the "hard page faults", caused by the swapping to/from HDD because of the lack of RAM... But the net effect is the application uses maybe 3% of its CPU life in user mode and 97% in the system calls. 1-2 CPU hours daily.

BTW, a long time ago I've already reported the Process Explorer's and Process Hacker's dev teams, that their applications have issues with such pagefaulting programs...

1) TThrottle version?
2) Is the system low on memory?
3) Do you see these page faults on the TThrottle process? How do you check this, what program do you use.
4) TThrottle uses almost no memory except for the graphics. You may switch them off in the 4th tab.
5) I checked this on several machines, but I hardly see any page faulting occur.

1. Ì believe any I've used during last 1/2 year, but I've checked just occasionaly.↲
2. 2 or 4 Gig , plenty available. Independent.
3. With mentioned P.Explorer or P.Hacker., on main window or TTh's process' properties dialog window. TaskMgr should show the same. I'll try also Resource Monitor. IIRC I've also played with various counters' graphs in PerfMon.
4. Not low on memory , I understand this.
5. I'll  have to post some graphs and screenshots tomorrow.

[Pepo]

But the net effect is the application uses maybe 3% of its CPU life in user mode and 97% in the system calls. 1-2 CPU hours daily.

The pattern is, a couple of seconds of silence, followed by a small red spike, taking a half to one and half second.

[fred]

But the net effect is the application uses maybe 3% of its CPU life in user mode and 97% in the system calls. 1-2 CPU hours daily.

The pattern is, a couple of seconds of silence, followed by a small red spike, taking a half to one and half second.

Even TThrottle has to work sometimes. Tab Extra Rebuild list after .. seconds.
But I will take a look at what is running. The cpu usage should be lower than that.

[Pepo]

2) Is the system low on memory?
4) TThrottle uses almost no memory [...]
5) I checked this on several machines, but I hardly see any page faulting occur.

2. 2 or 4 Gig , plenty available. Independent.
4. Not low on memory , I understand this.
5. I'll  have to post some graphs and screenshots tomorrow.

5. The promised screenshots (already 2 days old, I've since lowered the CPU frequency a bit, thus the throttling is not necessary and pagefaults happen just at the "list rebuild" rate (set to 60 seconds)).


TTh, 5.0s sampling - list rebuilds+throttling at 2.15GHz
(the term "sampling" under the screenshots means the Process Explorer's data sampling period)


TTh, 0.5s sampling - list rebuilds at 1.75GHz, later list rebuilds+throttling at 2.15GHz


TTh, 0.5s sampling - page faults + sys calls peak - list rebuilds+throttling at 2.15GHz

BTW, the Process Explorer's PF counter is again "over" and counting down, now at 1 002 307 988
At least the Task Manager is displaying the count correctly (I'm wondering what will happen at the 32-bit boundary):

TTh's page faults in Task Manager


2.+4. I've checked the Resources Monitor, and as I'm looking at it now, it displayed 2-4 Hard Faults/sec for maybe 5 aplications - as an average for the last 1 minute, then the average slowly faded out and there is currently 0 H-PF/s. Well, now some mcupdate.exe (MediaCenterUpdate?) has 45/s and again it slowly fades out.)
TThrotle simply has no (0) Hard Page Faults even when actively throttling, thus it has nothing to do with lack of RAM vs. HDD use.

TTh having 0 Hard page faults in Resource Monitor.png

[fred]

So TThrottle makes 200000 pagefaults a second for the one second every 60 seconds. The rest of the time no pagefaults even when throttling.
And you tested this on Win 7 64.

[Pepo]

Nearly exactly.

So TThrottle makes 200000 pagefaults a second for the one second every 60 seconds.

Yes. Sometimes more, sometimes less. I guess it takes a bit longer than one second, maybe with Process Hacker I could obtain even finer sampling to see the nice red gaussian curve

The rest of the time no pagefaults even when throttling.

Not exactly. No pagefaults inbetween when not throttling (see the left half of the second image), but also pagefaulting when throttling is enabled (see the smaller spikes at the right half of the second image, or on the third image).

Maybe these are the moments, when processes are paused or resumed? No, just checked on my slow old AMD Duron (err, already some Athlon) - the moments when BOINC processes are being paused or resumed (I can observe it with Process Explorer, paused processes are painted/highlighted with grey color) do not seem to correlate with page fault and CPU usage spikes.
And, as the ProcExp on WinXP displays the context switches (rather than CPU cycles on Vista and 7), I can se that now TTh consumes around 25-34-40 CSW per half second even during processes being throttled, but it rises somewhere to 80 CSW during these CPU (and PF) spikes.

And you tested this on Win 7 64.

Not only, I was observing it for a long time on different machines with Win XP SP3 x32 - Core2D T7200 and AMD Athlon.

[Pepo]

So TThrottle makes 200000 pagefaults a second for the one second every 60 seconds.

Yes. Sometimes more, sometimes less. I guess it takes a bit longer than one second, maybe with Process Hacker I could obtain even finer sampling to see the nice red gaussian curve

It goes down to 250 msec, and displays "huge curves" (proc list),  narrow needle spikes (each couple of seconds) and sometimes even tiny spikes inbetween.

TTh 0.25s sampling - list rebuilds+throttling at 2.15GHz

[...] the moments when BOINC processes are being paused or resumed (I can observe it with Process Explorer, paused processes are painted/highlighted with grey color) do not seem to correlate with page fault and CPU usage spikes.

I may be pretty wrong with this assumption. TTh may (un)throttle the processes at a much higher rate than the Process Explorer's sampling rate. [...] With Process Hacker I've seen the throttling periods (it highlights single threads) being shorter or longer fractions of a second.

There was one more interesting feature in Process Hacker - it displays also the Pagefile Usage in the process' memory details, and the (otherwise steady) values do change a bit during the spikes:

TTh memory details - silent

   
TTh memory details - busy states

The various values always seemed to return to the "steady state" afterwards:


So, the Page File seems indeed to be somehow used, but not because of usual VM paging. Maybe some shared memory block, backed through the Page File? Maybe accessing some feature like this might issue interrupts or page faults or else...??

BTW, a long time ago I've already reported the Process Explorer's and Process Hacker's dev teams, that their applications have issues with such pagefaulting programs, because when their counters overflow 31 bits, the values are either displayed negative, or incorrectly positive, or even the broken displaying routines get blocked

As you can see, the total Page Fault count already overflowed the 32 bit border and while being in sane ranges, the Process Hacker is fortunately not displaying a garbaged process' memory block details anymore ("just" the displayed system's total Page Faults Transition counter is displaying exactly the same damaged value like "18 446 744 07..")

[fred]

The problem is I don't see any of these things.
I am testing it now on a notebook with W7 64 and see no pagefaults at all.
TThrottle is throttling close to the max, all the time on both CPU and GPU.

It may well be that one of these hacker tools is messing things up.

If there  is anybody else with these kind of problems, please let me know.

And again pagefaults are no faults, just parts that are missing in active memory.
This is normally a OS problem and TThrottle has no influence on this.
Maybe another program is using a lot of memory and forces part or the memory TThrottle is using to disk.

[Pepo]

The problem is I don't see any of these things.
I am testing it now on a notebook with W7 64 and see no pagefaults at all.
TThrottle is throttling close to the max, all the time on both CPU and GPU.

Weird... I'm not painting it

It may well be that one of these hacker tools is messing things up.

Could be. Just if I knew what to deinstall... I'll try once to chek it if I'll get some clean system under my hands.

Maybe another program is using a lot of memory and forces part or the memory TThrottle is using to disk.

I believe I've proven it has nothing to do with the usual swapping from/to page file because of lack of RAM...

[fred]

The problem is I don't see any of these things.
I am testing it now on a notebook with W7 64 and see no pagefaults at all.
TThrottle is throttling close to the max, all the time on both CPU and GPU.

Weird... I'm not painting it

It may well be that one of these hacker tools is messing things up.

Could be. Just if I knew what to deinstall... I'll try once to chek it if I'll get some clean system under my hands.

Maybe another program is using a lot of memory and forces part or the memory TThrottle is using to disk.

I believe I've proven it has nothing to do with the usual swapping from/to page file because of lack of RAM...

The hacker program installs a driver and that should be the only part to worry about. Drivers are hard to test and debug, easy to make a mistake.
And yes... but any page fault other than a protection error is normally nothing to worry about.
But I keep it in mind and if I get a bright idea. But whatever works without any apparent errors.

[benDan]

Hi fred!
Could this be related to the problems some people have with lockup during startup?
I see this on my PC using Project Lasso and Windows task manager.
values jump form 0 to 23k and back.
I just sent you a log with debug on.

[fred]

Hi fred!
Could this be related to the problems some people have with lockup during startup?
I see this on my PC using Project Lasso and Windows task manager.
values jump form 0 to 23k and back.
I just sent you a log with debug on.

What lockup are you talking about?
Do you have any problems, doesn't TThrottle work properly?

[benDan]

Oh fred ... my mistake!
I have problems with SpeedFan, not Tthrottle.
I got confused about the lockup during startup problem I'm having.
I use both programs together as a super process to protect my PC.

But I do see the activity described.