There is one thing I wanted to ask a long time ago... observed on both Win XP SP3 and Win 7. When TTh is actively throttling BOINC applications, the process causes a lot of page faults. E.g. my currently not even 10 days running TTh has already had 1770 mil. page faults (2100/sec over the whole time, but has been throttling just a part of the time), I can often see up to 200 000 page faults during a second or two, followed by a silent pause.
I believe these are not the "hard page faults", caused by the swapping to/from HDD because of the lack of RAM, but "soft page faults", possibly caused by accessing memory protection boundaries, some types of system calls, etc. Hard to tell without any possibility to see symbolic call stacks. But the net effect is the application uses maybe 3% of its CPU life in user mode and 97% in the system calls. 1-2 CPU hours daily.
Any way to find out, what exactly is causing this? Could solving this allow lowering TTh's CPU footprint?
BTW, a long time ago I've already reported the Process Explorer's and Process Hacker's dev teams, that their applications have issues with such pagefaulting programs, because when their counters overflow 31 bits, the values are either displayed negative, or incorrectly positive, or even the broken displaying routines get blocked ;)
Quote from: Pepo on June 07, 2010, 06:01:13 PM
There is one thing I wanted to ask a long time ago... observed on both Win XP SP3 and Win 7. When TTh is actively throttling BOINC applications, the process causes a lot of page faults. E.g. my currently not even 10 days running TTh has already had 1770 mil. page faults (2100/sec over the whole time, but has been throttling just a part of the time), I can often see up to 200 000 page faults during a second or two, followed by a silent pause.
I believe these are not the "hard page faults", caused by the swapping to/from HDD because of the lack of RAM, but "soft page faults", possibly caused by accessing memory protection boundaries, some types of system calls, etc. Hard to tell without any possibility to see symbolic call stacks. But the net effect is the application uses maybe 3% of its CPU life in user mode and 97% in the system calls. 1-2 CPU hours daily.
Any way to find out, what exactly is causing this? Could solving this allow lowering TTh's CPU footprint?
BTW, a long time ago I've already reported the Process Explorer's and Process Hacker's dev teams, that their applications have issues with such pagefaulting programs, because when their counters overflow 31 bits, the values are either displayed negative, or incorrectly positive, or even the broken displaying routines get blocked ;)
1) TThrottle version?
2) Is the system low on memory?
3) Do you see these page faults on the TThrottle process? How do you check this, what program do you use.
4) TThrottle uses almost no memory except for the graphics. You may switch them off in the 4th tab.
5) I checked this on several machines, but I hardly see any page faulting occur.
http://en.wikipedia.org/wiki/Page_fault (http://en.wikipedia.org/wiki/Page_fault)
Quote from: fred on June 07, 2010, 08:19:10 PM
Quote from: Pepo on June 07, 2010, 06:01:13 PM
...observed on both Win XP SP3 and Win 7. When TTh is actively throttling BOINC applications, the process causes a lot of page faults, I can often see up to 200 000 page faults during a second or two, followed by a silent pause.
I believe these are not the "hard page faults", caused by the swapping to/from HDD because of the lack of RAM... But the net effect is the application uses maybe 3% of its CPU life in user mode and 97% in the system calls. 1-2 CPU hours daily.
BTW, a long time ago I've already reported the Process Explorer's and Process Hacker's dev teams, that their applications have issues with such pagefaulting programs...
1) TThrottle version?
2) Is the system low on memory?
3) Do you see these page faults on the TThrottle process? How do you check this, what program do you use.
4) TThrottle uses almost no memory except for the graphics. You may switch them off in the 4th tab.
5) I checked this on several machines, but I hardly see any page faulting occur.
1. Ì believe any I've used during last 1/2 year, but I've checked just occasionaly.↲
2. 2 or 4 Gig , plenty available. Independent.
3. With mentioned P.Explorer or P.Hacker., on main window or TTh's process' properties dialog window. TaskMgr should show the same. I'll try also Resource Monitor. IIRC I've also played with various counters' graphs in PerfMon.
4. Not low on memory , I understand this.
5. I'll have to post some graphs and screenshots tomorrow.
Quote from: Pepo on June 07, 2010, 06:01:13 PM
But the net effect is the application uses maybe 3% of its CPU life in user mode and 97% in the system calls. 1-2 CPU hours daily.
The pattern is, a couple of seconds of silence, followed by a small red spike, taking a half to one and half second.
Quote from: Pepo on June 07, 2010, 10:46:34 PM
Quote from: Pepo on June 07, 2010, 06:01:13 PM
But the net effect is the application uses maybe 3% of its CPU life in user mode and 97% in the system calls. 1-2 CPU hours daily.
The pattern is, a couple of seconds of silence, followed by a small red spike, taking a half to one and half second.
Even TThrottle has to work sometimes. Tab Extra Rebuild list after .. seconds.
But I will take a look at what is running. The cpu usage should be lower than that.
Quote from: Pepo on June 07, 2010, 10:34:21 PM
Quote from: fred on June 07, 2010, 08:19:10 PM
2) Is the system low on memory?
4) TThrottle uses almost no memory [...]
5) I checked this on several machines, but I hardly see any page faulting occur.
2. 2 or 4 Gig , plenty available. Independent.
4. Not low on memory , I understand this.
5. I'll have to post some graphs and screenshots tomorrow.
5. The promised screenshots (already 2 days old, I've since lowered the CPU frequency a bit, thus the throttling is not necessary and pagefaults happen just at the "list rebuild" rate (set to 60 seconds)).
(http://i488.photobucket.com/albums/rr244/peposobr/TThrottle%20BoincTasks/TTh5ssampling-listrebuildsthrottlin.png)
TTh, 5.0s sampling - list rebuilds+throttling at 2.15GHz
(the term "sampling" under the screenshots means the Process Explorer's data sampling period)
(http://i488.photobucket.com/albums/rr244/peposobr/TThrottle%20BoincTasks/TTh05ssampling-listrebuildsat175GHz.png)
TTh, 0.5s sampling - list rebuilds at 1.75GHz, later list rebuilds+throttling at 2.15GHz
(http://i488.photobucket.com/albums/rr244/peposobr/TThrottle%20BoincTasks/TTh05ssampling-pagefaultssyscallspe.png)
TTh, 0.5s sampling - page faults + sys calls peak - list rebuilds+throttling at 2.15GHz
BTW, the Process Explorer's PF counter is again "over" and counting down, now at 1 002 307 988 ;D
At least the Task Manager is displaying the count correctly (I'm wondering what will happen at the 32-bit boundary):
(http://i488.photobucket.com/albums/rr244/peposobr/TThrottle%20BoincTasks/TThpagefaultsinTaskMgr.png)
TTh's page faults in Task Manager
2.+4. I've checked the Resources Monitor, and as I'm looking at it now, it displayed 2-4
Hard Faults/sec for maybe 5 aplications - as an average for the last 1 minute, then the average slowly faded out and there is currently 0 H-PF/s. Well, now some mcupdate.exe (MediaCenterUpdate?) has 45/s and again it slowly fades out.)
TThrotle simply has no (0) Hard Page Faults even when actively throttling, thus it has nothing to do with lack of RAM vs. HDD use.
(http://i488.photobucket.com/albums/rr244/peposobr/TThrottle%20BoincTasks/TTh0HardpagefaultsinRsrcMon.png)
TTh having 0 Hard page faults in Resource Monitor.png
So TThrottle makes 200000 pagefaults a second for the one second every 60 seconds. The rest of the time no pagefaults even when throttling.
And you tested this on Win 7 64.
Nearly exactly.
Quote from: fred on June 10, 2010, 07:58:56 PM
So TThrottle makes 200000 pagefaults a second for the one second every 60 seconds.
Yes. Sometimes more, sometimes less. I guess it takes a bit longer than one second, maybe with Process Hacker I could obtain even finer sampling to see the nice red gaussian curve :P
QuoteThe rest of the time no pagefaults even when throttling.
Not exactly. No pagefaults inbetween when not throttling (see the left half of the second image (http://i488.photobucket.com/albums/rr244/peposobr/TThrottle%20BoincTasks/TTh05ssampling-listrebuildsat175GHz.png)), but also pagefaulting when throttling is enabled (see the smaller spikes at the right half of the second image (http://i488.photobucket.com/albums/rr244/peposobr/TThrottle%20BoincTasks/TTh05ssampling-listrebuildsat175GHz.png), or on the third image (http://i488.photobucket.com/albums/rr244/peposobr/TThrottle%20BoincTasks/TTh05ssampling-pagefaultssyscallspe.png)).
Maybe these are the moments, when processes are paused or resumed? No, just checked on my slow old AMD Duron (err, already some Athlon) - the moments when BOINC processes are being paused or resumed (I can observe it with Process Explorer, paused processes are painted/highlighted with grey color) do not seem to correlate with page fault and CPU usage spikes.
And, as the ProcExp on WinXP displays the context switches (rather than CPU cycles on Vista and 7), I can se that now TTh consumes around 25-34-40 CSW per half second even during processes being throttled, but it rises somewhere to 80 CSW during these CPU (and PF) spikes.
Quote
And you tested this on Win 7 64.
Not only, I was observing it for a long time on different machines with Win XP SP3 x32 - Core2D T7200 and AMD Athlon.
Quote from: Pepo on June 10, 2010, 09:37:13 PM
Quote from: fred on June 10, 2010, 07:58:56 PM
So TThrottle makes 200000 pagefaults a second for the one second every 60 seconds.
Yes. Sometimes more, sometimes less. I guess it takes a bit longer than one second, maybe with Process Hacker I could obtain even finer sampling to see the nice red gaussian curve :P
It goes down to 250 msec, and displays "huge curves" (proc list), narrow needle spikes (each couple of seconds) and sometimes even tiny spikes inbetween.
(http://i488.photobucket.com/albums/rr244/peposobr/TThrottle%20BoincTasks/TTh025ssampling-listrebuildsthrottl.png)
TTh 0.25s sampling - list rebuilds+throttling at 2.15GHz
Quote from: Pepo on June 10, 2010, 09:37:13 PM
[...] the moments when BOINC processes are being paused or resumed (I can observe it with Process Explorer, paused processes are painted/highlighted with grey color) do not seem to correlate with page fault and CPU usage spikes.
I may be pretty wrong with this assumption. TTh may (un)throttle the processes at a much higher rate than the Process Explorer's sampling rate. [...] With Process Hacker I've seen the throttling periods (it highlights single threads) being shorter or longer fractions of a second.
There was one more interesting feature in Process Hacker - it displays also the
Pagefile Usage in the process' memory details, and the (otherwise steady) values do change a bit during the spikes:
(http://i488.photobucket.com/albums/rr244/peposobr/TThrottle%20BoincTasks/TThmemorydetails-silent.png)
TTh memory details - silent
(http://i488.photobucket.com/albums/rr244/peposobr/TThrottle%20BoincTasks/TThmemorydetails-busy1.png) (http://i488.photobucket.com/albums/rr244/peposobr/TThrottle%20BoincTasks/TThmemorydetails-busy2.png) (http://i488.photobucket.com/albums/rr244/peposobr/TThrottle%20BoincTasks/TThmemorydetails-busy3.png)
TTh memory details - busy states
The various values always seemed to return to the "steady state" afterwards:
(http://i488.photobucket.com/albums/rr244/peposobr/TThrottle%20BoincTasks/TThmemorydetails-silent2.png)
So, the Page File seems indeed to be somehow used, but not because of usual VM paging. Maybe some shared memory block, backed through the Page File? Maybe accessing some feature like this might issue interrupts or page faults or else...??
Quote from: Pepo on June 07, 2010, 06:01:13 PM
BTW, a long time ago I've already reported the Process Explorer's and Process Hacker's dev teams, that their applications have issues with such pagefaulting programs, because when their counters overflow 31 bits, the values are either displayed negative, or incorrectly positive, or even the broken displaying routines get blocked ;)
As you can see, the total Page Fault count already overflowed the 32 bit border and while being in sane ranges, the Process Hacker is fortunately not displaying a garbaged process' memory block details anymore 8) ("just" the displayed system's total Page Faults Transition counter is displaying exactly the same damaged value like "18 446 744 07..")
The problem is I don't see any of these things.
I am testing it now on a notebook with W7 64 and see no pagefaults at all.
TThrottle is throttling close to the max, all the time on both CPU and GPU.
It may well be that one of these hacker tools is messing things up.
If there is anybody else with these kind of problems, please let me know.
And again pagefaults are no faults, just parts that are missing in active memory.
This is normally a OS problem and TThrottle has no influence on this.
Maybe another program is using a lot of memory and forces part or the memory TThrottle is using to disk.
Quote from: fred on June 11, 2010, 03:52:46 PM
The problem is I don't see any of these things.
I am testing it now on a notebook with W7 64 and see no pagefaults at all.
TThrottle is throttling close to the max, all the time on both CPU and GPU.
Weird... I'm not painting it ::)
QuoteIt may well be that one of these hacker tools is messing things up.
Could be. Just if I knew what to deinstall... I'll try once to chek it if I'll get some clean system under my hands.
QuoteMaybe another program is using a lot of memory and forces part or the memory TThrottle is using to disk.
I believe I've proven it has nothing to do with the usual swapping from/to page file because of lack of RAM...
Quote from: Pepo on June 11, 2010, 07:36:09 PM
Quote from: fred on June 11, 2010, 03:52:46 PM
The problem is I don't see any of these things.
I am testing it now on a notebook with W7 64 and see no pagefaults at all.
TThrottle is throttling close to the max, all the time on both CPU and GPU.
Weird... I'm not painting it ::)
QuoteIt may well be that one of these hacker tools is messing things up.
Could be. Just if I knew what to deinstall... I'll try once to chek it if I'll get some clean system under my hands.
QuoteMaybe another program is using a lot of memory and forces part or the memory TThrottle is using to disk.
I believe I've proven it has nothing to do with the usual swapping from/to page file because of lack of RAM...
The hacker program installs a driver and that should be the only part to worry about. Drivers are hard to test and debug, easy to make a mistake.
And yes... but any page fault other than a protection error is normally nothing to worry about.
But I keep it in mind and if I get a bright idea. ;D But whatever works without any apparent errors.
Hi fred!
Could this be related to the problems some people have with lockup during startup?
I see this on my PC using Project Lasso and Windows task manager.
values jump form 0 to 23k and back.
I just sent you a log with debug on.
Quote from: benDan on June 14, 2010, 05:33:53 PM
Hi fred!
Could this be related to the problems some people have with lockup during startup?
I see this on my PC using Project Lasso and Windows task manager.
values jump form 0 to 23k and back.
I just sent you a log with debug on.
What lockup are you talking about?
Do you have any problems, doesn't TThrottle work properly?
Oh fred ... my mistake!
I have problems with SpeedFan, not Tthrottle.
I got confused about the lockup during startup problem I'm having.
I use both programs together as a super process to protect my PC.
But I do see the activity described.
Quote from: fred on August 16, 2010, 07:03:21 AM
Changed: Drastically reduced TThrottle overhead, on some systems with NVIDIA cards and WIN 7.
In what ways actually?
(I had to nitpick, sorry.)Probably it does not concern my Win 7 notebook with nVidia G105M. I've made some screenshots of Process Explorer's properties window of TTh 1.90 after running 2 weeks (occasionally slightly throttling apps to some 5-15%), then installed a newer 2.10.1 and checked its consumption after 2 days:
- Kernel time usage 2:30 hours (spread nearly evenly to 2 threads), in line with 18 hours after 2 weeks,
- 26 000 000 000 000 CPU cycles, in line with 199 000 000 000 000 CPU cycles after 2 weeks, approximates to 0.15GHz? Seems fine to me, some 1-2% of a 4 core 1.8GHz CPU?
- Number of page faults just overflowed 31 bits a hour ago and continues,
- I/O Other is 3 434 000, is much less than 137 068 000 after 2 weeks, no idea whether this matters?
- I/O Bytes History graph shows steady 6 KB/s, my previous screenshots were showing 1/5th of this, again no idea whether this matters?
(I've still to check my old WinXP machine with Duron and ATI dinosaurs, and one more older dual-core + nVidia notebook.)
(Fred, enjoy your bike trip!)
Quote from: Pepo on August 27, 2010, 11:35:36 AM
In what ways actually? (I had to nitpick, sorry.)
(Fred, enjoy your bike trip!)
This only affects some cards on some OS. For some reason the driver interface consumed a lot of CPU time. Probably Win 7 with high end (multiple) Graphic cards.
Others OS or cards may not notice anything at all.
I'm hiking, almost the same but no wheels. ;D
Quote from: fred on August 27, 2010, 01:25:22 PM
Quote from: Pepo on August 27, 2010, 11:35:36 AM
(Fred, enjoy your bike trip!)
I'm hiking, almost the same but no wheels. ;D
:-[ I've probably misread the first character :-[
anyway...
Finally I could return to my "loving" thread with some new relevant information, which I found interesting (albeit posibly not useful ;D).
While checking the intensively-pagefaulting new SETI v7 app, I've noticed that the the Process Hacker (http://sourceforge.net/projects/processhacker/) has a tool named WS Watch, which is able to "monitor page faults that occur in a process" and break the PFs and list their counts according to functions, where they happen. It helped there (http://setiweb.ssl.berkeley.edu/beta//forum_thread.php?id=1831&nowrap=true#40780) to confirm, that the PFs are indeed happening in the used FFTW library during frequent (de)allocations of memory (no idea using which allocation function), which I believe causes the memory area always being get off the process' working set (to the Standby List?), and taking it back to process' working set has always to cause some (at least soft) page faults.
So I've got the idea to check the TTh's pagefaults too. Tested with 3.51, started some 2 weeks ago, the monitoring window was opened some 2 days ago and counted Page Faults for some 500 function calls, of which probably the following ones are somehow relevant (or I've considered them being of some interest ::)):
8 241 420, ~ ntoskrnl.exe!TmCurrentTransaction+0x159
1 723 150, + ntoskrnl.exe!ExGetPreviousMode+0x216
1 478 681, + win32k.sys!EngFntCacheLookUp+0x8e90
1 438 266, ~ kernel32.dll!Thread32Next+0x79
1 434 827, ~ kernel32.dll!Thread32Next+0x9a
799 873, ~ kernel32.dll!Process32NextW+0x7c
797 855, ~ kernel32.dll!Process32NextW+0xa0
721 538, + DpFbview.dll!Ordinal9+0xc9743
477 129, ~ ntdll.dll!RtlMoveMemory+0x250
218 769, + win32k.sys!memset+0x80
130 060, ~ kernel32.dll!CreateToolhelp32Snapshot+0x279
(other 14 calls with counts ranging 70 000 - 10 000)
7 614, ~ kernel32.dll!CreateToolhelp32Snapshot+0x2d6
6 713, ~ kernel32.dll!Thread32First+0x77
6 713, ~ kernel32.dll!Thread32First+0xb0
5 931, ~ kernel32.dll!Process32FirstW+0x7a
5 931, ~ kernel32.dll!Process32FirstW+0x98
5 931, ~ kernel32.dll!CreateToolhelp32Snapshot+0x28f
(other 13 calls with counts ranging 5 000 - 100)
(other 460 calls with counts ranging 50 - 1)
Function calls marked with (~) were incrementing their PF counts just occasionally (but all together), once in a couple of tens of seconds - apparently when TTh was enumerating existing processes and their threads (using kernel32.dll's "CreateToolhelp32Snapshot" + "Process/Thread32First/W" + "Process/Thread32Next/W").
Function calls marked with (+) are incrementing their PF counts continuously (the counters are being incremented each second).
However I was no sure anymore with what is causing occasional calls to "ntoskrnl.exe!TmCurrentTransaction" (increments in big jumps), whether it could be possible to reduce calls to "win32k.sys!EngFntCacheLookUp", "ntoskrnl.exe!ExGetPreviousMode" (happens probably in kernel mode, to check whether a kernel/user mode switch for a call has happened?). ("DpFbview.dll!Ordinal9" seems to be a driver from DigitalPersona, Inc. - belongs somehow to the fingerprint sensor on my notebook - WTH is it listed there? Is it being injected in each process??).
Quote from: Pepo on May 12, 2011, 03:11:45 PM
Finally I could return to my "loving" thread with some new relevant information, which I found interesting (albeit posibly not useful ;D).
Nothing wrong with a pagefault, it's just not in real memory.
The only option is to limit the amount of programs that are running and increasing the amount of available memory.
On my 6 & 12 G machine I get 0 pagefaults.
As I checked Win 7 needs about 2.5 Gbytes, so a system with about 2 G ... doesn't have enough memory.
Quote from: fred on May 13, 2011, 10:47:07 AM
Quote from: Pepo on May 12, 2011, 03:11:45 PM
Finally I could return to my "loving" thread with some new relevant information, which I found interesting (albeit posibly not useful ;D).
Nothing wrong with a pagefault, it's just not in real memory.
The only option is to limit the amount of programs that are running and increasing the amount of available memory.
On my 6 & 12 G machine I get 0 pagefaults.
As I checked Win 7 needs about 2.5 Gbytes, so a system with about 2 G ... doesn't have enough memory.
If my 4G Win7 machines are in a steady state, there is also no memory paging... If my WinXP machine with AMD Duron and 1.25 GB RAM is freshly booted, there is 1 GB RAM free and absolutely nothing HDD-related happens. Yet some applications on them continuously produce soft page faults. Some no at all, some less, some more...
No, and again no, this all is not about lack of or enough RAM (I've stated it multiple times before - it is not about hard page faults), it is rather about code calls' transitions between user/kernel modes and modifying (security + ownership) properties for RAM areas (to be allocated / deallocated).