Boinctasks History Analyzer & Project performance

[Joseph Stateson]

Fred has graciously allowed my program BThistory to be promoted as an add-on to Boinctasks.
This post is frequently updated, please use your browser refresh feature to get changes and make
sure you have the latest build date as shown on main menu under the Open History button.
The location of the executables are listed  at the end of this post.   I put a zip file with 32 and 64 bit executables here
I do not have an install package for windows so you will have to answer a lot of "are you sure" questions from windows and your anti-virus when downloading or executing the app the first time.

BTHistory reads one or more Boinctasks's history files and allows data analysis for elapsed time, throughput and idle time.  If more than one file is opened, then comparisons can be made between different systems.  New or unknown applications are reported, highlighted and can be compared.  The program is written in C# and compiled under Visual Studio 2017.  One can download the executables or build the sources at location GitHub/JStateson.  Additional utility programs are included in the VS2017 solution and are explained below.  This app uses dotnet framework 4.6.1 but I assume anything later could also be installed to allow it to run.

1.   BTHistory main form and throughput analysis.


The history file "z400-4-s9x00" has been opened and there are 2 project names available. One has been selected (milkyway) and that project has only one app running on this system.  The number 5517 indicates the number of results.  A figure over 20,000 may take a while to load but that can be restricted to only ore recent results.  The throughput filter is selected and the last hours of data was fetched (no problems are shown in info window).  The continuity check was done indicating at most 1.48 minutes between results.  Knowing that this system had 4 boards, with 5 concurrent WUs and and average credit was 224 points, the results show about 5 credits per second per device.  The "1" adjacent to the App Name box indicates there is only 1 app associated with Milkyway, at least on this particular system.

2.   BTAhistory and Elapsed Time



Elapsed time is in minutes, but the plot parameters were changed to show the effective ET since four tasks were being run concurrently each GPU.  If there were 5 GPUs that mean of 50 seconds (as shown above) indicates the system with 5 GPUs will produce about (50 / 5 = 10 seconds per work unit)

2.1.   BThistory Idle Time


The Idle Time analysis is useful to show when projects run out of data or, as in the case above, the project fails to provide data until all the data has been processed.   In that case, where the system is waiting for data to arrive, the gap is considered idle time.  The above data shows that about every two hours there is a 6 to 9 minute gap before the server provides data.

2.2.  BThistory Dataset Scatter Plot


This graph can be used to observe how different data sets compare to one another in elapsed time.  Once a project is selected, then all the data sets can be displayed or any subset of them.  The application can also be changed to see how that compares.

2.3.  BThistory Dataset Member


Possible to find what dataset (name) the data belongs to by clicking on or near the point.  Restrictions: under 250 points and only 1 series


3.   BThistory Project Structure


This shows which projects are in the BThistory database or are in the history file

4.   BThistory Select Multiple Systems

If more than one history file is opened, then the BThistory produces a comparison of the different systems.  Typically the files of interest end in CVS and do not have phrase "_long_" in the filename.  In the event that the "long" files do contain data then you should uncheck that exclusion.



4.1   BThistory Comparison


This feature allows comparison of systems across the same project and app.  Currently only SELECT and REPORT are the only defined operations.  You can use this tool to compare, for example, the computation of SETI using NVidia  or AMD boards.  Once the project and app are selected only those systems will be shown.  Select (example above shows sse2) and the system you want to use in the comparison and click "save" to copy the results into notepad.  Then select the avx app and the system that is desired and you can examine the statistics and compare to what you saved in notepad.


4.2  BThistory Scatter Plots

There are two scatter plot options.  The first one compares Elapsed time  between the same applications.  This can show the difference between nVidia, ATI , sse2 or avx for example.



The second scatter plot shows only the selected app but each system is represented.  This shows how a particular app performs on different systems.





5   Other programs peripheral to BThistory

   The BThistory program resides at
https://github.com/JStateson/Gridcoin/tree/master/BTHistoryReader
The executables are stored in a .7z file at this GitHub location or you can obtain from my web site as
listed below

However, the actual VS2017 solution is at GitHub/JStateson/GridCoin which will cause one or more of the following programs to be built in addition to BTHistory.   All programs listed were built using VS2017 C# except for the RPC library, in C, which is only used by BoincRpc.

5.1  HostProjectStats

The is an aspx program that creates a web page.   It can be compiled and executed on your windows system or you can run the program using most browsers by clicking on the link below
http://stateson.net/HostProjectStats

This program obtains elapsed time information from most Boinc Projects and, if you optionally know the load and idle wattage, it will calculate the average credit and wattage used to produce those credits for the system and the boards.  This program requires that the data be available so it may not work with anonymous access unless the project had allowed it for the specified HOSTID.

As shown below, the project Milkyway has been browsed to and hostID 705276 selected.  This is one of the top systems and is listed by default when first bringing up the program.  It may not always be available.



Inexpensive watt meters are available, but you can build your own as shown

Here

https://github.com/JStateson/Gridcoin/blob/master/HostProjectStats/wmeter_wiring.jpg

and assembled here

https://github.com/JStateson/Gridcoin/blob/master/HostProjectStats/wmeter_assembled.jpg

Full load results of a pair of GPUs running 4 concurrent threads are shown here
https://github.com/JStateson/Gridcoin/blob/master/HostProjectStats/e5620_s9000_milkyway_4t_load.png

   HostProjectStats produced the following results based on the above data
https://github.com/JStateson/Gridcoin/blob/master/HostProjectStats/e5620_s9000_milkyway_4t.png


The 32 and 64 bit executables here I do not have an install package so it may not run unless you got the latest visual c run-times and dot net modules and windows 10.   Visual studio 2017 is a free download and has all the stuff needed to build this program.  PM me if a problem.