CreditNew based on CPU time

[yoyo]

Hello,

in some projects with mt-apps and where workunits have very different unpredictable runtime, cheaters found ways to
cheat the creditNew to get much more credits than other user.
They e.g. run a workunit which was thought to run on 8 cores on a single core.
As far as I know base for creditNew is run time and they have much more run time, so they get much more credits than others.
After 1 or 2 workunits they create a new host an do it again and again.

Would it be better to base creditNew on cpu time instead of run time?

yoyo

[Richard Haselgrove]

Remember that many projects also supply work for GPUs. The CPU time for a GPU task can be anywhere between 0% and 100% of the run time, depending on the nature of the task, the programming language chosen, and the skills of the programmer.

The change you propose might have implications elsewhere, and needs to the thought through carefully.

--
You received this message because you are subscribed to the Google Groups "boinc_dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to boinc_dev+unsub...@ssl.berkeley.edu.
To view this discussion on the web visit https://groups.google.com/a/ssl.berkeley.edu/d/msgid/boinc_dev/ea073bfe-599a-d77b-ae3d-fe0bef65a247%40mailueberfall.de.

[Steffen Möller]

The Einstein@Home way is my preferred solution: Know the complexity of a WU and assign the reward accordingly. Ignore the CPU time completely, I suggest. Whoever uses the best technology to address the problem should get the most rewards. Or one orders the tasks relative to their (unknown) complexity when distributing them and then interpolates the complexity on a daily bases from the claims made.

This could be a BOF tomorrow.

Best,
Steffen

Am 27.04.21 um 10:04 schrieb 'Richard Haselgrove' via boinc_dev:

To unsubscribe from this group and stop receiving emails from it, send an email to boinc_dev+...@ssl.berkeley.edu.
To view this discussion on the web visit https://groups.google.com/a/ssl.berkeley.edu/d/msgid/boinc_dev/668e812b.2ec36.179125b72d8.Webtop.84%40btinternet.com.

[yoyo]

All your arguments are valid to base credits on known complexity.
But this is not possible in my case, since the used algorithm is a trial factoring algorithm.
This menas it can finish in just a minute or run hours. The runtime is not predictable and depends on many randomly
chosen seeds during runtime of the workunit.
Even calculating it on a second computer leads to total different runtime.

yoyo

Charles Elliott schrieb:
> Basing credit on wallclock time or CPU time does not make much sense.  Consider the following example:
>
>  
>
> I have the following GPUs in my system
>
>    One AMD Radeon RX 580 (circa 2019) completes two Einstein@Home (E@H) work units (WUs) in 16 minutes, or 8 minute per WU.
>
>    One NVidia Tesla K80 (circa 2014) completes two WUs in 56 minutes or 28 minutes per WU.
>
>  
>
> The technology advanced between 2014 and 2019 allowing more than 3 times the work to be accomplished in the same
> (wallclock, CPU, or GPU) time.
>
>  
>
> E@H gives credit per work unit correctly completed.  From its perspective, that makes sense.  It does not and cannot
> care about how much or how long it ties up the user’s system.  Users compete by buying the latest technology.  Everybody
> pays for results, not time consumed.  The returns to labor are its marginal productivity.  In a steel plant blast
> furnace workers are paid less than basic oxygen furnace (BOF) workers, are paid less than rolling mill workers, are paid
> less than finishing mill workers.  Yet all work 8 hours per day.  It is the value of their output that changes.  Pig
> iron from a blast furnace is worth less than a steel slab from a BOF, is worth less than a roll of semi-finished steel
> from a rolling mill, is worth less than galvanized steel cut to the exact width the customer’s stamping machine
> requires.  Likewise the cost of a worker error increases as more processing time is put into the product.  Therefore as
> the product becomes more valuable, workers are paid more to concentrate their attention and encourage fewer errors.
>
>  
>
> Charles Elliott
>
>  
>
> *From:*'Richard Haselgrove' via boinc_dev [mailto:boin...@ssl.berkeley.edu]
> *Sent:* Tuesday, April 27, 2021 4:04 AM
> *To:* BOINC Developers Mailing List <boin...@ssl.berkeley.edu>; yoyo <yo...@mailueberfall.de>
> *Cc:* BOINC Projects <boinc_p...@ssl.berkeley.edu>
> *Subject:* Re: [boinc_dev] CreditNew based on CPU time

>
>  
>
> Remember that many projects also supply work for GPUs. The CPU time for a GPU task can be anywhere between 0% and 100%
> of the run time, depending on the nature of the task, the programming language chosen, and the skills of the programmer.
>
>  
>
> The change you propose might have implications elsewhere, and needs to the thought through carefully.
>
>
>
>
>
> ------ Original Message ------
> From: "yoyo" <yo...@mailueberfall.de <mailto:yo...@mailueberfall.de>>
> To: "BOINC Developers Mailing List" <boin...@ssl.berkeley.edu <mailto:boin...@ssl.berkeley.edu>>
> Cc: "BOINC Projects" <boinc_p...@ssl.berkeley.edu <mailto:boinc_p...@ssl.berkeley.edu>>
> Sent: Tuesday, 27 Apr, 21 At 06:55
> Subject: [boinc_dev] CreditNew based on CPU time
>
> Hello,
>
> in some projects with mt-apps and where workunits have very different unpredictable runtime, cheaters found ways to
> cheat the creditNew to get much more credits than other user.
> They e.g. run a workunit which was thought to run on 8 cores on a single core.
> As far as I know base for creditNew is run time and they have much more run time, so they get much more credits than
> others.
> After 1 or 2 workunits they create a new host an do it again and again.
>
> Would it be better to base creditNew on cpu time instead of run time?
>
> yoyo
>
> --
> You received this message because you are subscribed to the Google Groups "boinc_dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> boinc_dev+...@ssl.berkeley.edu <mailto:boinc_dev+...@ssl.berkeley.edu>.

> To view this discussion on the web visit
> https://groups.google.com/a/ssl.berkeley.edu/d/msgid/boinc_dev/ea073bfe-599a-d77b-ae3d-fe0bef65a247%40mailueberfall.de.
>

> --
> You received this message because you are subscribed to the Google Groups "boinc_dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> boinc_dev+...@ssl.berkeley.edu <mailto:boinc_dev+...@ssl.berkeley.edu>.

> To view this discussion on the web visit

> https://groups.google.com/a/ssl.berkeley.edu/d/msgid/boinc_dev/668e812b.2ec36.179125b72d8.Webtop.84%40btinternet.com
> <https://groups.google.com/a/ssl.berkeley.edu/d/msgid/boinc_dev/668e812b.2ec36.179125b72d8.Webtop.84%40btinternet.com?utm_medium=email&utm_source=footer>.
>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> Virenfrei. www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

[David P. Anderson]

There are several options for how to assign credit; see

https://boinc.berkeley.edu/trac/wiki/CreditOptions

-David

To unsubscribe from this group and stop receiving emails from it, send an email to boinc_dev+...@ssl.berkeley.edu.
To view this discussion on the web visit https://groups.google.com/a/ssl.berkeley.edu/d/msgid/boinc_dev/3f479738-5d2e-7894-2ffa-05bed34ddd32%40mailueberfall.de.

[Raistmer the Sorcerer]

Well, the same cheating can be done in multi-GPU systems currently.

BOINC detects and reports (not right behavior but as it is ) fastest GPU.

Then such cheater could manually enable slower GPU and run task on it. Both run-time and CPU-time will be bigger than expected. So just switching from one to another time will not solve issue (and the difference between GPU devices speed of the same vendor is much bigger than difference between CPU devices).

 

Вторник, 27 апреля 2021, 11:04 +03:00 от 'Richard Haselgrove' via boinc_dev <boin...@ssl.berkeley.edu>:
 

To unsubscribe from this group and stop receiving emails from it, send an email to boinc_dev+...@ssl.berkeley.edu.
To view this discussion on the web visit https://groups.google.com/a/ssl.berkeley.edu/d/msgid/boinc_dev/668e812b.2ec36.179125b72d8.Webtop.84%40btinternet.com.

 

 

--
Raistmer the Sorcerer

 

[Eric Korpela]

It's amazing how confused the whole credit thing still is.

Basing credit on CPU time doesn't mean that two different machines get the same credit per CPU second.

The original BOINC credit system was based on CPU time multiplied by benchmark scores.  Credit_new changed that to be wall time multiplied by a new factor multiplied by benchmark scores, which causes the potential problem with multithreaded apps first mentioned.

As far as I am aware SETI@home is the only project that actually counted OPs (under the original credit system, credit_new wrecked that).  Just about every project overvalues their processing (especially on GPUs).  SETI@home to my knowledge, is the only project that ever valued GPU and CPU OPs equally by not splitting GPU and CPU apps.  Some projects that run multithreaded apps don't even value the multithreaded OPs and single CPU OPs equally as some use (for example) 8 threads, but only keep 5 CPUs busy, but still claim 8 times the credit.  Similarly the "cheat" on the part of volunteers mentioned in the first post.

I had seriously considered doing a modification of credit_new for multithreaded apps which would have been a defacto return to CPU time, by adding an n_cpus field to the host_app_version structure.  The n_cpus would have been calculated as the running median ratio of cpu time to run time, and would have been used in the credit calculation and work requests.

But nobody was interested.  

Cross projects credits are meaningless, and new projects are inflating credits by offering many times the credits that a project that started years ago did.  (Universe@home offers 25x the credit of rosetta or climateprediction).   It would have been nice to have offered a stable credit market, which could have been done by making resource share depend on RAC rather than a fraction of local resources.  Equal resource shares would equalize a projects RAC, so if a project grants 25 times as much credit, it will end up with 1/25th the processing power it would get if it offered on-par credit.   But again, nobody was interested because they had already inflated their own credits.

[Nicolás Alvarez]

You should make it deterministic, include the seed(s) in the workunit
parameters so that running the same workunit twice gives the same
result (and runtime). Resuming a suspended workunit could be difficult
though (you would need to save the RNG state in the checkpoint)...

--
Nicolás

[Nicolás Alvarez]

El mié, 28 de abr. de 2021 a la(s) 19:15, Eric Korpela
(kor...@ssl.berkeley.edu) escribió:

>
>
> It's amazing how confused the whole credit thing still is.
>
> Basing credit on CPU time doesn't mean that two different machines get the same credit per CPU second.
>
> The original BOINC credit system was based on CPU time multiplied by benchmark scores. Credit_new changed that to be wall time multiplied by a new factor multiplied by benchmark scores, which causes the potential problem with multithreaded apps first mentioned.
>
> As far as I am aware SETI@home is the only project that actually counted OPs (under the original credit system, credit_new wrecked that). Just about every project overvalues their processing (especially on GPUs). SETI@home to my knowledge, is the only project that ever valued GPU and CPU OPs equally by not splitting GPU and CPU apps. Some projects that run multithreaded apps don't even value the multithreaded OPs and single CPU OPs equally as some use (for example) 8 threads, but only keep 5 CPUs busy, but still claim 8 times the credit. Similarly the "cheat" on the part of volunteers mentioned in the first post.

If I remember correctly, when SETI@Home switched to counting OPs, it
also accidentally gave 2x the credits than before (and than other
projects), and when this was noticed, the definition of credit was
changed to match
https://boinc.berkeley.edu/w/?title=Computation_credit&diff=prev&oldid=2818

> I had seriously considered doing a modification of credit_new for multithreaded apps which would have been a defacto return to CPU time, by adding an n_cpus field to the host_app_version structure. The n_cpus would have been calculated as the running median ratio of cpu time to run time, and would have been used in the credit calculation and work requests.
>
> But nobody was interested.
>
> Cross projects credits are meaningless, and new projects are inflating credits by offering many times the credits that a project that started years ago did. (Universe@home offers 25x the credit of rosetta or climateprediction). It would have been nice to have offered a stable credit market, which could have been done by making resource share depend on RAC rather than a fraction of local resources. Equal resource shares would equalize a projects RAC, so if a project grants 25 times as much credit, it will end up with 1/25th the processing power it would get if it offered on-par credit. But again, nobody was interested because they had already inflated their own credits.

As I already said years ago, given some specific CPU and GPU, and
given two projects where:
Foo@home's GPU app is 3 times faster than its CPU app
Bar@home's GPU app is 6 times faster than its CPU app

It is *fundamentally* impossible to make a credit system where
#1: credits per day on the CPU are the same for both projects
#2: credits per day on the GPU are the same for both projects
#3: within each project, same WU should give same credits no matter
which processor was used

And we've spent years arguing about how to tweak the system and
measure things differently to make #1 more accurate while ignoring #2.
This has several consequences. If two projects have good "credit
parity" in terms of #1, then the less optimized a project's CPU app is
(and thus the longer it takes and the more credit per WU it gives),
the more credits it will give to GPU users due to #3.

What happens if one day a project finds a way to make the CPU app
faster, without changing the GPU app in any way? If they keep the same
credits per WU, then CPU users get more credits per hour than before
(and than other projects). If they keep the same credits per CPU hour
(less per WU), then GPU users get less credits per hour than before,
and maybe now they would prefer to go to another project. Giving
different credits to CPU and GPU for the same result (ignoring #3) is
nonsense, and easy to cheat: a cheater could run the app on GPU and
pretend it was on CPU.


Cross projects credits are indeed meaningless. Maybe the best thing
that can be done to improve the credit system is to make stats
websites stop showing a "total" adding multiple projects together.
They are in different units, you can't just add them. Or get rid of
the CPID and then they have no choice :)

The "resource share tied to RAC" proposal is also interesting...

--
Nicolás

[Richard Haselgrove]

There's only one place where credits really matter: that's when they are converted back into flops to report our rate of progress to the rest of the scientific community. Without something like the SETI flop-counting procedure, that figure is notably suspect.

I suspect that many of the flops reported on https://boinc.berkeley.edu/chart_list.php are really iops.

--
You received this message because you are subscribed to the Google Groups "boinc_dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to boinc_dev+unsub...@ssl.berkeley.edu.

To view this discussion on the web visit https://groups.google.com/a/ssl.berkeley.edu/d/msgid/boinc_dev/CANPC-tuOJMJ6d-56VDrOGg8i4Y6aKz22v2ZRgyr6B2dVSiOjKA%40mail.gmail.com.

[Raistmer the Sorcerer]

Running the same task twice and getting same results definitely benefit (if not required by) BOINC infrastructure, in its validation part.

How one can determine if host did correct computation if same initial parameter set each time gives new results?

So, initial seed for pseudo-random generator should be included in input parameter set.

 

But this will not help with «credit cost» of task estimation cause different tasks from same project still will take arbitrary time.

 

What does help is to estimate ops number during each individual task run. Via counting number of processing loop and knowing cost of single loop or smth similar.

Then base credit reward on this operations number multiplied on some coefficient (coefficient required only to get some sort of balance with other projects).

That way one will pay for work done, not for time spend.

 

 

 

Четверг, 29 апреля 2021, 1:28 +03:00 от Nicolás Alvarez <nicolas...@gmail.com>:
 

--
You received this message because you are subscribed to the Google Groups "boinc_dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to boinc_dev+...@ssl.berkeley.edu.
To view this discussion on the web visit https://groups.google.com/a/ssl.berkeley.edu/d/msgid/boinc_dev/CANPC-tvupNMNR0PQmc5o-exQ7XhfUmg-YVn%3D8Qo96GPJrzYo9g%40mail.gmail.com.

 

 

--
Raistmer the Sorcerer

 

[Raistmer the Sorcerer]

have been calculated as the running median ratio of cpu time to run time, and would have been used in the credit calculation and work requests. But nobody was interested. Cross projects credits are meaningless, and new projects are inflating credits by offering many times the credits that a project that started years ago did. (Universe@home offers 25x the credit of rosetta or climateprediction). It would have been nice to have offered a stable credit market, which could have been done by making resource share depend on RAC rather than a fraction of local resources. Equal resource shares would equalize a projects RAC, so if a project grants 25 times as much credit, it will end up with 1/25th the processing power it would get if it offered on-par credit. But again, nobody was interested because they had already inflated their own credits.

 

Great idea!

 

As I already said years ago, given some specific CPU and GPU, and
given two projects where:
Foo@home's GPU app is 3 times faster than its CPU app
Bar@home's GPU app is 6 times faster than its CPU app

It is *fundamentally* impossible to make a credit system where
#1: credits per day on the CPU are the same for both projects
#2: credits per day on the GPU are the same for both projects
#3: within each project, same WU should give same credits no matter
which processor was used

Then credits could be «colored», that is, separate credits for separate device classes.

Other way how this issue could be solved:

1. Establish common (for all projects) multiplier between CPU/accelerator pairs BOINC support (this should involve synthetic tests running on CPU and accelerator)
2. Count /estimate flops for particular task (tasks can have quite different flops from same project if they involve stochastic algorithms).
3. Equalize _wallclock_ run time per FLOP between projects for _CPU_apps  (this step could correct errors in FLOPS estimation by projects and account for non compute-oriented projects)
4. Base credit awarding on counted flops and common for all projects CPU/accelerator multiplier.
   
   Such approach also will encourage proper GPU/accelerator apps optimization for particular project. Cause providing GPU app that hugely underuses GPU will be penalized by small credit awards for such app.
   
   Of course this will not fully cover different state of CPU optimization for particular project but modern optimizing CPU compilers make difference in this field not so huge. Algorithm-level optimization still can shake balance though.
    

To unsubscribe from this group and stop receiving emails from it, send an email to boinc_dev+...@ssl.berkeley.edu.
To view this discussion on the web visit https://groups.google.com/a/ssl.berkeley.edu/d/msgid/boinc_dev/479c57fa.323fc.1791c32935e.Webtop.95%40btinternet.com.

 

 

--
Raistmer the Sorcerer