This is a draft of the paper I will submit to ACG13.
Title: CLOP: Confident Local Optimization for Noisy Black-Box Parameter Tuning
Abstract: Artificial intelligence in games often leads to the problem of parameter tuning. Some heuristics may have coefficients, and they should be tuned to maximize the win rate of the program. A possible approach consists in building local quadratic models of the win rate as a function of program parameters. Many local regression algorithms have already been proposed for this task, but they are usually not robust enough to deal automatically and efficiently with very noisy outputs and non-negative Hessians. The CLOP principle, which stands for Confident Local OPtimization, is a new approach to local regression that overcomes all these problems in a simple and efficient way. It consists in discarding samples whose estimated value is confidently inferior to the mean of all samples. Experiments demonstrate that, when the function to be optimized is smooth, this method outperforms all other tested algorithms.
I've been trying the CLOP software out, and I've written some glue code to support using it with Gomill.
This uses Gomill as the 'twogtp' back-end, and combines the CLOP settings and the engine configuration into a single configuration file (rather than putting the latter in the connection script).
If anyone else wants to use it, it's included as an example script in Gomill 0.7.2, which can be downloaded from http://mjw.woodcraft.me.uk/gomill/ .
[mailto:computer-go-boun...@dvandva.org] On Behalf Of Rémi Coulom Sent: Thursday, September 01, 2011 6:01 AM To: computer...@dvandva.org Subject: [Computer-go] CLOP: Confident Local Optimization for NoisyBlack-Box Parameter Tuning
Hi,
This is a draft of the paper I will submit to ACG13.
Title: CLOP: Confident Local Optimization for Noisy Black-Box Parameter Tuning
Abstract: Artificial intelligence in games often leads to the problem of parameter tuning. Some heuristics may have coefficients, and they should be tuned to maximize the win rate of the program. A possible approach consists in building local quadratic models of the win rate as a function of program parameters. Many local regression algorithms have already been proposed for this task, but they are usually not robust enough to deal automatically and efficiently with very noisy outputs and non-negative Hessians. The CLOP principle, which stands for Confident Local OPtimization, is a new approach to local regression that overcomes all these problems in a simple and efficient way. It consists in discarding samples whose estimated value is confidently inferior to the mean of all samples. Experiments demonstrate that, when the function to be optimized is smooth, this method outperforms all other tested algorithms.
> I am going through the paper, and there is a point where I do not > understand.
> When the weights are recalculated in Algorithm 1, the expression for wk is > exp((qk(x) - mk) / H * sk).
> Should the formula have a square? That is, exp((qk(x) - mk) * (qk(x) - mk) / > H * sk)?
> Thanks, > Brian
No. The idea is that the weight of a sample should be low when it is far below the mean, not when it is far from the mean. That is to say, samples whose value is very low according to the regression get a low weight. But samples whose strength is estimated to be above average keep a full weight of 1 (because of the "min", the weight can never get above 1).
Note BTW that since my previous message I updated the web site of CLOP with some data, screenshots, and a link to the computer-chess forum with more discussions about the algorithm: http://remi.coulom.free.fr/CLOP/
[mailto:computer-go-boun...@dvandva.org] On Behalf Of Rémi Coulom Sent: Saturday, September 10, 2011 11:36 AM To: computer...@dvandva.org Subject: Re: [Computer-go] CLOP: Confident Local Optimization forNoisyBlack-Box Parameter Tuning
On 10 sept. 2011, at 17:20, Brian Sheppard wrote:
> I am going through the paper, and there is a point where I do not > understand.
> When the weights are recalculated in Algorithm 1, the expression for > wk is > exp((qk(x) - mk) / H * sk).
> Should the formula have a square? That is, exp((qk(x) - mk) * (qk(x) - > mk) / H * sk)?
> Thanks, > Brian
No. The idea is that the weight of a sample should be low when it is far below the mean, not when it is far from the mean. That is to say, samples whose value is very low according to the regression get a low weight. But samples whose strength is estimated to be above average keep a full weight of 1 (because of the "min", the weight can never get above 1).
Note BTW that since my previous message I updated the web site of CLOP with some data, screenshots, and a link to the computer-chess forum with more discussions about the algorithm: http://remi.coulom.free.fr/CLOP/
> Yes, that makes sense. You don't want Gaussian there.
> -----Original Message----- > From: computer-go-boun...@dvandva.org > [mailto:computer-go-boun...@dvandva.org] On Behalf Of Rémi Coulom > Sent: Saturday, September 10, 2011 11:36 AM > To: computer...@dvandva.org > Subject: Re: [Computer-go] CLOP: Confident Local Optimization > forNoisyBlack-Box Parameter Tuning
> On 10 sept. 2011, at 17:20, Brian Sheppard wrote:
>> I am going through the paper, and there is a point where I do not >> understand.
>> When the weights are recalculated in Algorithm 1, the expression for >> wk is >> exp((qk(x) - mk) / H * sk).
>> Should the formula have a square? That is, exp((qk(x) - mk) * (qk(x) - >> mk) / H * sk)?
>> Thanks, >> Brian
> No. The idea is that the weight of a sample should be low when it is far > below the mean, not when it is far from the mean. That is to say, samples > whose value is very low according to the regression get a low weight. But > samples whose strength is estimated to be above average keep a full weight > of 1 (because of the "min", the weight can never get above 1).
> Note BTW that since my previous message I updated the web site of CLOP with > some data, screenshots, and a link to the computer-chess forum with more > discussions about the algorithm: > http://remi.coulom.free.fr/CLOP/
Hi, Remi. I have a question about the "burn-in" process for CLOP.
Normally you need a lot of data to make a decent regression function. For example, if you have N arguments in your function, then CLOP (Correlated-All) needs 1 + N * (N+3) / 2 parameters. So if you want 10 observations per parameter, then you need 10 + 5N(N+3) samples.
But even getting *one* sample can be tricky, because the 'logit' for a sample is +INF if the sample wins all of its games, and -INF if the sample loses all of its games. So you need a sample that has some wins and some losses. If the true value of the function is near 0.5, then the average number of trials required to obtain a sample is around 3, which is fine.
But some of the test functions in your paper are very different. For example, the Correlated2 function is nearly 0 for most of the domain [-1,1]^4. When I sample randomly, it takes ~5K samples (that is, ~20K trials) to turn up enough samples to fit a regression line.
I tried initializing my win/loss counters to epsilon instead of zero. But that technique was not robust, because any reasonable epsilon is actually larger than Correlated2 for most of the domain. Consequently, the "reduce the weights" step does not reduce enough weights, and the logistic regression ends up fitting epsilon, rather than Correlated2.
So I cannot get a valid measurement with less than 20K trials before the first regression step. But your paper shows regret curves that start out at 10 trials.
> Hi, Remi. I have a question about the "burn-in" process for CLOP.
> Normally you need a lot of data to make a decent regression function. For > example, if you have N arguments in your function, then CLOP > (Correlated-All) needs 1 + N * (N+3) / 2 parameters. So if you want 10 > observations per parameter, then you need 10 + 5N(N+3) samples.
> But even getting *one* sample can be tricky, because the 'logit' for a > sample is +INF if the sample wins all of its games, and -INF if the sample > loses all of its games. So you need a sample that has some wins and some > losses. If the true value of the function is near 0.5, then the average > number of trials required to obtain a sample is around 3, which is fine.
I deal with +INF/-INF with a prior: the Gaussian prior regularizes the regression, so its tends to remain flat and close to 0.5 when very few samples have been collected.
> But some of the test functions in your paper are very different. For > example, the Correlated2 function is nearly 0 for most of the domain > [-1,1]^4. When I sample randomly, it takes ~5K samples (that is, ~20K > trials) to turn up enough samples to fit a regression line.
I am not sure I understand what you mean. If you use regularization, you can perform regression even with zero samples. Of course, it is very inaccurate. But if you are careful to take confidence intervals into consideration, you can still do statistics with very few samples, and determine with some significance that an area is bad.
> I tried initializing my win/loss counters to epsilon instead of zero. But > that technique was not robust, because any reasonable epsilon is actually > larger than Correlated2 for most of the domain. Consequently, the "reduce > the weights" step does not reduce enough weights, and the logistic > regression ends up fitting epsilon, rather than Correlated2.
> So I cannot get a valid measurement with less than 20K trials before the > first regression step. But your paper shows regret curves that start out at > 10 trials.
> What am I missing?
I am not sure what you are missing.
In the case of Correlated2: In the beginning CLOP will sample uniformly at random (if you run the algorithm in the paper with N=0, then w(x)=1 everywhere). As soon at it find its first win, it will start focusing around that first win. You should be able to easily run CLOP on Correlated2. Just edit DummyExperiment.clop and DummyScript.py. You can also take a look at Gian-Carlo's chess data: it is a bit similar, as most games are lost in the beginning.
One important aspect of CLOP is the use of the confidence interval. It does not matter if the regression is very inaccurate. Even with an inaccurate regression, it can get confident that some areas of the search space are below average, so they should not be sampled.
If you sample uniformly at random until you get an accurate regression, then, yes, it will take forever. Maybe what you are missing is that CLOP does not need an accurate regression at all to already focus its sampling on a promising region.
My implementation is missing the Gaussian prior. That seems to explain all of the issues.
It is especially important that having the prior will focus attention on the region of success. In the case of Correlated2, where only a tiny fraction of the space is non-zero, that will massively reduce the burn-in period.
[mailto:computer-go-boun...@dvandva.org] On Behalf Of Rémi Coulom Sent: Tuesday, October 04, 2011 3:18 PM To: computer...@dvandva.org Subject: Re: [Computer-go] CLOP: Confident Local Optimization forNoisyBlack-Box Parameter Tuning
Hi Brian,
On 4 oct. 2011, at 18:54, Brian Sheppard wrote:
> Hi, Remi. I have a question about the "burn-in" process for CLOP.
> Normally you need a lot of data to make a decent regression function. For > example, if you have N arguments in your function, then CLOP > (Correlated-All) needs 1 + N * (N+3) / 2 parameters. So if you want 10 > observations per parameter, then you need 10 + 5N(N+3) samples.
> But even getting *one* sample can be tricky, because the 'logit' for a > sample is +INF if the sample wins all of its games, and -INF if the sample > loses all of its games. So you need a sample that has some wins and some > losses. If the true value of the function is near 0.5, then the average > number of trials required to obtain a sample is around 3, which is fine.
I deal with +INF/-INF with a prior: the Gaussian prior regularizes the regression, so its tends to remain flat and close to 0.5 when very few samples have been collected.
> But some of the test functions in your paper are very different. For > example, the Correlated2 function is nearly 0 for most of the domain > [-1,1]^4. When I sample randomly, it takes ~5K samples (that is, ~20K > trials) to turn up enough samples to fit a regression line.
I am not sure I understand what you mean. If you use regularization, you can perform regression even with zero samples. Of course, it is very inaccurate. But if you are careful to take confidence intervals into consideration, you can still do statistics with very few samples, and determine with some significance that an area is bad.
> I tried initializing my win/loss counters to epsilon instead of zero. But > that technique was not robust, because any reasonable epsilon is actually > larger than Correlated2 for most of the domain. Consequently, the "reduce > the weights" step does not reduce enough weights, and the logistic > regression ends up fitting epsilon, rather than Correlated2.
> So I cannot get a valid measurement with less than 20K trials before the > first regression step. But your paper shows regret curves that start out at > 10 trials.
> What am I missing?
I am not sure what you are missing.
In the case of Correlated2: In the beginning CLOP will sample uniformly at random (if you run the algorithm in the paper with N=0, then w(x)=1 everywhere). As soon at it find its first win, it will start focusing around that first win. You should be able to easily run CLOP on Correlated2. Just edit DummyExperiment.clop and DummyScript.py. You can also take a look at Gian-Carlo's chess data: it is a bit similar, as most games are lost in the beginning.
One important aspect of CLOP is the use of the confidence interval. It does not matter if the regression is very inaccurate. Even with an inaccurate regression, it can get confident that some areas of the search space are below average, so they should not be sampled.
If you sample uniformly at random until you get an accurate regression, then, yes, it will take forever. Maybe what you are missing is that CLOP does not need an accurate regression at all to already focus its sampling on a promising region.
I've been working with UCT search for other games than Go, and one interesting thing I"ve learned is that the results can change dramatically depending on how the UCT values are manipulated as the tree grows.
Consider the root node; at the beginning of the search it's desirable to sample all the children equally, to be sure each has a fair chance to be noted as winning or losing. However, as the simulations continue, if this egalitarian distribution continues, the simulations from losing nodes dilutes the results (as well as wasting time), so it's necessary to start concentrating on the winning nodes. The exact method of transitioning from broad to narrow focus can have dramatic effect on the results.
On Mon, Oct 24, 2011 at 1:45 PM, Dave Dyer <dd...@real-me.net> wrote:
> I've been working with UCT search for other games than Go, and one interesting thing I"ve learned is that the results can change dramatically depending on how the UCT values are manipulated as the tree grows.
> Consider the root node; at the beginning of the search it's desirable to sample all the children equally, to be sure each has a fair chance to be noted as winning or losing. However, as the simulations continue, if this egalitarian distribution continues, the simulations from losing nodes dilutes the results (as well as wasting time), so it's necessary to start concentrating on the winning nodes. The exact method of transitioning from broad to narrow focus can have dramatic effect on the results.
Doesn't the UCB formula basically encode this behavior? What I think I learned about UCT from experimenting with dimwit is that, for nodes other than the root, you need to reduce exploration so scores are not too polluted by bad moves, but then the principal variation gets ridiculously deep, which means that more exploration is needed. At the root you can make the search explore more, since you don't need to back out a score.
I don't know if go has an equivalent to queen sacrifices in chess, but it would be very hard to make a UCT program that plays something like that correctly: The queen sacrifice would look like a horrible move, with really low score, and if you make the search explore enough to figure out that it's a good move (by finding several correct continuation moves) in a practical amount of time, the score will be horribly polluted in the mean time.
The solution has to be disassociating how much time you spend exploring a move and how much it contributes to the score of its parent. I feel that UCT is great for making up a score out of repeated simulations, but eventually we should end up thinking of it as an evaluation function and using something much closer to minimax for the parts of the tree close to the root. Unfortunately, I don't have any successful experiments to back out this feeling. _______________________________________________ Computer-go mailing list Computer...@dvandva.org http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
>Doesn't the UCB formula basically encode this behavior?
Yes, but the formula is not cast in stone. There are infinite variations that implement the basic concept.
I guess the lesson I wanted to convey is that this formula, or perhaps an algorithm too complicated to be expressed as a simple formula, is part of the space that needs to be explored.
>Doesn't the UCB formula basically encode this behavior?
Yes, but the formula is not cast in stone. There are infinite variations that implement the basic concept.
I guess the lesson I wanted to convey is that this formula, or perhaps an algorithm too complicated to be expressed as a simple formula, is part of the space that needs to be explored.
> >Doesn't the UCB formula basically encode this behavior?
> Yes, but the formula is not cast in stone. There are > infinite variations that implement the basic concept.
Right.
Making MCTS (or UCT) a success in practice consists of 3 % principle-understanding and 97 % fine-tuning. There are myriads of ways to implement Monte Carlo in a favorable way.
Are the optimized values the "Mean" column on the "Max" tab? How does
one get them out? Copy to clipboard only works for a single cell at a
time. I'm on Windows.
On Thu, Sep 1, 2011 at 3:01 AM, Rémi Coulom <Remi.Cou...@free.fr> wrote:
> Hi,
> This is a draft of the paper I will submit to ACG13.
> Title: CLOP: Confident Local Optimization for Noisy Black-Box Parameter Tuning
> Abstract: Artificial intelligence in games often leads to the problem of parameter tuning. Some heuristics may have coefficients, and they should be tuned to maximize the win rate of the program. A possible approach consists in building local quadratic models of the win rate as a function of program parameters. Many local regression algorithms have already been proposed for this task, but they are usually not robust enough to deal automatically and efficiently with very noisy outputs and non-negative Hessians. The CLOP principle, which stands
> for Confident Local OPtimization, is a new approach to local regression that overcomes all these problems in a simple and efficient way. It consists in discarding samples whose estimated value is confidently inferior to the mean of all samples. Experiments demonstrate that, when the function to be optimized is smooth, this method outperforms all other tested algorithms.
> Are the optimized values the "Mean" column on the "Max" tab? How does
> one get them out? Copy to clipboard only works for a single cell at a
> time. I'm on Windows.
> On Thu, Sep 1, 2011 at 3:01 AM, Rémi Coulom <Remi.Cou...@free.fr> wrote:
>> Hi,
>> This is a draft of the paper I will submit to ACG13.
>> Title: CLOP: Confident Local Optimization for Noisy Black-Box Parameter Tuning
>> Abstract: Artificial intelligence in games often leads to the problem of parameter tuning. Some heuristics may have coefficients, and they should be tuned to maximize the win rate of the program. A possible approach consists in building local quadratic models of the win rate as a function of program parameters. Many local regression algorithms have already been proposed for this task, but they are usually not robust enough to deal automatically and efficiently with very noisy outputs and non-negative Hessians. The CLOP principle, which stands
>> for Confident Local OPtimization, is a new approach to local regression that overcomes all these problems in a simple and efficient way. It consists in discarding samples whose estimated value is confidently inferior to the mean of all samples. Experiments demonstrate that, when the function to be optimized is smooth, this method outperforms all other tested algorithms.
> If you can edit the source code and re-compile, you can try replacing: > Qt::ItemIsEnabled
> by
> Qt::ItemIsEnabled | Qt::ItemIsSelectable
> in MainWindow.cpp
> I don't have time to test or prepare a new version, sorry.
> Rémi
> On 8 juil. 2012, at 08:52, Michael Williams wrote:
>> Are the optimized values the "Mean" column on the "Max" tab? How does
>> one get them out? Copy to clipboard only works for a single cell at a
>> time. I'm on Windows.
>> On Thu, Sep 1, 2011 at 3:01 AM, Rémi Coulom <Remi.Cou...@free.fr> wrote:
>>> Hi,
>>> This is a draft of the paper I will submit to ACG13.
>>> Title: CLOP: Confident Local Optimization for Noisy Black-Box Parameter Tuning
>>> Abstract: Artificial intelligence in games often leads to the problem of parameter tuning. Some heuristics may have coefficients, and they should be tuned to maximize the win rate of the program. A possible approach consists in building local quadratic models of the win rate as a function of program parameters. Many local regression algorithms have already been proposed for this task, but they are usually not robust enough to deal automatically and efficiently with very noisy outputs and non-negative Hessians. The CLOP principle, which stands
>>> for Confident Local OPtimization, is a new approach to local regression that overcomes all these problems in a simple and efficient way. It consists in discarding samples whose estimated value is confidently inferior to the mean of all samples. Experiments demonstrate that, when the function to be optimized is smooth, this method outperforms all other tested algorithms.
The CLOP is for noisy black-box parameter tuning. However, your test
functions (LOG, FLAT, POWER, ANGLE, and STEP) are noise-free functions as
shown in Table 1. It is very difficult to prove that CLOP can work very
well on noisy functions.
I suggest that the problem definition f(x) = 1/(1+exp(-r(x))) should be
perturbed with some random variables with a defined zero-mean distribution,
such as Gaussian distribution, uniform distribution, or any others.
Specifically, the problem definitions can be g(x) = 1/(1+exp(-r(x) + n(x)))
where n(x) is an additional noise. The performance of the algorithms can be
evaluated in terms of solution error measure, which is defined as f(x) -
g(x*) where x* is the global optimum of the noise-free function f.
There may exist more appropriate performance evaluation methods than
aforementioned ones for win/loss outcomes. Anyway, in this paper, the
experiment uses noise-free functions as test functions. It cannot prove
anything for noisy optimization.
> This is a draft of the paper I will submit to ACG13.
> Title: CLOP: Confident Local Optimization for Noisy Black-Box Parameter
> Tuning
> Abstract: Artificial intelligence in games often leads to the problem of
> parameter tuning. Some heuristics may have coefficients, and they should be
> tuned to maximize the win rate of the program. A possible approach consists
> in building local quadratic models of the win rate as a function of program
> parameters. Many local regression algorithms have already been proposed for
> this task, but they are usually not robust enough to deal automatically and
> efficiently with very noisy outputs and non-negative Hessians. The CLOP
> principle, which stands
> for Confident Local OPtimization, is a new approach to local regression
> that overcomes all these problems in a simple and efficient way. It
> consists in discarding samples whose estimated value is confidently
> inferior to the mean of all samples. Experiments demonstrate that, when the
> function to be optimized is smooth, this method outperforms all other
> tested algorithms.
> The CLOP is for noisy black-box parameter tuning. However, your test
> functions (LOG, FLAT, POWER, ANGLE, and STEP) are noise-free functions as
> shown in Table 1. It is very difficult to prove that CLOP can work very
> well on noisy functions.
> I suggest that the problem definition f(x) = 1/(1+exp(-r(x))) should be
> perturbed with some random variables with a defined zero-mean distribution,
> such as Gaussian distribution, uniform distribution, or any others.
> Specifically, the problem definitions can be g(x) = 1/(1+exp(-r(x) + n(x)))
> where n(x) is an additional noise. The performance of the algorithms can be
> evaluated in terms of solution error measure, which is defined as f(x) -
> g(x*) where x* is the global optimum of the noise-free function f.
Sorry, the definition of solution error measure should be g(x) - f(x*).
> There may exist more appropriate performance evaluation methods than
> aforementioned ones for win/loss outcomes. Anyway, in this paper, the
> experiment uses noise-free functions as test functions. It cannot prove
> anything for noisy optimization.
>> This is a draft of the paper I will submit to ACG13.
>> Title: CLOP: Confident Local Optimization for Noisy Black-Box Parameter
>> Tuning
>> Abstract: Artificial intelligence in games often leads to the problem of
>> parameter tuning. Some heuristics may have coefficients, and they should be
>> tuned to maximize the win rate of the program. A possible approach consists
>> in building local quadratic models of the win rate as a function of program
>> parameters. Many local regression algorithms have already been proposed for
>> this task, but they are usually not robust enough to deal automatically and
>> efficiently with very noisy outputs and non-negative Hessians. The CLOP
>> principle, which stands
>> for Confident Local OPtimization, is a new approach to local regression
>> that overcomes all these problems in a simple and efficient way. It
>> consists in discarding samples whose estimated value is confidently
>> inferior to the mean of all samples. Experiments demonstrate that, when the
>> function to be optimized is smooth, this method outperforms all other
>> tested algorithms.
> The CLOP is for noisy black-box parameter tuning. However, your test
> functions (LOG, FLAT, POWER, ANGLE, and STEP) are noise-free functions as
> shown in Table 1. It is very difficult to prove that CLOP can work very
> well on noisy functions.
Waow :-) that would be a very strange noisy optimization paper if it was
about testing on noise-free functions.
The functions are certainly not noise-free; what you read (and which is
noise-free...) is their _expected_ values.
>> The CLOP is for noisy black-box parameter tuning. However, your test
>> functions (LOG, FLAT, POWER, ANGLE, and STEP) are noise-free functions as
>> shown in Table 1. It is very difficult to prove that CLOP can work very
>> well on noisy functions.
> Waow :-) that would be a very strange noisy optimization paper if it was
> about testing on noise-free functions.
> The functions are certainly not noise-free; what you read (and which is
> noise-free...) is their _expected_ values.
Thanks for replying me that what I read is their expected values.
Since the functions are not noise-free, they should be defined in terms
of some noise. I really need the definition of the noise for comparison
between CLOP and other optimizers.
I have downloaded the source codes, but I cannot find the codes related to
the noise currently.
It's a Bernoulli noise.
define f (x) = 1/ (1 + e(-r(x)) )
and the objective function at x is 1 with probability f(x).
So the expected value at x is f(x), but the values you get are noisy.
Best regards,
Olivier
2013/3/6 Chin-Chang Yang <chin.chang.y...@gmail.com>
>>> The CLOP is for noisy black-box parameter tuning. However, your test
>>> functions (LOG, FLAT, POWER, ANGLE, and STEP) are noise-free functions as
>>> shown in Table 1. It is very difficult to prove that CLOP can work very
>>> well on noisy functions.
>> Waow :-) that would be a very strange noisy optimization paper if it was
>> about testing on noise-free functions.
>> The functions are certainly not noise-free; what you read (and which is
>> noise-free...) is their _expected_ values.
> Thanks for replying me that what I read is their expected values.
> Since the functions are not noise-free, they should be defined in terms
> of some noise. I really need the definition of the noise for comparison
> between CLOP and other optimizers.
> I have downloaded the source codes, but I cannot find the codes related to
> the noise currently.
It's a Bernoulli noise.
define f (x) = 1/ (1 + e(-r(x)) )
and the objective function at x is 1 with probability f(x).
So the expected value at x is f(x), but the values you get are noisy.
> Since the functions are not noise-free, they should be defined in terms
>> of some noise. I really need the definition of the noise for comparison
>> between CLOP and other optimizers.
>> I have downloaded the source codes, but I cannot find the codes related
>> to the noise currently.
> It's a Bernoulli noise.
> define f (x) = 1/ (1 + e(−r(x)) )
> and the objective function at x is 1 with probability f(x).
> So the expected value at x is f(x), but the values you get are noisy.
> Best regards,
> Olivier
>> Since the functions are not noise-free, they should be defined in terms
>>> of some noise. I really need the definition of the noise for comparison
>>> between CLOP and other optimizers.
>>> I have downloaded the source codes, but I cannot find the codes related
>>> to the noise currently.
