fast sigmoid aproximations
Kasper H
approximations you may know about. I'd be grateful if you could detail
them here for me.
Thanks.
dan michaels
> I'm curious about any fast sigmoid (logistic & tanh) function
> approximations you may know about. I'd be grateful if you could detail
> them here for me.
What about a piecewise linear function, with a constant slope region
in the middle and saturation levels on the 2 ends? Fast and nonlinear.
.........+----
......../
......./
...---+
- dan michaels
www.oricomtech.com
===========================
Thies Heidecke
"Kasper H" <kaspe...@franklin.net> schrieb im Newsbeitrag
news:grjjnus78n8n3uff3...@4ax.com...
> I'm curious about any fast sigmoid (logistic & tanh) function
> approximations you may know about. I'd be grateful if you could detail
> them here for me.
there are many possible solutions. Which is appropriate for your task
depends on your needs:
Is precision important or just raw speed?
Do you have memory to spare or do you want a "cpu-only"-approximation?
just two ideas:
-Precalculate a look-up-table with the Sigmoid-function values
and perform a linear (or perhaps cubic) interpolation to get
intermediate values.
Pro: very fast
Con: occupies memory
-Get a good interpolation-polynom that fits your performance- and
precision-needs and calculate the function-values this way.
just for fun i let my GA calculate the coefficients of the following
polynom. It's designed for the range [-4.5, 4.5] and has a maximum
deviation of 0.002 from the exact values in this range. For values
between 4.5 and 9 i evolved another poly of lower order. for higher
values the error is so low that the function returns just 1 or -1
respectively.
some c-code:
float fast_sigmoid(float x) {
float absx = (float)fabs(x);
float xx;
if(absx>8.713655f) {
if(x>0) return 1.0f;
else return 0.0f;
} else {
xx = x*x;
if(absx>4.5f) {
if(x>0) return (float)(((3.2e-7*xx-8.544e-5)*xx+9.99869e-3)*x+0.953157);
else return (float)(((3.2e-7*xx-8.544e-5)*xx+9.99869e-3)*x+0.046843);
} else {
return
(float)((((((-5e-8*xx+3.6e-6)*xx-1.0621e-4)*xx+1.75410e-3)*xx-0.02045660)*xx+0
.24990936)*x+0.499985);
}
}
}
The values returned by this code have a maximum error of ~0.0026 at -4.5 and
4.5
I hope this was helpful to you.
> Thanks.
No problem
Greetings,
Thies
Kasper H
TrolletAtskynetDOTbe
I've been puzzled by the idea that it may look like a transistor's
transfer function - the most complex part of a neuron implemented with
one transistor, calucating output in one cycle?
... but you need some stuff to try it :-)
Atle
Jonathan Dinerstein
I discovered a fast activation function that I like a lot. It's very
simple, fast, and doesn't require a lookup table. I only found one
reference to it on the internet, where it was called fast-sigmoid or
fsigmoid, but here it is:
x / (1 + abs(x))
I've used it in a backprop neural net, and it works great. The only
challenge is the derivative, which is a little complex, but I computed
it using Mathematica.
Jonathan Dinerstein
jondin...@yahoo.com
Alexis
> approximations you may know about. I'd be grateful if you could detail
> them here for me.
>
Thats funny, I just made some tests about this yesterday. I basically
used a simple look-up table instead of computing the sigmoid each
time, and its exactly 100 times faster!
Heres the C++ code (sigmoid is defined a C++ function object)
#include <iostream>
#include <sstream>
#include <memory>
#include <vector>
#include <list>
#include <cmath>
#include <memory>
#include <boost/timer.hpp>
class Sigmoid
{
public:
Sigmoid(int start, int end, int precision);
~Sigmoid() {};
float operator() (float value) const;
float sigmoid(float value){
return (1.0f / (1.0f + exp(-value)));
}
private:
// std::vector<float> array;
float* array;
long precision_, start_, end_;
};
Sigmoid::Sigmoid(int start, int end, int precision):
// array((end-start)*precision),
precision_(precision),
start_(start),
end_(end)
{
array = new float[(end-start)*precision];
for(long i=0 ; i< (end-start)*precision ; ++i){
float value = (static_cast<float>(i)/
static_cast<float>(precision))-
static_cast<float>(end_);
// std::cout << value << " ";
array[i] = (1.0f / (1.0f + exp(-value)));
// std::cout << i << " " << array[i] << std::endl;
}
}
// float Sigmoid::operator() (float value) const
// {
// float ret=0.0f;
// if(value > static_cast<float>(end_)){
// ret = 1.0f;
// }else if(value < static_cast<float>(start_)){
// ret = 0.0f;
// }else{
// ret = array[static_cast<long>(((value +
// static_cast<float>(end_))*
// static_cast<float>(precision_)))];
// }
// return ret;
// }
float Sigmoid::operator() (float value) const
{
float ret=0.0f;
if(value > static_cast<float>(end_)){
ret = 1.0f;
}else if(value < static_cast<float>(start_)){
ret = 0.0f;
}else{
ret = (1.0f / (1.0f + exp(-value)));
}
return ret;
// return (1.0f / (1.0f + exp(-value)));
}
using boost::timer;
int main(void)
{
timer full;
double tstart=0.0, tstart2=0.0;
int start=-15, end=15, precision=1000000;
Sigmoid activation(start, end, precision);
// return 0;
timer loop;
for(float value=start-10 ; value<end+10 ; value +=
1.0f/(float)precision){
// std :: cout << activation(value) << " ";
// std::cout << activation.sigmoid(value) << " " << value <<
std::endl;
activation(value);
}
std::cout << "loop=" << loop.elapsed() << std::endl;
std::cout << "run=" << full.elapsed() << std::endl;
}
SKarrila
got further refs.
Seppo
"Jonathan Dinerstein" <jondin...@yahoo.com> wrote in message
news:bf0fe52f.02092...@posting.google.com...
a
http://www.dontveter.com/bpr/activate.html
It has this act.fn. plus a few other fast ones.
It states the deriv.fn. of
y = x / (1 + |x|)
is
1/((1+|x|)*(1+|x|))
Marc
GOGAR
to all the multiplications involved with propagating the values through the
network?
Chris
"Alexis" <nur...@ematic.com> wrote in message
news:2f0c7ab9.02092...@posting.google.com...
Alexis
biased in any way. Therefore, any potential improvement is a step
torwards the final goal. I havent had time to benchmark the speed
gain, but I will.
2) I dont totally agree with you. I dont think the sigmoid function is
not significant compared to the rest of processing. Its probably not
the main bottleneck, but you can deny its not costly to make a call to
the exp function. And as I said, any speed gain is welcome.
3) you should read again the subject of this thread.
Alexis
"GOGAR" <ange...@kabelfoon.nl> wrote in message news:<an45ej$g3t$1...@news.kabelfoon.nl>...
