Softplus Activation

[Rhett Espinoza]

Dueto the log() call I fear that there might be issues with computational stability.

Is there a computationally more stable way of computing log-softplus?

I do not want to change my parameterization.

Thank you!

On the other hand, if your argument to log-softplus can reasonably

become that negative, then your suggested version does make

sense, except that you should use a 'breakpoint" (x > -5) that is

significantly more negative than -5 (perhaps something like -40).

As it stands, your breakpoint of -5 does more harm than good,

because in this range x is an imperfect approximation to log-softplus.

What I did not mention (my bad) is that I want to apply gradient backprop through this computation.

This changes things a lot: If setting the breakpoint too low, the error in the gradient can be 7 orders of magnitude bigger than the error in the log-softplus output.

The following code is an extension of yours.

A simple option is to use numpy.logaddexp with 0 as the first argument. That is, softplus(x) (or log1pexp(x)) is equivalent to np.logaddexp(0, x). It will avoid the overflow problem that the naive implementation has. For example,

In the paper, Dombrowski et al. (2019) show that attribution (explanation) maps can be arbitrarily manipulated. They argue that this is because the output manifold of the ReLU neural network has a large curvature, and it causes gradients wrt. to the input to be highly unstable when the input is slightly perturbed.

They show that one can prevent such manipulations by replacing ReLU with the softplus activation. Based on the theorem 2, they argue and empirically show that doing so has the same effect (attribution maps) as SmoothGrad, in which the attribution map is averaged from several maps of the input perturbed by some noise.

It is often mentioned that rectified linear units (ReLU) have superseded softplus units because they are linear and faster to compute.

Does softplus it still have the advantage of inducing sparsity or is that restricted to the ReLU?

The reason I ask is it I wonder about negative consequences of the zero slope of the ReLU. Doesn't this property "trap" units at zero where it might be beneficial to give them the possibility of reactivation?

The use of softplus is generally discouraged. ... one might expect it to have advantage over the rectifier due to being differentiable everywhere or due to saturating less completely, but empirically it does not.

ReLUs can indeed be permanently switched off, particularly under high learning rates. This is a motivation behind leaky ReLU, and ELU activations, both of which have non-zero gradient almost everywhere.

The main reason ReLU works better than Softplus is that for ReLU we have the idea of sparsity in the model. This means that some of the neurons of the model output zero which does not have any effect for the next layers. this idea is something like Dropout. Neurons in hidden layers learn hidden concepts. If the input does not contain the corresponding concept, some neurons will output zero and they will not be engaged in the calculations of the next layers. This idea cannot be in Softplus, because the output cannot be zero like ReLU.

Response functions that link regression predictors to properties of the response distribution are fundamental components in many statistical models. However, the choice of these functions is typically based on the domain of the modeled quantities and is usually not further scrutinized. For example, the exponential response function is often assumed for parameters restricted to be positive, although it implies a multiplicative model, which is not necessarily desirable or adequate. Consequently, applied researchers might face misleading results when relying on such defaults. For parameters restricted to be positive, we propose to construct alternative response functions based on the softplus function. These response functions are differentiable and correspond closely to the identity function for positive values of the regression predictor implying a quasi-additive model. Consequently, the proposed response functions allow for an additive interpretation of the estimated effects by practitioners and can be a better fit in certain data situations. We study the properties of the newly constructed response functions and demonstrate the applicability in the context of count data regression and Bayesian distributional regression. We contrast our approach to the commonly used exponential response function.

Regression analysis is an essential tool for understanding relationships between variables in many fields, including economics, social sciences, and engineering. Response functions and their inverse, known as link functions, play a pivotal role in modern regression methods as they relate distribution parameters to predictors. While default response functions such as the logistic and exponential functions are widely used, they may not always provide the best fit for the specific problem at hand. The selection of an appropriate response function impacts the model in two ways: First, unsuitable response functions lead to poor model fit, potentially violating model assumptions. Second, the choice of the response function significantly impacts the interpretation of covariate effects in a regression model.

The exponential function is popular for strictly positive parameters due to the interpretability of effects as multiplicative. However, this assumption of multiplicativity can be restrictive, as additive effects are often desired in statistical modeling. Traditionally, researchers resort to not using a response function if they desire additivity of effects. However, this can be problematic when the parameter modeled is strictly positive. Certain covariate combinations might lead to a negative value and, thus, invalidate the model.

In addition to the quasi-additive interpretation, the softplus function enables the design of response functions with interesting properties. Augmented with an additional parameter, it yields further flexibility to model the data given, (i) it avoids exponential growth, which can be an issue under certain covariate combinations, and (ii) it enables the construction of an exponential-like function that avoids potential numerical overflow when evaluating it for large positive predictor values.

For the choice of the response function, most researchers rely on default choices such as the logistic response function for parameters restricted to the unit interval (e.g., probabilities) or the exponential response function for strictly positive parameters. In generalized linear models (GLMs, McCullagh and Nelder 1989), these defaults can often be justified by their characterization as natural link functions arising in the context of exponential families. In other cases, the default response functions are chosen to entail specific modes of interpretation, e.g., multiplicative effects on odds in the case of the logistic response function or multiplicative effects on the parameter of interest in the case of the exponential response function. The straightforward interpretability is also the reason why Fahrmeir et al. (2013) recommend using the exponential response function for gamma-distributed responses in the GLM framework instead of the canonical link function. Additionally, special situations require the use of rather exotic response functions. For example, the square root link for Poisson distributed data helps to mimic a least squares estimation on the square root transformed data with a likelihood approach within the GLM framework.

To determine the correct response function in a set of candidate functions, Ntzoufras et al. (2003) propose employing a reversible jump algorithm instead of resorting to model selection criteria. An alternative to pre-chosen response functions is to estimate the response function flexibly from the data. The most well-known example of this approach is the single-index model introduced by Ichimura (1993). The kernel-based single-index models share the disadvantage that the estimated response function is often too flexible. To counter this characteristic, Yu and Ruppert (2002) and Yu et al. (2017) introduced penalization to single-index models. Recently, Spiegel et al. (2019) presented an approach that combines the single-index models based on penalized splines with the flexibility of generalized additive models (Hastie and Tibshirani 1986).

One practical challenge when employing flexible link functions lies in interpreting the resulting model since restrictions must be assigned to the regression predictor to render the response function estimate identifiable. In contrast, simple, fixed response functions considerably facilitate interpretation. Having easily interpretable effects may be why the exponential response function is still the most common approach for positively bounded parameters.

Regardless of how well default choices can be justified, no single response function can fit all situations. Pregibon (1980) points out that a misspecification of the response function is systematic model misspecification. Moreover, domain-specific knowledge about the application can invalidate a multiplicative model entirely and, e.g., suggest an additive model. Therefore, investigating alternative response functions is a worthwhile and relevant endeavor.

The remainder of this paper is structured as follows: Sect. 2 introduces the softplus response function, justifies the quasi-additive interpretation, and gives a guideline for its proper use. Furthermore, Sect. 2 describes statistical inference when employing the softplus response function. Section 3 investigates the softplus response function in simulation studies. The practical applicability of softplus-based regression specifications is demonstrated in Sect. 4. The final Sect. 5 summarizes our findings and discusses future research directions. The code associated with this manuscript can be found on GitHub.Footnote 1

3a8082e126