Re: Isn't log($) arbitrary?

Posted by Pete Moss on July 15, 1997 at 13:24:50: In Reply to: Isn't log($) arbitrary? posted by Thicko on July 15, 1997 at 04:02:09:

Applying any monotonically increasing function to the bankroll growth would not change the maximum of it. The reason we use the log function is that it is very convenient. Bankroll growth is a product. It is easier to work with sums than products. We ask, 'How much "utility" should I add for this decision?' rather than, 'By what fraction will this tend to multiply my bankroll on average?' It is also easier to calculate the derivative of sums than of products.

You have some familiarity with utility or score functions.
Just about all error scores or "objective functions" that I know of are logarithms. That's because we typically want to optimize posterior likelihood conditioned on our observations. For example, when we do "least-squares" regression analysis, we implicitly assume that the noise in the output is Gaussian, having the form,



        Noise = 1/(sqrt(2pi)*s) exp(-.5 x^2/s^2)

(or it's multidimensional counterpart), where s is the standard deviation of the noise. Working with that noise distribution to obtain the joint likelihood of the estimates as a product of their individual likelihoods would be quite unweildy. So instead, we take the logarithm of it and discard some additive and multiplicative constants which do not change the maximum of the score, yielding,


          score = -sum x*x

We now add those rather than multiply them. It's the familiar "sum of squares" error score". Now you know why. It is the logarithm of assumed Gaussian noise, disregarding some constants.

Consider also that in classification and pattern-recognition problems, the log-likelihood utility function, or it's generalization, cross-entropy or "Kulback Leibler Information Criterion", is used when one wishes to maximize posterior likelihood. The likelihood of the combination of a number of independent events is the product of their likelihoods. Working with that product, which would typically be very tiny, would be a pain, particularly since the derivative of a products is unweildy. So we take the logarithm, add rather than multiply, and everything works out great.


Pete


Follow Ups: