Issue #3

MATHEMATICS

Mathematical Optimization

We will give a brief overview of what is mathematical optimization.

Overview

Optimization is a major field in mathematics generally consisting of finding the maximum or minimum values of a function. These functions can model all sorts of physical phenomena around us. In the Football domain, these functions can be statistical models such as the Poisson Regression model or the Dixon-Coles model. Optimizing these functions means optimizing our statistical models.

Optimization of a Function

Calculus

The derivatives of a function are a powerful tool in mathematical optimization. A derivative is the rate of change of a function. For example, if you have a function that describes your position as time passes, the derivative would describe your velocity. Derivatives can also be used to make linear approximations of functions. Linearity is a good mathematical property to have!

If a function has a maximum/minimum, it will be where the derivative is 0! If you look at the above image of the 3-dimensional function, the red dot at the top of the function lies in a flat area where the function is neither increasing or decreasing. This is what we mean by the derivative being 0 (no rate of change).

A very popular technique for solving the max./min. problem for complex multivariable functions (statistical models, machine learning algorithms, etc...) consists of following the derivative vector or more formally known as the gradient. This technique is called gradient descent.

Gradient Descent

Mind you that the gradient is the first derivative vector of a function. The second derivative matrix called the Hessian is an entire other beast that provides further information on the max./min. points of a function. When dealing with vectors and matrices, Calculus overlaps with Linear Algebra which deals with these objects.

Statistical Models

Complex statistical models that do not have easily available closed solutions (read as nice solutions) generally require numerical methods to solve them, and by solving we mean finding the optimal parameters that minimize the errors of that model (read as best fit).

Derivatives provide a linear approximation to functions that make these numerical methods computationally feasible! This is very desirable as we do not want to spend days on end waiting for our computer to solve our statistical models, or do we?

But like everything in life, things get very complicated down the road. Firstly, finding the global max./min. is not as straightforward as it seems. As our algorithms try to find where the derivative is equal to 0, they might mistake a local max./min. for the global max./min. (think of a function with several hills, one bigger than the other). Another issue for statistical models is how we defined the "error" that we wish to minimize. What will we consider an "error"? What metric will we use to define an "error"? (Remember, a metric is a mathematical function that defines the distance between two points. As long as it satisfies three important conditions, we may define a multitude of metrics favorable to our problem).

Local/Global Maximum Minimum

That was a workout!

Keep the brain trained and strong. I have barely skimmed the surface of mathematical optimization. There are too many techniques out there to mention in one email, too many smart approaches to fit in one book. It's amazing how derivatives, a concept developed in the 17th century by Newton or Leibniz (you pick) plays such an important part in our everyday life.

What's new?

PoisonFoot just got a much needed design and user experience improvement! Check it out and let me know what you think!

I am always striving to make the user experience the best possible.

Beginning of Year outage

While out in vacation for the New Years the server I set up to automatically update the models failed due to 1 line of code! Unbelievable, but that's how programming goes sometimes. I am still working to find some type of compensation for that one week of outage and I will be contacting subscribers shortly! I appreciate everyone's patience and I hope you had a great start to the raging 20's!

Like what you read? I'd like to hear your ideas and suggestions! Contributors are welcome!

Email me at [email protected]