Optimization with Python

April 11, 2021  [python]  [optimization]  [datascience] 

Mathematical optimization is a pervasive task in numerous engineering applications. It is concerned with algorithmic search for a value (usually a vector) that minimizes or maximizes a predefined objective function. In machine learning (ML), for instance, a search for the best configuration of model parameters is performed with an optimization algorithm minimizing the objective function that measures misfit between the known response values and those predicted by the ML model.

Problems of different nature require different optimization algorithms. Luckily, the open-source Python ecosystem provides a good selection of those. This post is aimed as an overview of the available Python libraries/modules suitable for different classes of mathematical optimization, along with some good resources on where to learn the respective methods and tools.

The primary module containing most of the optimization-related functionality is scipy.optimize. Let’s call it opt (as in from scipy import optimize as opt) in our further discussion.

Let’s take a look at the classes of optimization problems and the associated tools to tackle them:

Other useful tools I wanted to highlight are those related to automatic computation of gradients to be further used with the methods relying on the objective function derivatives:

To learn more about using optimization methods in the Python ecosystem, I would recommend the following resources:

comments powered by Disqus