Optimization with Python
Mathematical optimization is a pervasive task in numerous engineering applications. It is concerned with algorithmic search for a value (usually a vector) that minimizes or maximizes a predefined objective function. In machine learning (ML), for instance, a search for the best configuration of model parameters is performed with an optimization algorithm minimizing the objective function that measures misfit between the known response values and those predicted by the ML model.
Problems of different nature require different optimization algorithms. Luckily, the opensource Python ecosystem provides a good selection of those. This post is aimed as an overview of the available Python libraries/modules suitable for different classes of mathematical optimization, along with some good resources on where to learn the respective methods and tools.
The primary module containing most of the optimizationrelated functionality is scipy.optimize
. Let’s call it opt
(as in from scipy import optimize as opt
) in our further discussion.
Let’s take a look at the classes of optimization problems and the associated tools to tackle them:

Leastsquares minimization: the very common problem of minizing the sum of squred residuals. The tools of the trade are
opt.curve_fit
, as well asnp.polyfit
andnp.linalg.lstsq
. 
Constrained optimization: in addition to the objective function, the problem involves a set of constraints that have to be satisfied. These problems typically fall into the categories of linear programming (LP) or quadratic programming (QP). The methods that can be invoked via
opt.minimize
function areCOBYLA
andSLSQP
. Additionally, thecvxopt
library for convex optimization is available. 
Unconstrained optimization: the most general category of optimization problems, only the objective function (and sometimes its first and second derivatives) are specified. Different methods available through the
opt.minimize
function call can be used to tackle these problems. Some methods, such asBFGS
andNelderMead
require only the objective function itself. Conversely, some methods rely on the gradient and sometimes Hessian of the objective function to select the direction of the most profound change. Examples of these methods areCG
andNetwonCG
. One might be interested in looking at the mystic library, tackling specifically the unconstrained optimization problems and providing an objectorinted API with extensive callback functionality. 
Solving multivariate equations: this boils down to finding roots of vector functions, with the main entry point being the
opt.root
function. 
Global bruteforce optmization: some complex problems require timeconsuming bruteforce methods, especially in situation with multiple local minima. Examples of the functions of this sort are
opt.differential_evolution
andopt.basinhopping
.
Other useful tools I wanted to highlight are those related to automatic computation of gradients to be further used with the methods relying on the objective function derivatives:
autograd
: computes derivatives of arbitrary Python functions based on NumPy code with automatic differentiation.theano
/aesara
: allows for constructing, evaluation and optimizing mathematical expression with multidimensional arrays.
To learn more about using optimization methods in the Python ecosystem, I would recommend the following resources:
 Video tutoriasl by Mike McKerns:
 RealPython tutorials