The nonlinear programming FAQ was developed at the Optimization Technology Center of Northwestern University and Argonne National Laboratory. The FAQ is no longer maintained. The last update of the content was January 22, 2007 with some broken links updated on July 31, 2013.
Contents 
What is Nonlinear Programming?
A Nonlinear Program (NLP) is a problem of the following the form
minimize  \(F(x)\)  
subject to  \(g_i(x) = 0\)  for \(i = 1, ..., m_1\) 
\(h_j(x) \geq 0\)  for \(j = m_1+1, ..., m\) 
where \(m_1 \geq 0\) and \(m \geq m_1\). That is, there is one scalarvalued function \(F\), of several variables (\(x\) here is a vector), that we seek to minimize subject (perhaps) to one or more other such functions that serve to limit or define the values of these variables. \(F(x)\) is called the "objective function", while the various other functions are called the "constraints". (If maximization is sought, it is trivial to do so, by multiplying \(F(x)\) by 1.)
Because NLP is a difficult field, researchers have identified special cases for study. A particularly well studied case is the one where all the constraints \(g\) and \(h\) are linear. The name for such a problem, unsurprisingly, is "linearly constrained optimization". If, as well, the objective function is quadratic at most, this problem is called Quadratic Programming (QP). An even more special case of great importance is where the objective function and the constraints are entirely linear; this is called Linear Programming (LP) and is discussed in a separate FAQ list.
Another important special case, called unconstrained optimization, is where there are no constraints at all. One of the greatest challenges in NLP is that some problems exhibit "local optima"; that is, spurious solutions that merely satisfy the requirements on the derivatives of the functions. Think of a nearsighted mountain climber in a terrain with multiple peaks, and you'll see the difficulty posed for an algorithm that tries to move from point to point only by climbing uphill. Algorithms that propose to overcome this difficulty are termed "Global Optimization". The word "Programming" is used here in the sense of "planning"; the necessary relationship to computer programming was incidental to the choice of name. Hence the phrase "NLP program" to refer to a piece of software is not a redundancy, although I tend to use the term "code" instead of "program" to avoid the possible ambiguity.
What software is there for nonlinear optimization?
It's unrealistic to expect to find one general NLP code that's going to work for every kind of nonlinear model. Instead, you should try to select a code that fits the problem you are solving. If your problem doesn't fit in any category except "general", or if you insist on a globally optimal solution (except when there is no chance of encountering multiple local optima), you should be prepared to have to use a method that boils down to exhaustive search.If you simply want to try solving a particular model, consider the Optimization Technology Center. The centerpiece of this ambitious project is NEOS, the NetworkEnhanced Optimization System, consisting of a library of optimization software, an optimization server for using this library over the network, and a guide to more information about the software packages. Linear and nonlinear models are covered. Capabilities and access modes are still being enhanced  this is an ongoing process.
Several of the commercial linear programming codes referenced in the LP FAQ have specialized routines, particularly for quadratic programming (QP). You don't generally get source code when you license one of these products, but many of them can be licensed as a callable library of solver routines. Many general nonlinear problems can be solved (or at least confronted) by application of a sequence of LP or QP approximations.There are routines from ACM Transactions on Mathematical Software for QP, #559 by J.T. Betts (volume 6, page 432) and #587 by R.J. Hanson and K.H. Haskell (volume 8, page 323).The opt directory of Netlib contains a number of freely available optimization routines, including:
 conmax (general nonlinearly constrained function minimization)
 donlp2 (differentiable nonlinear optimization, dense linear algebra)
 dqed (nonlinear least squares with linear constraints)
 hooke (derivativefree unconstrained optimization, via [#Hooke Hooke and Jeeves method])
 lbfgs (nonlinear boundconstraint optimization, limited memory BFGS method)
 lsnno (nonlinear optimization subject to linear network constraints)
 praxis (unconstrained optimization, without requiring derivatives)
 simann (unconstrained optimization using Simulated Annealing)
 subplex (unconstrained optimization, general multivariate functions)
 tn (Newton method for unconstrained or simplebound optimization)
 varpro (separable nonlinear least squares)
 ve08 (optimization of unconstrained separable function).
A newer version of donlp2, mentioned above, is available. This is P. Spellucci's implementation of a SQP method for general nonlinear optimization problems including nonlinear equality and inequality constraints (generally referred to as the NLP problem). It is freely available for noncommercial and research use, and includes a number of nontrivial examples. There are Fortran 77, Fortran 90 and f2cconverted C versions, with a variety of options for the gradient computations.
The NLopt library provides free/opensource implementation of a number of different local and global optimization algorithms for nonlinear programming, including bound constraints and general nonlinear inequality/equality constraints, with function values only or exploiting usersupplied gradients.
Packages for large boundconstrained problems (that is, having only simple lower and upper bounds on the variables as constraints) include TRON, a trustregion Newton method implemented by Lin and Mor, and LBFGSB, a limitedmemory method by Zhu and Nocedal.
A package called conmin (unrelated to the one by Vanderplaats and Associates) is made available by Murray Dow (m.dow@anusf.anu.edu.au). The author states that it is reliable but not state of the art; surpassed, for instance, by FSQP.
WNLIB by Bill Chapman and Will Naylor includes routines for unconstrained and constrained nonlinear optimization based on conjugategradient and conjugatedirections algorithms (as well as a general simulated annealing routine).
The NSWC Library of Mathematical Subroutines has a routine to minimize a function of n variables (OPTF) and a routine to solve a system of nonlinear equations (HBRD). These are also available in the NMS library SolvOpt, by Alexei Kuntsevich (alex@bedvgm.kfunigraz.ac.at) and Franz Kappel (franz.kappel@kfunigraz.ac.at), is designed for local optimization of nonsmooth nonlinear problems. Free source code is available in C and Fortran, and also as Mfunctions for use with MATLAB. Further information is provided by a manual that is also available for downloading.
The popular PC and Mac spreadsheet packages come with options or addins that can do some nonlinear optimization. They can be really convenient if you already have your data in a spreadsheet and if your problem is not too large or difficult. A much broader range of more powerful solvers are available for separate purchase from LINDO Systems and Frontline Systems. Some of these solvers give special attention to nonsmooth nonlinear functions such as MIN, IF, and LOOKUP that are common in spreadsheet formulas.
If you have access to MATLAB, you can take advantage of a number of useful nonlinear optimization packages:
 The MATLAB Optimization Toolbox includes routines for a variety of constrained and unconstrained nonlinear optimization problems.
 The TOMLAB Optimization Environment offers a broad variety of nonlinear optimization tools based on MATLAB toolboxes. It has a unified inputoutput format, and automatic handling of derivatives. Interfaces to numerous solvers are provided, with the option of compilation to standalone applications using the Matlab compiler.
 LOQO has a MATLAB interface for quadratic programming.
 Several recently developed semidefinite programming solvers are based on MATLAB.
 MCS is a global optimization package that runs under MATLAB.
 The optimization listings at mathtools.net include many solvers written in MATLAB (as well as others in general programming languages).
There are also the following Mathematica packages:
 MathOptimizer, for global and local (numerical) solution of a very general class of nonlinear optimization problems defined by a finite number of realvalued, continuous functions over a finite ndimensional interval region; intended particularly for dense, highly nonlinear systems, including those that have a (typically unknown) number of local optima.
 Global Optimization is a collection of functions for global nonlinear optimization that uses the Mathematica system as an interface for defining the nonlinear system to be solved and for computing function numeric values.
Semidefinite Programming is a generalization of linear programming to the space of block diagonal, symmetric, positive semidefinite matrices. Interest in this topic, which has numerous engineering applications, has been greatly stimulated by the extension of interiorpoint methods from linear programming to the semidefinite case. Software packages currently under development for semidefinite programming include:
 CSDP, a standalone solver and subroutine library, written in C, which implements a predictorcorrector semidefinite programming algorithm with support for linear equality and inequality constraints.
 DSDP, a C code that implements a dual scaling algorithm for mixed linear and positive semidefinite programming problems with rankone matrix constraints.
 PENSDP, a semidefinite programming code that can be used as a standalone program, called from applications, or accessed via the TOMLAB /PENSDP MATLAB interface.
 SBmethod, a C++ code for for minimizing the maximum eigenvalue of an affine matrix function, which is capable in particular of solving the duals of most large scale semidefinite relaxations of combinatorial optmization problems.
 SDPHA, a MATLAB package using the socalled homogeneous formulation.
 SDPpack, a MATLAB package for semidefinite programs and their extension to mixed semidefinitequadraticlinear programs, currently in version 0.9 beta.
 SDPSOL, a parser/solver for semidefinite programming and related determinant maximization problems with matrix structure.
 SDPT3, another MATLAB package, incorporating infeasible pathfollowing and homogeneous selfdual algorithms for standard semidefinite programming (possibly with complex data); sparsity in the data is exploited whenever possible.
 SeDuMi, a MATLAB toolbox for solving optimization problems over selfdual homogeneous cones, which allows for linear, quasiconvex quadratic and positive semidefiniteness constraints, both with real and complex entries.
 LMITOOL, a graphical interface for linear matrix inequalities (equality and positivedefiniteness constraints, linear objective) expressed in the form of MATLAB .m files, which provides a choice of several semidefinite programming solvers.
See the semidefinite programming home pages maintained by Farid Alizadeh and Christoph Helmberg for links and further information.
An extensive index of information on Global Optimization is maintained by Arnold Neumaier (Arnold.Neumaier@univie.ac.at) of the Computational Mathematics group at the University of Vienna. An overview is given by János Pintér's Continuous Global Optimization: An Introduction to Models, Solution Approaches, Tests and Applications (volume 2, number 2 of Interactive Transactions of OR/MS). The following are a few of the codes available in this area:
 BARON consists of a "core" module for global nonlinear optimization in continuous and/or discrete variables, and a variety of specialized modules for such problems as bilinear programming, fixedcharge programming, indefinite quadratic programming, linear multiplicative programming, and univariate polynomial programming. Information is available from Nick Sahinidis, nikos@uiuc.edu.
 LGO integrates several global and local optimization solvers, which can be activated by the user in interactive or automatic execution modes. The PC Windows version is embedded under a menudriven interface and incorporates an integrated development environment.
 MCS (Multilevel Coordinate Search) is a MATLAB program for bound constrained global optimization using function values only, based on a multilevel coordinate search that balances global and local search.
 TOMLAB /CGO is a Matlab toolbox for CPUintensive nonconvex global optimization, employing a choice of several methods.
 A software package for solving structured global optimization problems, cGOP, is available from the ComputerAided Systems Laboratory at Princeton University. It implements a primaldual decomposition algorithm applicable to general constrained biconvex problems, using a set of C subroutines to solve these problems via decomposition and branchandbound techniques; version 1.1 has been updated to use CPLEX 4.0 and MINOS 5.4 as the solvers for various linear and convex subproblems. For assistance, write to cgop@titan.princeton.edu.
 Fortran source code for global minimization using a stochastic integration algorithm is available as ACM Transactions on Mathematical Software algorithm #667 by F. AluffiPentini et al. (volume 14, page 345).
 The Mathematica packages MathOptimizer and Global Optimization are applicable to a variety of problems.
When some of the variables are required to take integer values, the problem becomes one of mixedinteger nonlinear programming (sometimes abbreviated, a bit confusingly, as MINLP). Software for this particularly difficult kind of global optimization has followed two approaches (see also the [#Hansen survey article] by Hansen, Jaumard and Mathon):
 Outer Approximation/Generalized Bender's Decomposition. These algorithms alternate between solving a mixed integer linear programming master problem and nonlinear programming subproblems. Examples include DICOPT++ and MINOPT.
 Branch and Bound. B&B methods for mixedinteger linear programming can be extended in a straight forward way to the nonlinear case. There are a number of other tricks that can be used to improve the performance of branch and bound codes for MINLP. One example of this approach is incorporated into the BARON package.
For difficult problems like Global Optimization, heuristic search methods have been extensively studied; some of the best known types are Simulated Annealing, Tabu Search, and Genetic (or Evolutionary) Algorithms. As a practical matter these methods cannot be expected to find a provably optimal solution, or even a provably good solution. They sometimes find the best known solutions, however, which may be more than adequate for the task at hand. They tend to do best when they can be tailored to odd characteristics of your problem that defy treatment by more general or conventional approaches. If you're interested, here are some places to start looking:

 Darrel Whitley of Colorado State University has written a 37page Genetic Algorithm Tutorial. Chris Lucas offers a page on Practical Multiobjective Optimisation through genetic algorithms and extensions.
 The GA Playground offers a generalpurpose toolkit for experimenting with genetic algorithms and solving optimization problems. The toolkit is written in Java, so for introductory purposes it may be run as an applet through certain Web browsers, though it is primarily designed to be used as an application in conjunction with a Java compiler.
 The Usenet newsgroup on genetic algorithms, comp.ai.genetic, has a FAQ on the topic, otherwise known as "The HitchHiker's Guide to Evolutionary Computation".
 New Light Industries distributes GENERATOR, a genetic algorithm Solver for the Microsoft Excel spreadsheet package. Special versions are available for problems in finance and in chemistry.
 In the area of simulated annealing, software for Adaptive Simulated Annealing is available from Lester Ingber (ingber@alumni.caltech.edu), and for Ensemble Based Simulated Annealing from Richard Frost (frost@sdsc.edu). And there is the simann code available on Netlib, mentioned above.
 Several recently developed heuristic search approaches, including Ant Colony Optimization, Differential Evolution, Immune System Methods, Memetic Algorithms, and Scatter Search, are described in a book titled New Ideas in Optimization by Corne et al.
For other ideas on Global Optimization, you may want to consult one of the books given in the section on references , such as Nemhauser or one of the ones with "Global" in its title. There are also the Journal of Global Optimization and the Journal of Heuristics.
Complementarity problems are closely related to nonlinear optimization problems. The most familiar complementarity conditions are the complementary slackness conditions for optimality in linear programming, which say for instance that either a certain dual variable is zero or the corresponding dual slack is zero, or both. Pure complementarity problems consist of these and related eitheror conditions; combinations of these conditions with ordinary objectives and constraints produce socalled mathematical programs with equilibrium constraints, or MPECs. More information, including a list of solvers, is available from the Complementarity Problem Net home page.
The following products can also be useful for nonlinear optimization, but don't fall into any of the usual categories:

 Best Value Solver, "the first universal mass market numerical processing package," offers a significant extension of the spreadsheed paradigm. It has facilities for a variety of optimization problems.
 Fortran Calculus is a programming front end to a variety of nonlinear problem solvers, available from Optimal Designs, OptDesigns@Lanset.com.
 IBM ILOG CPLEX CP Optimizer, a C++ class library mainly known for solving hard combinatorial problems by constraint logic programming methods, has introduced facilities for global optimization of nonlinear programs that have a finite number of isolated solutions.
 OptSolve++ is a C++ class library for nonlinear programming. The current release incorporates several solution techniques, including the (NelderMead) simplex method, Powell's method, a conjugate gradient method, and the LevenbergMarquardt method.
 UniCalc, for "arbitrary algebraic systems of equations and inequalities," is said to combine interval constraint propagation, interval mathematics, computer algebra, original search strategies, and a friendly user interface. It has some options for specifying objective functions.
 TK Solver, "a rulebased declarative environment for creating mathematical models and solving them multidirectionally," has a number of library models for optimization.
Commercial codes
The following table is a condensed version of a survey of NLP software published in the June 1998 issue of OR/MS Today, a publication of INFORMS. Several of the software vendors listed in the survey offer multiple products, in keeping with the conventional wisdom that no one algorithm will be best for all NLP models. Hence I have grouped the solver products by vendor, rather than listing them alphabetically by product name. Since the information won't fit on one line, I've broken the SOLVERS part of the table into two pieces.
Solver Vendor  Phone  Email address 
6789901121  info@aemdesign.com  
2064327855  info@aptech.com  
7758310300  info@frontsys.com  
46 60 573 748  info@grabitech.se  
INRIA  33 139635557  scilab@inria.fr 
3129887422  info@lindo.com  
6305791190  craigloehl@aol.com  
5086531415  info@mathworks.com  
45 39179907  info@mosek.com  
NAG (Numerical Algorithms Group) 
6309712337  naginfo@nag.com 
9024435910  jdpinter@hfx.eastlink.ca  
Princeton University Licensing  6092581570  jritter@Princeton.edu 
8006365327  projects@quantumleap.us  
7322644700  sales@saitechinc.com  
8007270025  
49 921553278  Klaus.Schittkowski@unibayreuth.de  
49 6151 162293  spellucci@mathematik.tudarmstadt.de  
6508561695  info@sbsisoloptimize.com  
4159628719  vanderplaats@vma.com  
7137843131  info@boulder.vni.com  
8474912504  info@ziena.com 
Vendor  Product  Method (see key below) 
AEM Design  SQP  
CCLRC  Augmented lagrangian  
Aptech 
GAUSS CO, CML modules 
SQP 
Frontline Systems  GRG  
INRIA  SQP  
Loehle Enterprises  various  
Grabitech  simplex (NelderMead type)  
LINDO Systems  GRG, SLP, global search  
Mathworks  various  
MOSEK ApS  IP  
NAG  various  
Pinter Consulting  LGO  various local and global search 
Princeton Univ.  IP  
Quantum Leap  multiple in cooperation  
SAITECH  SQP, IP  
SAS  various  
K. Schittkowski  SQP various 

P. Spellucci  SQP  
Stanford Bus Soft  Reduced gradient, augmented lagrangian SQP SQP 

Vanderplaats  various  
Visual Numerics  various  
Ziena Optimization  IP, SQP 
Key to Methods:
 IP = InteriorPoint
 GRG = Generalized Reduced Gradient
 SLP = Successive (Sequential) Linear Programming
 SQP = Successive (Sequential) Quadratic Programming
Modeling systems
Communicating with a nonlinear programming code can be particularly tedious and errorprone, especially if you have to write programs in a language like Fortran or C to compute function (and maybe gradient) values for your objective and constraints. If your functions can be stated mathematically, then chances are good that you can use an algebraic modeling language to communicate them in the same mathematical form to a variety of solvers. Several packages are oriented toward nonlinear optimization in engineering design, though they have other applications as well:
 APMonitor, a differential and algebraic (DAE) modeling language, is suited for largescale problems. A unique feature of APMonitor is the ability to perform dynamic parameter estimation through OPC or ODBC connections. This allows realtime model fitting as new data become available. APMonitor does not solve the problems directly, but calls appropriate external solvers such as IPOPT.
 ASCEND IV, a largescale, equationbased environment featuring a stronglytyped, objectoriented model description language, has been developed at Carnegie Mellon University. It has particularly useful features for problems in engineering and mathematical sciences. Versions for Unix, Linux and several varieties of Windows are available free for downloading.
 OptdesX incorporates several methods for continuous and discrete optimization are supported, mainly on Unix workstations. There is no modeling language in the usual sense; models are described through a combination of a graphical interface and external userspecified routines.
 Pointer combines a variety of methods to bring optimization to engineering design activities, especially in aerospace and mechanical engineering.
MProbe is a software tool for analyzing nonlinear functions to discern their shapes in a region of interest  for example, whether they are linear or almost linear, convex or almost convex  together with related information about objectives and constraints. Such information is often crucial to developing and solving nonlinear optimization models. MProbe uses the AMPL language for describing functions to be analyzed.
Several commercial algebraic modeling languages are noteworthy for being available with various nonlinear solvers. AMPL seems to have the largest number of different nonlinear solver interfaces.
 AIMMS links to CONOPT, SNOPT, MINOS, and KNITRO. Includes nonlinear presolver and multistart algorithm.
 AMPL
 GAMS
 MINOPT
Other nonlinear programming codes
Here is a summary of other NLP codes mentioned in newsgroups in the past few years (or, further information on the ones in the above table), sorted alphabetically. In the fullness of time, these references may be reorganized in a more structured format.
 ACCPM  implements an analytic center cutting plane method for convex optimization problems.
 Amoeba  in Numerical Recipes
 Brent's Method  in Numerical Recipes
 CO/CML  Applications written in GAUSS language, for general constrained optimization and constrained maximum likelihood estimation using SQP. CML includes statistical inference (also Bayesian, bootstrap) for constrained parameters.
 COINOR, a COmputational INfrastructure for Operations Research, is an open source repository established with support from IBM Corporation and now operating as an independent nonprofit organization. Fortran source code is available under the Common Public License for several nonlinear optimization packages, notably the following:
 CONMAX  Fortran program for solving nonlinearly constrained problems of the form:
 Choose x1,...,xn to minimize w subject to
 abs(fi  gi(x1,...,xn)) .LE. w, 1 .LE. i .LE. m1
 gi(x1,...,xn) .LE. w, m1 + 1 .LE. i .LE. m2
 gi(x1,...,xn) .LE. 0, m2 + 1 .LE. i .LE. m3 where the fi are given real numbers, and the gi are given smooth functions. Constraints of the form gi(x1,...,xn) = 0 can also be handled. Each iteration of our algorithm involves approximately solving a certain nonlinear system of first order ordinary differential equations to get a search direction. The program and its extensive user's guide can be obtained from the opt folder of Netlib.
 CONMIN  Vanderplaats, Miura Ô Associates, Colorado Springs, Colorado, 7195272691.
 COOOL (CWP [Colorado School of Mines Center for Wave Phenomena] ObjectOriented Optimization Library) is a library of C++ classes for developing optimization codes, together with generalpurpose routines for solving varied linear and nonlinear optimization problems.
 DistOpt  A general distributed optimization package with interfaces to various solvers.
 DFPMIN  in Numerical Recipes (DavidonFletcherPowell)
 Direct Optimizer is an addin to Microsoft Excel, based on the HookeJeeves algorithm.
 filterSQP is a package of Fortran 77 subroutines for finding local solutions to nonlinear programming problems. It is based on a sequential quadratic programming trustregion algorithm, using a recently developed "filter" technique to promote global convergence. This approach has the advantage that the user need not supply any estimates of penalty parameters. Interfaces to CUTE and AMPL are available.
 FSQP and related sequential quadratic programming codes for general nonlinearly constrained problems, including constrained minimax problems, are available for download for research and development purposes from AEM Design. A separate interface to AMPL is also available.
 GENOCOP III  Solves nonlinearly constrained optimization problems via a genetic algorithm.
 geom_prog by M. Dow (m.dow@anu.edu.au) is specialized to solve Duffin geometric programming problems.
 Harwell Library routines
 VF01: based on R. Fletcher algorithm
 VF02: based on M. Powell algorithm
 VF03: using "watchdog techniques" for line search. An improved version of VF02.
 VF04: Automatically calculate 1st order derivatives, VF03 ia required to provide the derivatives.
 Hooke and Jeeves algorithm  see Hooke reference below. May be useful because it handles nondifferentiable and/or discontinuous functions.
 LEVMAR is an implementation of the LevenbergMarquardt algorithm for nonlinear least squares problems, written by Manolis Lourakis.
 MINPACK solves dense nonlinear leastsquares problems. Contact Jorge More' (more@mcs.anl.gov).
 NLopt  free/opensource implementations of a number of nonlinear optimization algorithms (local and global, nonlinear constraints, with or without usersupplied gradients)
 OMatrix for Windows  incorporates several nonlinear optimization tools.
 Omuses/HQP comprises a solver for nonlinearly constrained largescale optimization problems (especially with regular sparsity structure) and a front end based on C++ and ADOLC. It uses a Newtontype SQP method, with an interiorpoint procedure for the treatment of constraints. It is available under the GNU Lesser General Public License.
 OPT++  An objectoriented package for nonlinear optimization, including support for general linear and nonlinear constraints, a parallel optimization method, and a nonlinear interior point method. Other capabilities include parallel direct search, nonlinear conjugate gradient, quasiNewton and full Newton methods. The package has been ported to Linux and other Unixrelated platforms.
 Simple code for the NelderMead method (the "other simplex method") can be found in Numerical Recipes and Barabino.
 A variety of packages for nonlinear optimization, equation solving and related problems are collected at the home page of Prof. Klaus Schittkowski of the University of Bayreuth.
 Semiinfinite programming: Ismael Vaz has developed SIPAMPL and NSIPS, which provide an AMPL interfaces to solvers for semiinfinite programs.
 SLATEC  Quadratic solvers dbocls, dlsei, and other routines of the National Energy Software Center.
An extremely useful book is the Optimization Software Guide, by Jorge More' and Stephen Wright, from SIAM Books. It contains references to 75 available software packages, and goes into more detail than is possible in this FAQ. A Web version is available.
I would be interested in hearing of people's experiences with the codes they learn about from reading this FAQ  particularly if they can provide moreorless independent evidence of the codes' practicality or impracticality.
I wrote an optimization code. Where are some test models?
There are several large collections of test problems, as well as many smaller collections of problems of particular kinds.
The Handbook of Test Problems in Local and Global Optimization by Floudas et al. collects a large group of test problems of diverse types and applications. Corresponding input files are available in GAMS or MINOPT format.
Jorge Moré and collaborators are developing COPS, a collection of largescale nonlinearly Constrained Optimization ProblemS intended to provide difficult test cases for optimization software. The 17 problems in the current version (2.0) of the collection come from such diverse applications as elastic plastic torsion, catalyst mixing, cam shape optimization, and marine population dynamics. Each is accompanied by a description and formulation, implementations in the AMPL and GAMS modeling languages, and benchmarking results.
CUTEr (Constrained and Unconstrained Testing Environment, revisited) is designed to be a versatile testing environment for optimization and linear algebra solvers. The package contains a large collection of test problems, along with Fortran 77, Fortran 90/95 and Matlab tools intended to help developers design, compare and improve new and existing solvers. The provided test problems are written in socalled Standard Input Format (SIF), for which a converter to welldefined Fortran 77 and data files is available as a separate package. Once translated, these files may be manipulated to provide tools suitable for testing optimization packages. Readytouse interfaces to MINOS, SNOPT, filterSQP, IPOPT, KNITRO, LOQO, and other solvers are available.
Hans D. Mittelmann and P. Spellucci have prepared a downloadable test environment of over 400 unconstrained and constrained nonlinear optimization problems, plus code to facilitate interfacing solvers to them. See also the extensive benchmark results reported at this site.
Klaus Schittkowski has collected over 600 data fitting test problems motivated by parameter estimation in dynamical systems. Nonlinear functions are represented in the PCOMP language, which can be either interpreted directly by solvers or translated to Fortran code.
Most modeling systems for nonlinear programming have associated collections of test problems in their modeling languages. A collection of nonlinear AMPL models  in over 25 categories from Antenna Array Synthesis to Trajectory Optimization  has been compiled by Robert Vanderbei of Princeton University. The GAMS Model Library contains over 160 entries, many of them nonlinear.
Nonlinear optimization test problems are described in many papers, including the following:
 A. Corana et al, "Minimizing Multimodal Functions of Continuous Variables with the Simulated Annealing Algorithm," ACM Transactions on Mathematical Software, Vol. 13, No. 3, Sept 1987, pp. 262280. (Difficult unconstrained nonlinear models)
 C.A. Floudas and P.M. Pardalos, A Collection of Test Problems for Constrained Global Optimization Algorithms, SpringerVerlag, Lecture Notes in Computer Science 455 (1990).
 W.W. Hager, P.M. Pardalos, I.M. Roussos, and H.D. Sahinoglou, Active Constraints, Indefinite Quadratic Programming, and Test Problems, Journal of Optimization Theory and Applications Vol. 68, No. 3 (1991), pp. 499511.
 J. Hasselberg, P.M. Pardalos and G. Vairaktarakis, Test case generators and computational results for the maximum clique problem, Journal of Global Optimization 3 (1993), pp. 463482.
 B. Khoury, P.M. Pardalos and D.Z. Du, A test problem generator for the steiner problem in graphs, ACM Transactions on Mathematical Software, Vol. 19, No. 4 (1993), pp. 509522.
 Y. Li and P.M. Pardalos, Generating quadratic assignment test problems with known optimal permutations, Computational Optimization and Applications Vol. 1, No. 2 (1992), pp. 163184.
 P. Pardalos, "Generation of LargeScale Quadratic Programs", ACM Transactions on Mathematical Software, Vol. 13, No. 2, p. 133.
 P.M. Pardalos, Construction of test problems in quadratic bivalent programming, ACM Transactions on Mathematical Software, Vol. 17, No. 1 (1991), pp. 7487.
 P.M. Pardalos, Generation of largescale quadratic programs for use as global optimization test problems, ACM Transactions on Mathematical Software, Vol. 13, No. 2 (1987), pp. 133137.
 F. Schoen, "A Wide Class of Test Functions for Global Optimization", J. of Global Optimization, 3, 133137, 1993.
See also the publications listed in the Q4 references section of this FAQ, particularly those by Schittkowski, by Hock and Schittkowski, and by Torn and Zilinskas.
Additional possibilities for global optimization include:

 A collection of linear reverse convex programs and concave minimization problems, compiled by Steve Jacobsen and Khosrow Moshirvaziri.
 Fortran test functions from the end of the TOMS 667 code.
 Seven functions intended to thwart global minimization algorithms, from "The optimization problem: An introduction" by L.C.W. Dixon and G.P. Szego, in Towards Global Optimization II, edited by Dixon and Szego, North Holland, 1978. (Some now consider these easy.)
A generator for quadratic programming test problems is described by Calamai, Vicente and Judice in "A new technique for generating quadratic programming test problems," Mathematical Programming 61 (1993) 215231. SDPLIB is a collection of semidefinite programming test problems from a variety of applications including control systems engineering, truss topology design, graph partitioning, and relaxations of quadratic assignment problems. The problems are stored in the SDPA sparse format.
A collection of linear complementarity problems in GAMS and AMPL is available through the Complementarity Problem Net home page. A collection of MPEC test problems in AMPL is available at the MacMPEC page.
What references are there in this field?
What follows here is an idiosyncratic list, a few books that I like, or have been recommended on the net, or are recent. I have not reviewed them all.
General reference:
 Nemhauser, Rinnooy Kan, & Todd, eds, Optimization: Handbooks in Operations Research and Management Science, Volume 1, NorthHolland, 1989. (Very broadreaching, with large bibliography; a good reference, and worth checking first. Expensive, though, and tough reading for beginners.)
Books:
 Bazaraa, Shetty and Sherali, Nonlinear Programming: Theory and Applications. Wiley, 1994.
 Bertsekas, Dimitri P., Dynamic Programming and Optimal Control. Belmont, MA: Athena Scientific, 1995.
 Bertsekas, Dimitri P., Nonlinear Programming, second edition. Athena Scientific, 1999.
 Bryson, Arthur E., Dynamic Optimization. PrenticeHall, 1998. (Comes with CD of Matlabbased tools.)
 Coleman and Li, Large Scale Numerical Optimization. SIAM Books, 1990.
 Dennis and Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Prentice Hall, 1983.
 Du and Pardalos, Minimax and applications. Kluwer, 1995.
 Du and Sun, Advances in Optimization and Approximation. Kluwer, 1994.
 Duffin, Peterson and Zener, Geometric Programming. Wiley, 1967.
 Fiacco and McCormick, Sequential Unconstrained Minimization Techniques. SIAM Books. (An old standby, given new life by the interior point LP methods.)
 Fletcher, R., Practical Methods of Optimization. Wiley, 1987. (Good reference for Quadratic Programming, among other things.)
 Floudas, Christodoulos A., Deterministic Global Optimization: Theory, Algorithms and Applications, Kluwer, 1999.
 Floudas and Pardalos, Recent Advances in Global Optimization. Princeton University Press, 1992.
 Gill, Murray and Wright, Practical Optimization. Academic Press, 1981. (An instant NLP classic when it was published.)
 Himmelblau, Applied Nonlinear Programming. McGrawHill, 1972. (Contains some famous test problems.)
 Hock and Schittkowski, Test Examples for Nonlinear Programming Codes. SpringerVerlag, 1981.
 Horst and Pardalos, Handbook of Global Optimization. Kluwer, 1995.
 Horst, Pardalos, and Thoai, Introduction to global optimization. Kluwer, 1995.
 Horst and Tuy, Global Optimization. SpringerVerlag, 1993.
 Kahaner, Moler Ô Nash, Numerical Methods and Software. PrenticeHall.
 Lau, H.T., A Numerical Library in C for Scientists and Engineers. CRC Press, 1994. (Contains a section on optimization.)
 Luenberger, Introduction to Linear and Nonlinear Programming. Addison Wesley, 1984. (Updated version of an old standby.)
 Moré and Wright, Optimization Software Guide. SIAM, 1993.
 Nash, S. and Sofer, A., Linear and Nonlinear Programming. McGrawHill, 1996.
 Nocedal and Wright, Numerical Optimization. Springer Verlag, 1999.
 Pardalos and Wolkowicz, Quadratic Assignment and Related Problems. American Mathematical Society, DIMACS series in discrete mathematics, 1994.
 Pinter, Computational Global Optimization in Nonlinear Systems: An Interactive Tutorial. Lionheart, 2001.
 Pinter, Global Optimization in Action: Continuous and Lipschitz Optimization: Algorithms, Implementations and Applications. Kluwer, 1996. (Winner of the 2000 INFORMS Computing Society Award.)
 Press, Flannery, Teukolsky Ô Vetterling, Numerical Recipes, Cambridge, 1986.
 Schittkowski, Nonlinear Programming Codes SpringerVerlag, 1980.
 Schittkowski, More Test Examples for Nonlinear Programming Codes. Lecture Notes in Economics and Math. Systems 282, Springer, 1987.
 Torn and Zilinskas, Global Optimization. SpringerVerlag, 1989.
 Wismer and Chattergy, Introduction to Nonlinear Optimization. NorthHolland, 1978. (Undergraduate text.)
Other publications:
 Barabino, et al, "A Study on the Performances of Simplex Methods for Function Minimization," 1980 Proceedings of the IEEE International Conference on Circuits and Computers, (ICCC 1980), pp. 11501153.
 Borchers Ô Mitchell, "An Improved Branch and Bound Algorithm for Mixed Integer Nonlinear Programs", Computers and Operations Research, Vol 21, No. 4, p 359367, 1994.
 N.I.M. Gould, D. Orban, and Ph.L. Toint, "GALAHAD, a library of threadsafe Fortran 90 packages for largescale nonlinear optimization," Technical Report RALTR2002014 (2002), Rutherford Appleton Laboratory, Chilton, England.
 Hansen, Jaumard, and Mathon, "Constrained Nonlinear 01 Programming", ORSA Journal on Computing, 5, 2 (1993).
 Hooke Ô Jeeves, "Direct Search Solution of Numerical and Statistical Problems", Journal of the ACM, Vol.8 pp. 212229, April 1961.
 Moré, "Numerical Solution of Bound Constrained Problems", in Computational Techniques Ô Applications, CTAC87, Noye Ô Fletcher, eds, NorthHolland, 2937, 1988.
 Moré Ô Toraldo, Algorithms for Bound Constrained Quadratic Programming Problems, Numerische Mathematik 55, 377400, 1989.
 Nocedal, J., summary of algorithms for unconstrained optimization in "Acta Numerica 1992".
 Powell, M.J.D., "A Fast Algorithm for Nonlinearly Constrained Optimization Calculations", SpringerVerlag Lecture Notes in Mathematics, vol. 630, pp. 144157. (Implemented in the Harwell Library)
 Wright, M., "Interior methods for constrained optimization", Acta Mathematica, Cambridge University Press, 1992. (Survey article.)
 Wright, M., "Direct Search Methods: Once Scorned, Now Respectable." Numerical Analysis 1995 (Proceedings of the 1995 Biennial Conference on Numerical Analysis, Dundee), Addison Wesley Longman, 1996, pp. 191208.
Heuristic Search Methods:
 Corne, Dorigo, and Glover, eds., New Ideas in Optimization. McGrawHill (1999).
 De Jong, "Genetic algorithms are NOT function optimizers" in Foundations of Genetic Algorithms II, D. Whitley (ed.), Morgan Kaufman (1993).
 Glover and Laguna, Tabu Search, Kluwer, 1997.
 Goldberg, D.E., Genetic Algorithms in Search, Optimization Ô Machine Learning, AddisonWesley (1989).
 Ingber "Very Fast Simulated Reannealing," Mathematical and Computer Modeling 12 (1989) 967973.
 Kirkpatrick, Gelatt, and Vecchi, "Optimization by Simulated Annealing," Science 220 (1983) 671680.
 Michalewicz et al., "A Nonstandard Genetic Algorithm for the Nonlinear Transportation Problem," ORSA Journal on Computing 3 (1991) 307316.
 Michalewicz, Genetic Algorithms + Data Structures = Evolution Programs, 3rd ed., Springer Verlag (1996).
 Reeves, ed., Modern Heuristic Techniques for Combinatorial Problems, Wiley (1993). Contains chapters on tabu search, simulated annealing, genetic algorithms, neural nets, and Lagrangean relaxation.
What's available online in this field?
Though traditional publications may remain the best places to learn about optimization theory and algorithms, the Web is the place to go for optimization software. In addition to numerous sources of optimization codes and optimizationrelated home pages, the Web is increasingly a source of optimization services that let you try out solvers without having to install them on your own equipment.
Online sources of optimization services
The following websites offer, in some sense, to run your nonlinear programming problem and return a result. Check their home pages for details, which vary considerably. (See also the analogous listing in the LP FAQ.)
 AMPL. A convenient interface supports experimentation with the AMPL modeling language on test problems up to 300 variables and 300 constraints + objectives. Users can request any example from the AMPL book, or provide their own model and data files. There is a choice among 8 solvers for nonlinear optimization and complementarity problems (as well as linear and integer programs).
 APMonitor. The APMonitor modeling language solves problems with nonlinear differential and algebraic (DAE) equations. The online trial interface allows unlimited size algebraic models (up to 10 million variables), using IPOPT with MUMPS. The full commercial version includes:
 Nonlinear constrained optimization
 Dynamic simulation
 Dynamic parameter estimation (moving horizon estimation)
 Model predictive control
 BARON. General and various specialized problems of nonlinear integer programming can be submitted in a simple algebraic format, through a web form or from a local file. Prospective users must first request a password.
 FSQP and related codes. Nonlinear problems in up to 5 variables may be submitted in a simple algebraic format.
 Kestrel. This interface permits varied linear, integer, and nonlinear optimization problems to be sent directly from the AMPL and GAMS modeling systems to the NEOS Server. Results are sent back to AMPL or GAMS, where they may be displayed and analyzed just as if the solving had been done locally.
 NetworkEnabled Optimization System (NEOS) Server. Offers access to several dozen solvers for linear and nonlinear programming, network and stochastic linear programming, unconstrained and boundconstrained optimization of nonlinear functions, and nonlinear complementarity.
 Nonlinear problems are accepted in the form of a C or Fortran program or in modeling languages such as AMPL and GAMS.
 Derivatives are calculated automatically on the server, using ADIOR for Fortran code, ADOLC for C code, and facilities built into the modeling language translators.
 Submissions may be made by sending email, by submitting names of local files through a Web form, or via a highspeed socketbased Unix interface.
 NIMBUS. A nondifferentiable multiobjective optimization system that accepts algebraic problem specifications through a series of Web forms.
 NumaWWW  Numerische Mathematik Interaktiv includes teaching tools for various linear and nonlinear optimization methods (in German).
 UniCalc. A variety of nonlinear optimization problems may be submitted in UniCalc's algebraic modeling language, through a web form interface.
Online sources of optimization software
The Netlib Repository contains a huge collection of mathematical software, papers, and databases, maintained at the University of Tennessee, Knoxville and Oak Ridge National Laboratory. There are also mirror sites in the United States, Norway, the United Kingdom, and other locations. Once you know what you want from Netlib, you may may prefer to make requests by anonymous ftp or email; alternative access methods and many other relevant matters are explained in the Netlib FAQ.
Many optimization packages are distributed from their own Web sites. Numerous links to these sites are provided elsewhere in this FAQ.
Online sources of optimization information
Varied general sources on optimization include information on nonlinear programming. They include:
 The Optimization Online repository of papers and reports on optimization. Contributions are accepted from anyone working in the field, and are moderated by a team of volunteer coordinators; you can subscribe to a monthly digest that summarizes reports recently received.
 The eOptimization.com community, a web meeting place for optimization users and developers, featuring a who's who of people in optimization as well as listings of optimization resources, applications, vendors, case studies, and news.
 The NEOS Guide, the companion site to the NEOS Server.
 The Decision Tree for Optimization Software by Hans Mittelmann and P. Spellucci, and the related Benchmarks for Optimization Software.
 Harvey Greenberg's Mathematical Programming Glossary and Bibliography for the Development of An Intelligent Mathematical Programming System.
 Listings of optimization software and related information at the Center for Advanced Modeling and Optimization.
More specialized sources include:
 The Global Optimization website maintained by Arnold Neumaier.
 InteriorPoint Methods Online, containing most new reports in the area of interiorpoint methods that have appeared since December 1994. Abstracts for all reports are available, as are links to postscript source for most reports; a mailing list carries announcements of new papers, conferences, software, and special journal issues relevant to interiorpoint methods.
 A bibliography of books and papers on InteriorPoint methods (including many older ones) compiled by Eberhard Kranich.
 A resource website in semidefinite programming maintained by Christoph Helmberg, and a semidefinite programming bibliography by Henry Wolkowicz.
 The Stochastic Programming Community Home Page, with numerous links and several tutorials. Also an extensive bibliography for stochastic programming and a specialized one for stochastic integer programming, compiled by Maarten van der Vlerk.
The following sites cover operations research and management science more generally, but have a large amount of content relevant to nonlinear programming and other optimization problems:
 INFORMS Online, the website of the Institute for Operations Research and the Management Sciences, including:
 Searchable databases of INFORMS conference presentations going back to 1990, and publications in operations research back to 1982.
 Michael Trick's Operations Research Blog.
 WORMS (WorldWideWeb for Operations Research and Management Science) at the Department of Mathematics and Statistics, University of Melbourne, Australia.
The following cover nonlinear optimization as part of broader mathematical fields:
 The Mathematical Optimization page at Oak Ridge National Laboratory.
 The Computational Mathematics Archive at the London and SouthEast Centre for High Performance Computing.
 Nonlinear Science Today, an electronic adjunct of the Journal of Nonlinear Science.
Other Online Resources for Optimization
 The related Linear Programming FAQ.
 The Mathematical Programming Glossary.
 The Optimization Online repository of reports and papers.
 The eOptimization.community website.
 The NEOS Guide to optimization models and software.
 The Decision Tree for Optimization Software by H.D. Mittelmann and P. Spellucci.
 Software for Optimization: A Buyer's Guide by Robert Fourer.
 The NEOS Server for access to optimization solvers through the internet
 Hans Mittelmann's Benchmarks for Optimization Software.
 The Computational Optimization and Applications Software Forum
Finally and don't forget the Web search engines! Services such as Google (especially Google Scholar), Bing, and Yahoo can be surprisingly helpful in finding pages on particular problem types or application areas.
Who maintains this FAQ list?
This list is no longer maintained. It was maintained for many years by Robert Fourer (4er@iems.northwestern.edu) and the Optimization Technology Center of Northwestern University and Argonne National Laboratory. This article is Copyright © 2000 by Robert Fourer. It may be freely redistributed in its entirety provided that this copyright notice is not removed. It may not be sold for profit or incorporated in commercial documents without the written permission of the copyright holder. Permission is expressly granted for this document to be made available for file transfer from installations offering unrestricted anonymous file transfer on the Internet.The material in this document does not reflect any official position taken by any organization. While all information in this article is believed to be correct at the time of writing, it is provided "as is" with no warranty implied.If you wish to cite this FAQ formally  this may seem strange, but it does come up  you may use:
Robert Fourer (4er@iems.northwestern.edu), "Nonlinear Programming Frequently Asked Questions," Optimization Technology Center of Northwestern University and Argonne National Laboratory, (2000).