Issues with new solver interface for IPOPT #3736
Description
Summary
Recently, when updating a contribution to the IDAES-Examples repo to use the new solver interface, we encountered several errors.
- The first issue is an error in the IPOPT output log parser. When the option
"OF_print_info_string": "yes",is passed, there is an additional column in the IPOPT output:
iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls
0 4.3126674e+00 1.34e+00 1.00e+00 -5.0 0.00e+00 - 0.00e+00 0.00e+00 0
Reallocating memory for MA57: lfact (2247250)
1r 4.3126674e+00 1.34e+00 9.99e+02 0.1 0.00e+00 -4.0 0.00e+00 3.29e-10R 2
2r 3.0519246e+08 1.13e+00 9.90e+02 0.1 2.30e+02 - 2.60e-02 9.32e-03f 1
3r 2.2712595e+09 1.69e+00 9.73e+02 0.1 2.23e+02 - 2.54e-02 1.71e-02f 1 Nhj
4 2.2712065e+09 1.69e+00 1.37e+09 -5.0 3.08e+03 - 1.32e-05 1.17e-05f 1 q
5 1.9062986e+09 1.55e+00 1.25e+09 -5.0 5.13e+03 - 1.19e-01 8.38e-02f 1This leads to an error in the log parser, because any lines in the log that don't have the expected number of columns are skipped:
for line in iter_table:
tokens = line.strip().split()
if len(tokens) != len(columns):
continue
- The second issue is an error in the solver reader. When dual variables are requested for a problem without an objective function, this line causes an exception:
obj_scale = self._nl_info.scaling.objectives[0]
For problems without objectives, either 1) the dual variables are zero or 2) the problem is infeasible. Nevertheless, a better error message should be raised.
Error Message
IPOPT output log parser error:
Solving controller model at time = 0.000000
Ipopt 3.13.2: linear_solver="ma57"
max_iter=1000
nlp_scaling_method="gradient-based"
tol=0.0001
halt_on_ampl_error="no"
mu_init=1e-05
bound_push=1e-06
bound_relax_factor=1e-06
ma57_pivtol=1e-06
warm_start_init_point="yes"
warm_start_mult_bound_push=1e-06
warm_start_bound_push=1e-06
option_file_name="C:\Users\dallan\AppData\Local\Temp\tmps0xkth8o\unknown.26400.33444.opt"
Using option file "C:\Users\dallan\AppData\Local\Temp\tmps0xkth8o\unknown.26400.33444.opt".
******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
Ipopt is released as open source code under the Eclipse Public License (EPL).
For more information visit http://projects.coin-or.org/Ipopt
This version of Ipopt was compiled from source code available at
https://github.com/IDAES/Ipopt as part of the Institute for the Design of
Advanced Energy Systems Process Systems Engineering Framework (IDAES PSE
Framework) Copyright (c) 2018-2019. See https://github.com/IDAES/idaes-pse.
This version of Ipopt was compiled using HSL, a collection of Fortran codes
for large-scale scientific computation. All technical papers, sales and
publicity material resulting from use of the HSL codes within IPOPT must
contain the following acknowledgement:
HSL, a collection of Fortran codes for large-scale scientific
computation. See http://www.hsl.rl.ac.uk/.
******************************************************************************
This is Ipopt version 3.13.2, running with linear solver ma57.
Number of nonzeros in equality constraint Jacobian...: 77541
Number of nonzeros in inequality constraint Jacobian.: 0
Number of nonzeros in Lagrangian Hessian.............: 51855
Total number of variables............................: 15468
variables with only lower bounds: 3491
variables with lower and upper bounds: 5026
variables with only upper bounds: 186
Total number of equality constraints.................: 15417
Total number of inequality constraints...............: 0
inequality constraints with only lower bounds: 0
inequality constraints with lower and upper bounds: 0
inequality constraints with only upper bounds: 0
iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls
0 4.3126674e+00 1.34e+00 1.00e+00 -5.0 0.00e+00 - 0.00e+00 0.00e+00 0
Reallocating memory for MA57: lfact (2247250)
1r 4.3126674e+00 1.34e+00 9.99e+02 0.1 0.00e+00 -4.0 0.00e+00 3.29e-10R 2
2r 3.0519246e+08 1.13e+00 9.90e+02 0.1 2.30e+02 - 2.60e-02 9.32e-03f 1
3r 2.2712595e+09 1.69e+00 9.73e+02 0.1 2.23e+02 - 2.54e-02 1.71e-02f 1 Nhj
4 2.2712065e+09 1.69e+00 1.37e+09 -5.0 3.08e+03 - 1.32e-05 1.17e-05f 1 q
5 1.9062986e+09 1.55e+00 1.25e+09 -5.0 5.13e+03 - 1.19e-01 8.38e-02f 1
6 1.7041594e+09 1.46e+00 1.18e+09 -5.0 5.66e+03 - 7.06e-02 5.45e-02f 1
7 1.4763158e+09 1.36e+00 1.10e+09 -5.0 3.94e+03 - 2.30e-01 6.92e-02f 1
8 8.5873108e+08 1.04e+00 8.41e+08 -5.0 2.38e+05 - 3.49e-06 2.37e-01f 1
9 4.4215572e+08 7.45e-01 6.03e+08 -5.0 1.63e+06 - 7.97e-02 2.82e-01f 1
iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls
10 5.0251884e+01 1.65e-01 1.57e+04 -5.0 1.24e+06 - 3.92e-05 1.00e+00f 1
11 4.9121733e+01 4.97e-02 4.68e+03 -5.0 8.11e+04 - 4.31e-02 7.01e-01h 1
12 4.1483985e+01 2.24e-02 5.97e+03 -5.0 1.15e+06 - 5.93e-02 1.00e+00f 1
13 3.5762585e+01 1.75e-02 5.00e+03 -5.0 1.03e+06 - 1.25e-01 1.00e+00f 1
14 3.2291014e+01 1.08e-02 3.51e+03 -5.0 8.25e+05 - 6.68e-01 1.00e+00f 1
15 3.2274630e+01 3.31e-05 1.17e+00 -5.0 4.26e+04 - 9.92e-01 1.00e+00h 1
16 3.2274631e+01 7.45e-09 2.71e-03 -5.0 6.11e+02 - 8.97e-01 1.00e+00h 1
17 3.2274635e+01 7.45e-09 2.35e-03 -5.0 2.71e+04 - 1.32e-01 1.00e+00f 1
18 3.2274635e+01 7.45e-09 1.15e-04 -5.0 5.53e+03 - 9.51e-01 1.00e+00h 1
19 3.2274635e+01 7.45e-09 2.84e-05 -5.0 4.41e+04 - 7.54e-01 1.00e+00f 1
iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls
20 3.2274635e+01 7.45e-09 8.54e-07 -5.0 1.83e+04 - 1.00e+00 1.00e+00h 1
Number of Iterations....: 20
(scaled) (unscaled)
Objective...............: 3.2274635418964841e+01 3.2274635418964841e+01
Dual infeasibility......: 8.5365078678328669e-07 8.5365078678328669e-07
Constraint violation....: 8.0780625068607930e-13 7.4505805969238281e-09
Complementarity.........: 1.2275904566414160e-05 1.2275904566414160e-05
Overall NLP error.......: 1.2275904566414160e-05 1.2275904566414160e-05
Number of objective function evaluations = 23
Number of objective gradient evaluations = 20
Number of equality constraint evaluations = 23
Number of inequality constraint evaluations = 0
Number of equality constraint Jacobian evaluations = 22
Number of inequality constraint Jacobian evaluations = 0
Number of Lagrangian Hessian evaluations = 20
Total CPU secs in IPOPT (w/o function evaluations) = 10.450
Total CPU secs in NLP function evaluations = 1.651
EXIT: Optimal Solution Found.
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
File ~\Repos\scratch\SOC-Jupyternotebook\rSOC.py:475
469 # ## Run NMPC
470
471 # In[8]:
474 economic = True
--> 475 data_dict_updated_v2 = run_nmpc_v2(economic= economic, sim_time_point = sim_time_point)
478 # In[ ]:
481 data_dict_updated_v1 = run_nmpc_v1(economic= economic, sim_time_point = sim_time_point)
File ~\Repos\scratch\SOC-Jupyternotebook\rSOC.py:422, in run_nmpc_v2(economic, sim_time_point)
420 # Solve the scaled controller model
421 print("Solving controller model at time = %f" %t_base)
--> 422 controller_results = solver_v2.solve(m_controller, tee= True)
423 pyo.assert_optimal_termination(controller_results)
425 # Apply control actions to the plant model
File ~\miniforge3\envs\idaes\lib\site-packages\pyomo\contrib\solver\common\base.py:685, in LegacySolverWrapper.solve(self, model, tee, load_solutions, logfile, solnfile, timelimit, report_timing, solver_io, suffixes, options, keepfiles, symbolic_solver_labels, raise_exception_on_nonoptimal_result, solver_options, writer_config)
682 filtered_args = {k: loc[k] for k in map_args if loc.get(k, None) is not None}
683 self._map_config(**filtered_args)
--> 685 results: Results = super().solve(model)
686 legacy_results, legacy_soln = self._map_results(model, results)
687 legacy_results = self._solution_handler(
688 load_solutions, model, results, legacy_results, legacy_soln
689 )
File ~\miniforge3\envs\idaes\lib\site-packages\pyomo\contrib\solver\solvers\ipopt.py:467, in Ipopt.solve(self, model, **kwds)
463 timer.stop('subprocess')
465 # This is the data we need to parse to get the iterations
466 # and time
--> 467 parsed_output_data = self._parse_ipopt_output(ostreams[0])
469 if proven_infeasible:
470 results = Results()
File ~\miniforge3\envs\idaes\lib\site-packages\pyomo\contrib\solver\solvers\ipopt.py:645, in Ipopt._parse_ipopt_output(self, output)
639 except (ValueError, TypeError):
640 logger.warning(
641 "Error converting Ipopt log entry to "
642 f"float:\n\t{sys.exc_info()[1]}\n\t{line}"
643 )
--> 645 assert len(iterations) == iter_data.pop('iter'), (
646 f"Parsed row in the iterations table\n\t{line}\ndoes not "
647 f"match the next expected iteration number ({len(iterations)})"
648 )
649 iterations.append(iter_data)
651 parsed_data['iteration_log'] = iterations
AssertionError: Parsed row in the iterations table
5 1.9062986e+09 1.55e+00 1.25e+09 -5.0 5.13e+03 - 1.19e-01 8.38e-02f 1
does not match the next expected iteration number (3)Dual variable interface error:
Solving plant model at time = 0.000000
Ipopt 3.13.2: linear_solver="ma57"
max_iter=1000
nlp_scaling_method="gradient-based"
tol=0.0001
halt_on_ampl_error="no"
mu_init=1e-05
bound_push=1e-06
bound_relax_factor=1e-06
ma57_pivtol=1e-06
warm_start_init_point="yes"
warm_start_mult_bound_push=1e-06
warm_start_bound_push=1e-06
option_file_name="C:\Users\dallan\AppData\Local\Temp\tmp4k0guac4\unknown.26340.17868.opt"
Using option file "C:\Users\dallan\AppData\Local\Temp\tmp4k0guac4\unknown.26340.17868.opt".
******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
Ipopt is released as open source code under the Eclipse Public License (EPL).
For more information visit http://projects.coin-or.org/Ipopt
This version of Ipopt was compiled from source code available at
https://github.com/IDAES/Ipopt as part of the Institute for the Design of
Advanced Energy Systems Process Systems Engineering Framework (IDAES PSE
Framework) Copyright (c) 2018-2019. See https://github.com/IDAES/idaes-pse.
This version of Ipopt was compiled using HSL, a collection of Fortran codes
for large-scale scientific computation. All technical papers, sales and
publicity material resulting from use of the HSL codes within IPOPT must
contain the following acknowledgement:
HSL, a collection of Fortran codes for large-scale scientific
computation. See http://www.hsl.rl.ac.uk.
******************************************************************************
This is Ipopt version 3.13.2, running with linear solver ma57.
Number of nonzeros in equality constraint Jacobian...: 34545
Number of nonzeros in inequality constraint Jacobian.: 0
Number of nonzeros in Lagrangian Hessian.............: 21570
Total number of variables............................: 8896
variables with only lower bounds: 1441
variables with lower and upper bounds: 3413
variables with only upper bounds: 60
Total number of equality constraints.................: 8896
Total number of inequality constraints...............: 0
inequality constraints with only lower bounds: 0
inequality constraints with lower and upper bounds: 0
inequality constraints with only upper bounds: 0
iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls
0 0.0000000e+00 1.82e-01 1.00e+00 -5.0 0.00e+00 - 0.00e+00 0.00e+00 0
Reallocating memory for MA57: lfact (707336)
1 0.0000000e+00 5.57e-03 8.49e+00 -5.0 7.33e-02 - 5.67e-01 1.00e+00h 1
2 0.0000000e+00 8.06e-06 1.47e-01 -5.0 7.33e-02 - 9.95e-01 1.00e+00h 1
Number of Iterations....: 2
(scaled) (unscaled)
Objective...............: 0.0000000000000000e+00 0.0000000000000000e+00
Dual infeasibility......: 0.0000000000000000e+00 0.0000000000000000e+00
Constraint violation....: 8.0648345965883550e-06 8.0648345965883550e-06
Complementarity.........: 0.0000000000000000e+00 0.0000000000000000e+00
Overall NLP error.......: 8.0648345965883550e-06 8.0648345965883550e-06
Number of objective function evaluations = 3
Number of objective gradient evaluations = 3
Number of equality constraint evaluations = 3
Number of inequality constraint evaluations = 0
Number of equality constraint Jacobian evaluations = 3
Number of inequality constraint Jacobian evaluations = 0
Number of Lagrangian Hessian evaluations = 2
Total CPU secs in IPOPT (w/o function evaluations) = 0.409
Total CPU secs in NLP function evaluations = 0.046
EXIT: Optimal Solution Found.
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
File ~\Repos\scratch\SOC-Jupyternotebook\rSOC.py:475
469 # ## Run NMPC
470
471 # In[8]:
474 economic = True
--> 475 data_dict_updated_v2 = run_nmpc_v2(economic= economic, sim_time_point = sim_time_point)
478 # In[ ]:
481 data_dict_updated_v1 = run_nmpc_v1(economic= economic, sim_time_point = sim_time_point)
File ~\Repos\scratch\SOC-Jupyternotebook\rSOC.py:443, in run_nmpc_v2(economic, sim_time_point)
441 # Solve plant model
442 print("Solving plant model at time = %f" %t_base)
--> 443 plant_results = solver_v2.solve(m_plant, tee= True)
444 pyo.assert_optimal_termination(plant_results)
446 # Shift time points in plant
File ~\miniforge3\envs\idaes\lib\site-packages\pyomo\contrib\solver\common\base.py:685, in LegacySolverWrapper.solve(self, model, tee, load_solutions, logfile, solnfile, timelimit, report_timing, solver_io, suffixes, options, keepfiles, symbolic_solver_labels, raise_exception_on_nonoptimal_result, solver_options, writer_config)
682 filtered_args = {k: loc[k] for k in map_args if loc.get(k, None) is not None}
683 self._map_config(**filtered_args)
--> 685 results: Results = super().solve(model)
686 legacy_results, legacy_soln = self._map_results(model, results)
687 legacy_results = self._solution_handler(
688 load_solutions, model, results, legacy_results, legacy_soln
689 )
File ~\miniforge3\envs\idaes\lib\site-packages\pyomo\contrib\solver\solvers\ipopt.py:538, in Ipopt.solve(self, model, **kwds)
532 results.solution_loader.load_vars()
533 if (
534 hasattr(model, 'dual')
535 and isinstance(model.dual, Suffix)
536 and model.dual.import_enabled()
537 ):
--> 538 model.dual.update(results.solution_loader.get_duals())
539 if (
540 hasattr(model, 'rc')
541 and isinstance(model.rc, Suffix)
542 and model.rc.import_enabled()
543 ):
544 model.rc.update(results.solution_loader.get_reduced_costs())
File ~\miniforge3\envs\idaes\lib\site-packages\pyomo\contrib\solver\solvers\sol_reader.py:141, in SolSolutionLoader.get_duals(self, cons_to_load)
139 else:
140 scale_list = self._nl_info.scaling.constraints
--> 141 obj_scale = self._nl_info.scaling.objectives[0]
142 if cons_to_load is None:
143 cons_to_load = set(self._nl_info.constraints)
IndexError: list index out of rangeInformation on your system
Pyomo version: 6.9.4
Python version: 3.10
Operating system: Windows 11
How Pyomo was installed: IDAES advanced user installation
Solver (if applicable): IPOPT_v2