[Includes #3960] More flexible `LinearStandardFormCompiler` by bknueven · Pull Request #3949 · Pyomo/pyomo

bknueven · 2026-05-12T16:24:21Z

Fixes # N/A

Summary/Motivation:

LinearStandardFormCompiler is a great tool for rapidly expanding a Pyomo model into the form need for (mixed-integer) linear programming solvers. Many of these solvers also handle ranged (linear) constraints, quadratic constraints, SOS constraints, quadratic objectives, and even sometimes general nonlinear constraints and objectives. This PR would augment the LinearStandardFormCompiler two significant ways:

Functional forms it cannot compile directly are either passed on or ignored and do not raise an error (new option allow_nonlinear and extra_valid_ctypes).
Ranged linear constraints are directly returned as part of the LinearStandardFormInfo when the new option keep_range_constraints=True.

See the usage proof of concept on #3879, which includes these changes.

Changes proposed in this PR:

Add keep_range_constraints to LinearStandardFormCompiler and compile range constraints appropriately (84afd9a, e330102, efc1c74)
Add option for LinearStandardFormCompiler to return non-linear constraints / objectives it encounters instead of raising an error (option allow_nonlinear aa9ce77)
Add option for additional valid ctypes, to be handled by the caller. Useful for SOS constraints (36953d1).
Bugfix for constraints with +/-inf (a13c2e0, c0741a8)
Bugfix: allow TemplateVarRecorder to correctly handle pyomo.kernel variables (6a2c69a)

This PR was created with the help of GitHub Copilot CLI.

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the BSD license.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

…aints

codecov · 2026-05-14T06:28:09Z

Codecov Report

❌ Patch coverage is 95.87629% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.13%. Comparing base (dccdbdd) to head (436284b).

Files with missing lines	Patch %	Lines
pyomo/repn/plugins/standard_form.py	95.65%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3949      +/-   ##
==========================================
+ Coverage   90.11%   90.13%   +0.01%     
==========================================
  Files         905      905              
  Lines      107502   107579      +77     
==========================================
+ Hits        96878    96964      +86     
+ Misses      10624    10615       -9

Flag	Coverage Δ
builders	`29.11% <4.12%> (-0.02%)`	⬇️
default	`86.47% <95.87%> (?)`
expensive	`35.50% <4.12%> (?)`
linux	`87.62% <95.87%> (-2.00%)`	⬇️
linux_other	`87.62% <95.87%> (+0.01%)`	⬆️
oldsolvers	`28.05% <4.12%> (-0.02%)`	⬇️
osx	`83.00% <95.87%> (+0.01%)`	⬆️
win	`85.55% <95.87%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jsiirola

I like this approach. I am still digging through the review, but in the interests of getting things moving one big initial question: have you quantified how this change impacts the standard_form performance?

jsiirola · 2026-05-13T17:25:21Z

+        ),
+    )
+    CONFIG.declare(
+        'extra_valid_ctypes',


Potentially rename this ignore_ctypes? Should this be promoted to an ADVANCED_OPTION?

I think that name is better. Should any other options be promoted to an ADVANCED_OPTION?

michaelbynum

This is a clever way to enable using the standard form compiler with solvers that support nonlinear expressions. Nice work.

Question - are you actually using kernel, or were you just trying to get tests to pass?

michaelbynum · 2026-05-19T12:26:05Z

+                        offset, linear_index, linear_data, lb, ub = (
+                            template_visitor.expand_expression(obj, obj.template_expr())
+                        )
+                    except InvalidExpressionError:


Is it possible to get an InvalidExpressionError for some reason other than a nonlinear expression?

I think I am not terribly concerned about this. If this error is raised for some other reason, then the solver interface who is handling the nonlinear constraints will have to deal with this.

I did check with template constraints on a small NLP, and this is in fact the exception we get on main:

ERROR: Error compiling expanded template expressions for TemplateScalarConstraint 'c' LinearTemplateRepn does not support expressions containing general nonlinear terms. --------------------------------------------------------------------------- KeyError Traceback (most recent call last) File ~/Software/pyomo/pyomo/repn/linear_template.py:380, in LinearTemplateRepnVisitor.expand_expression(self, obj, template_info) 379 try: --> 380 body, lb, ub = self.expanded_templates[id(template_info)] 381 except KeyError: KeyError: 6408477952 During handling of the above exception, another exception occurred: InvalidExpressionError Traceback (most recent call last) Cell In[22], line 1 ----> 1 repn = LinearStandardFormCompiler().write(m) File ~/Software/pyomo/pyomo/repn/plugins/standard_form.py:268, in LinearStandardFormCompiler.write(self, model, ostream, **options) 264 # Pause the GC, as the walker that generates the compiled LP 265 # representation generates (and disposes of) a large number of 266 # small objects. 267 with PauseGC(): --> 268 return _LinearStandardFormCompiler_impl(config).write(model) File ~/Software/pyomo/pyomo/repn/plugins/standard_form.py:432, in _LinearStandardFormCompiler_impl.write(self, model) 428 last_parent = con._component 430 if hasattr(con, 'template_expr'): 431 offset, linear_index, linear_data, lb, ub = ( --> 432 template_visitor.expand_expression(con, con.template_expr()) 433 ) 434 N = len(linear_data) 435 else: 436 # Note: lb and ub could be a number, expression, or None. 437 # Non-fixed expressions will raise an InvalidConstraintError. File ~/Software/pyomo/pyomo/repn/linear_template.py:383, in LinearTemplateRepnVisitor.expand_expression(self, obj, template_info) 381 except KeyError: 382 try: --> 383 body, lb, ub = self._generate_expanded_template(obj, template_info) 384 except: 385 # Provide some context clues for what component failed 386 # compilation 387 msg = sys.exc_info()[1] File ~/Software/pyomo/pyomo/repn/linear_template.py:448, in LinearTemplateRepnVisitor._generate_expanded_template(self, obj, template_info) 446 env = self.env 447 if body is not None: --> 448 body = self.walk_expression(body).compile( 449 self.env, smap, self.expr_cache, args 450 ) 451 if lb is not None: 452 lb = self.walk_expression(lb).compile( 453 self.env, smap, self.expr_cache, args 454 ) File ~/Software/pyomo/pyomo/core/expr/visitor.py:274, in StreamBasedExpressionVisitor.walk_expression(self, expr) 271 root = expr 273 try: --> 274 result = self._process_node(root, RECURSION_LIMIT) 275 _nonrecursive = None 276 except RevertToNonrecursive: File ~/Software/pyomo/pyomo/core/expr/visitor.py:440, in StreamBasedExpressionVisitor._process_node_bex(self, node, recursion_limit) 436 args.__exit__(None, None, None) 438 # We are done with this node. Call exitNode to compute 439 # any result --> 440 return self.exitNode(node, data) File ~/Software/pyomo/pyomo/repn/linear.py:868, in LinearRepnVisitor.exitNode(self, node, data) 864 return data.walker_exitNode() 865 # 866 # General expressions... 867 # --> 868 return self.exit_node_dispatcher[(node.__class__, *map(itemgetter(0), data))]( 869 self, node, *data 870 ) File ~/Software/pyomo/pyomo/repn/linear.py:305, in _handle_product_nonlinear(visitor, node, arg1, arg2) 303 ans = visitor.Result() 304 if not visitor.expand_nonlinear_products: --> 305 ans.nonlinear = to_expression(visitor, arg1) * to_expression(visitor, arg2) 306 return _GENERAL, ans 307 # 308 # We are multiplying (and expanding) m(A + Bx + C(x)) * m(A + Bx + C(x)) File ~/Software/pyomo/pyomo/repn/linear.py:239, in to_expression(visitor, arg) 237 return arg[1] 238 else: --> 239 return arg[1].to_expression(visitor) File ~/Software/pyomo/pyomo/repn/linear_template.py:104, in LinearTemplateRepn.to_expression(self, visitor) 99 def to_expression(self, visitor): 100 # to_expression() is only used by the underlying 101 # LinearRepnVisitor to generate nonlinear expressions. We are 102 # explicitly disallowing nonlinear expressions here, so we are 103 # going to bail now: --> 104 raise InvalidExpressionError( 105 "LinearTemplateRepn does not support expressions containing " 106 "general nonlinear terms." 107 ) InvalidExpressionError: LinearTemplateRepn does not support expressions containing general nonlinear terms.

michaelbynum · 2026-05-19T12:30:30Z

+                        offset, linear_index, linear_data, lb, ub = (
+                            template_visitor.expand_expression(con, con.template_expr())
+                        )
+                    except InvalidExpressionError:


Same question here: Is there a way to get an InvalidExpressionError for a reason other than a nonlinear expression?

michaelbynum · 2026-05-19T12:32:08Z

                linear_index = map(var_recorder.var_order.__getitem__, repn.linear)
                linear_data = repn.linear.values()

+            # Normalize ±inf to None: both kernel constraints and AML


I don't normally see symbols like the +/- symbol here (which show up in several places). Will this cause problems? In other words, do we have any reason to limit ourselves to ASCII (I believe these characters are UTF-8)?

@mrmundt - do you have any thoughts on this?

We generally don't have a problem with UTF-8 characters in files (several others already have them). However, the file needs to start with the following line so that editors and interpreters parse it correctly (without complaint).

# -*- coding: utf-8 -*-

michaelbynum · 2026-05-19T12:52:22Z

+        constraint is not misclassified as a range constraint, and a fully
+        unbounded constraint is skipped rather than emitted as a range row.
+        """
+        import pyomo.kernel as pmo


Do we want new tests for kernel?

@blnicho, @jsiirola - this is really a question for you.

michaelbynum · 2026-05-19T12:55:18Z

+        mk = pmo.block()
+        mk.x = pmo.variable()
+        # lb=-inf (unbounded below) → should become a pure ≤ row, not a range row
+        mk.c_ub = pmo.constraint(ub=2.0, body=mk.x)
+        # ub=+inf (unbounded above) → should become a pure ≥ row, not a range row
+        mk.c_lb = pmo.constraint(lb=-3.0, body=mk.x)
+        # Explicit finite range → should still be a range row
+        mk.c_rng = pmo.constraint((-1.0, mk.x, 4.0))


I was surprised that I didn't see any inf bounds here given the name of the test and the docstring for the function. This is not necessarily a bad test, but I don't see how it tests inf bounds.

michaelbynum · 2026-05-19T12:57:56Z

+        when ``allow_nonlinear=False`` (the default).  When
+        ``allow_nonlinear=True``, holds the list of objectives with nonlinear
+        terms that were omitted from the compiled matrices (may be empty).
+


Should this class also contain a list of the active ctypes that it found but ignored (when extra_valid_ctypes is used)?

This can always be added later if we think it is a good idea. It is really just for convenience anyway.

michaelbynum · 2026-05-19T13:01:28Z

        except AttributeError:
-            # Note that this only works for the AML, as kernel does not
-            # provide a parent_component()
-            _iter = (None, var)


I don't understand how this worked previously. Wouldn't an error get raised below because there was no tuple to unpack?

michaelbynum · 2026-05-19T13:03:18Z

I don't see anything that could impact performance unless allow_nonlinear is set to True. Well, there is the try/except statement in repn/util - I don't know if that could impact things or not.

bknueven · 2026-05-19T19:15:23Z

I like this approach. I am still digging through the review, but in the interests of getting things moving one big initial question: have you quantified how this change impacts the standard_form performance?

I have not done this. My hope would be that with the defaults there should not be a huge performance impact, but obviously there's more branches in this version.

jsiirola · 2026-05-19T20:01:57Z

OK - this PR does two things:

adds support for preserving ranged constraints as single rows in the "A" matrix
adds support for tabulating constraints that cannot be converted to simple linear constraints.

I really like the first, and can see the value in the second. However, combining them in a single PR is making the review (more) complicated. In particular, I am wary of the extra branching and logic in the tight loops for processing constraints. What would you think of the following:

split this into 2 PRs: one for ranged linear constraints and one for handling nonlinear constraints?
- I think the ranged review could be very fast.
- I would like to spend more time discussing / designing the nonlinear handling. For example,
  - should it be reworked so that nonllinear support is implemented in a derived class? That would avoid a lot of the branching / complexity / repeated logic in this implementation.
  - I also would rather promote the handling into the base walkers instead of running the linear walker and then catching the exception.

jsiirola · 2026-05-20T14:50:40Z

+            if lb == -float('inf'):
+                lb = None
+            if ub == float('inf'):
+                ub = None


What's the path for getting +/-inf here?

bknueven · 2026-05-20T16:40:23Z

OK - this PR does two things:

* adds support for preserving ranged constraints as single rows in the "A" matrix

* adds support for tabulating constraints that cannot be converted to simple linear constraints.

I really like the first, and can see the value in the second. However, combining them in a single PR is making the review (more) complicated. In particular, I am wary of the extra branching and logic in the tight loops for processing constraints. What would you think of the following:

* split this into 2 PRs: one for ranged linear constraints and one for handling nonlinear constraints?
  
  * I think the ranged review could be very fast.
  * I would like to spend more time discussing / designing the nonlinear handling.  For example,
    
    * should it be reworked so that nonllinear support is implemented in a derived class?  That would avoid a lot of the branching / complexity / repeated logic in this implementation.
    * I also would rather promote the handling into the base walkers instead of running the linear walker and then catching the exception.

Thanks for having a look @jsiirola. I will create a separate PR for just the ranged linear constraints change. If for some reason that doesn't move expeditiously I'll return to this PR and remove the ranged linear constraint changes here.

bknueven · 2026-05-21T16:41:32Z

Closing -- replaced by #3960 and #3963. @jsiirola @michaelbynum I have attempted to address you initial comments in those PRs.

Bernard Knueven added 12 commits May 11, 2026 12:27

add keep_range_constraints to LinearStandardFormCompiler

84afd9a

NFC: formatting

28e3529

allow LinearStandardFormCompiler to return unhandled nonlinear constr…

aa9ce77

…aints

cleanup range constraints

e330102

update standard form test for better range constraint handling

efc1c74

NFC: black on standard_form.py

ea34736

allow SOS constraints (and other ctypes) to pass for standard form

36953d1

NFC: black

e431ec9

handle kernel range constraints in LSFC

a13c2e0

bugfix: standard form normalizes constraints with +/-inf bounds

c0741a8

fix bug pyomo.kernel variabls in TemplateVarRecorder

6a2c69a

update LSFI to be self-consistent

ca7f765

bknueven mentioned this pull request May 12, 2026

[Depends on #3960 and #3963] Performance enhancements for XpressDirect / XpressPersistent for large LPs / MIPs #3879

Draft

Merge branch 'main' into flexible-linear-standard-repn

436284b

jsiirola added the AT: PRE-TEST INSPECTED label May 14, 2026

pyomo-autotest removed the AT: PRE-TEST INSPECTED label May 14, 2026

jsiirola reviewed May 18, 2026

View reviewed changes

michaelbynum reviewed May 19, 2026

View reviewed changes

Merge branch 'main' into flexible-linear-standard-repn

e45195f

jsiirola reviewed May 20, 2026

View reviewed changes

bknueven marked this pull request as draft May 20, 2026 22:00

bknueven changed the title ~~More flexible LinearStandardFormCompiler~~ [Includes #3960] More flexible LinearStandardFormCompiler May 20, 2026

bknueven closed this May 21, 2026

bknueven deleted the flexible-linear-standard-repn branch May 21, 2026 16:41

Conversation

bknueven commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fixes # N/A

Summary/Motivation:

Changes proposed in this PR:

Legal Acknowledgement

Uh oh!

codecov Bot commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jsiirola left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelbynum left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelbynum commented May 19, 2026

Uh oh!

bknueven commented May 19, 2026

Uh oh!

jsiirola commented May 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bknueven commented May 20, 2026

Uh oh!

bknueven commented May 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bknueven commented May 12, 2026 •

edited

Loading

codecov Bot commented May 14, 2026 •

edited

Loading