Feat/lazy expression container

doc/api.rst

@@ -247,6 +247,61 @@ Post-solve access
    expressions.Expressions.solution
 
 
+LazyExpression
+==============
+
+Placeholder for an expression that is built on demand. Returned by
+:meth:`Model.add_expressions <linopy.model.Model.add_expressions>` when
+`data` is a callable; arithmetic on a ``LazyExpression`` returns another
+``LazyExpression`` rather than evaluating immediately.
+
+.. autosummary::
+   :toctree: generated/
+
+   expressions.LazyExpression
+
+Evaluation
+----------
+
+.. autosummary::
+   :toctree: generated/
+
+   expressions.LazyExpression.evaluate
+   expressions.LazyExpression.promote
+   expressions.LazyExpression.is_evaluatable
+
+Arithmetic and constraints
+---------------------------
+
+Named counterparts of the arithmetic dunders, shared with
+``LinearExpression``/``QuadraticExpression`` via
+:class:`AbstractExpression <linopy.expressions.AbstractExpression>`. These
+stay lazy where possible; ``to_constraint``/``le``/``ge``/``eq`` (and the
+comparison operators) force evaluation and return a ``Constraint``.
+
+.. autosummary::
+   :toctree: generated/
+
+   expressions.LazyExpression.add
+   expressions.LazyExpression.sub
+   expressions.LazyExpression.mul
+   expressions.LazyExpression.div
+   expressions.LazyExpression.pow
+   expressions.LazyExpression.dot
+   expressions.LazyExpression.le
+   expressions.LazyExpression.ge
+   expressions.LazyExpression.eq
+   expressions.LazyExpression.to_constraint
+
+Post-solve access
+-----------------
+
+.. autosummary::
+   :toctree: generated/
+
+   expressions.LazyExpression.solution
+
+
 LinearExpression
 ================
 
@@ -687,3 +742,13 @@ These warning classes can be silenced or filtered via
 
    EvolvingAPIWarning
    PerformanceWarning
+   NonLinearExpressionWarning
+
+
+Exceptions
+==========
+
+.. autosummary::
+   :toctree: generated/
+
+   NonLinearOperationError

doc/release_notes.rst

@@ -5,6 +5,15 @@ Upcoming Version
 ----------------
 
 
+*Named expressions*
+
+* ``Model.add_expressions`` also accepts a callable for ``data``, in which case the expression is not built immediately: a ``LazyExpression`` placeholder is registered instead, and the callable (``data(model, **params)``) only runs when the expression is evaluated, via ``.evaluate()``, ``.promote()``, ``.solution``, or a comparison (``<=``, ``>=``, ``==``). ``mask`` accepts a callable too (resolved at the same time as `data`), in addition to a concrete array. Arithmetic between ``LazyExpression`` objects — and between a ``LazyExpression`` and anything else — stays lazy, returning a new, unnamed ``LazyExpression`` that composes the operands rather than evaluating them.
+  ``Model.to_netcdf`` gained a ``lazy={"evaluate", "skip", "raise"}`` parameter (default ``"evaluate"``) controlling what happens to lazy entries, since an arbitrary evaluator callable cannot itself be serialized to netcdf.
+* ``LazyExpression`` now shares its arithmetic/constraint protocol with ``LinearExpression``/``QuadraticExpression`` via a common ``AbstractExpression`` base, and gained the named counterparts (``add``, ``sub``, ``mul``, ``div``, ``pow``, ``dot``, ``le``, ``ge``, ``eq``, ``to_constraint``) that were previously eager-only — a ``LazyExpression`` is now a drop-in substitute for an eager expression wherever those are called, including with a ``join`` argument. ``lazy ** 2`` no longer evaluates the underlying evaluator twice, and ``-lazy`` now matches eager negation exactly (previously it filled masked/NaN coefficients with 0 before negating, like ``lazy * -1`` does).
+* Dividing a ``LazyExpression`` by a variable or another expression (e.g. ``cost / output`` for a unit cost), and other operations with no linear/quadratic form (e.g. ``lazy ** 3``), no longer raise immediately: they build a ``LazyExpression`` that is only readable via ``.solution`` once the model has been solved, since every operand is then just a number. ``.evaluate()``, ``.promote()`` and constraint-building still raise — now a dedicated ``linopy.NonLinearOperationError`` (a ``TypeError`` subclass, so existing ``except TypeError`` code keeps working) — pointing at ``.solution`` instead. Where this is already decidable at construction time (as opposed to only inside a leaf callable's body), a ``linopy.NonLinearExpressionWarning`` is raised immediately, and ``LazyExpression.is_evaluatable`` reports it without forcing evaluation.
+* A lazy expression's callable may also read post-solve-only data that is not itself an expression — most notably a constraint's ``.dual`` — and return a plain ``DataArray``/constant, or a ``LazyExpression`` built from one. It is then only readable via ``.solution``; ``.promote()`` raises since there is nothing to store as a named expression.
+
+
 Version 0.9.1
 -------------
 
@@ -47,7 +56,6 @@ Version v0.9.0
 
 * ``Model.to_netcdf`` now records the writing linopy version in the ``_linopy_version`` dataset attribute. Files written by older versions (without the attribute) continue to read unchanged. (`#780 <https://github.com/PyPSA/linopy/pull/780>`__)
 
-
 *Other*
 
 * Default internal integer labels to ``int32``, cutting memory ~25% and speeding up model build 10-35%. Models exceeding the int32 maximum (~2.1 billion labels) widen to ``int64`` automatically with a ``UserWarning``; pass ``Model(dtypes={"labels": np.int64})`` upfront to avoid the mid-build upcast (exposed read-only via ``Model.dtypes``). (`#566 <https://github.com/PyPSA/linopy/pull/566>`__)

examples/creating-expressions.ipynb

@@ -575,6 +575,229 @@
     "   optimal solution — handy for inspecting derived quantities without\n",
     "   rebuilding the expression by hand."
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "53",
+   "metadata": {},
+   "source": [
+    "## Deferred (lazy) expressions\n",
+    "\n",
+    "Passing a callable to ``data`` builds the expression on demand instead of\n",
+    "right away. ``Model.add_expressions`` registers a ``LazyExpression``\n",
+    "placeholder and only calls the callable once the expression is actually\n",
+    "needed — via ``.evaluate()``, ``.promote()``, ``.solution``, or a comparison.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "54",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "deferred = m.add_expressions(lambda model: model.variables[\"x\"] * 3, name=\"deferred\")\n",
+    "deferred"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "55",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "deferred.evaluate()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "56",
+   "metadata": {},
+   "source": [
+    "Extra keyword arguments are forwarded to the callable every time it runs,\n",
+    "and ``mask`` may itself be a callable, resolved at the same time:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "57",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def scaled(model, factor):\n",
+    "    return model.variables[\"x\"] * factor\n",
+    "\n",
+    "\n",
+    "scaled_expr = m.add_expressions(scaled, name=\"scaled\", factor=5)\n",
+    "scaled_expr.evaluate()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "58",
+   "metadata": {},
+   "source": [
+    "Arithmetic between lazy expressions — and between a lazy expression and\n",
+    "anything else — stays lazy: it returns a new, unnamed ``LazyExpression``\n",
+    "that composes the operands, rather than evaluating them immediately.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "59",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "combined = deferred + scaled_expr\n",
+    "combined  # still a LazyExpression"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "60",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "combined.evaluate()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "61",
+   "metadata": {},
+   "source": [
+    "Named methods work the same way as on eager expressions, including\n",
+    "the ``join`` parameter, and stay lazy too:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "62",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "combined.add(1, join=\"outer\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "63",
+   "metadata": {},
+   "source": [
+    "``.promote()`` runs the evaluator once and replaces the placeholder\n",
+    "in-place with the resulting concrete expression:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "64",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "deferred.promote()\n",
+    "m.expressions[\"deferred\"]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "65",
+   "metadata": {},
+   "source": [
+    "## Ratios and other post-solve-only expressions\n",
+    "\n",
+    "Dividing by a variable or another expression has no linear or quadratic\n",
+    "form, so it cannot be built into a LinearExpression/QuadraticExpression.\n",
+    "On a lazy expression it no longer raises outright, though: it defers to a\n",
+    "LazyExpression that is only readable via .solution, once the model has\n",
+    "been solved and every operand is just a number.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "66",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "cost = m.add_expressions(lambda model: 3 * model.variables[\"x\"], name=\"cost\")\n",
+    "output = m.add_expressions(lambda model: model.variables[\"x\"], name=\"output\")\n",
+    "unit_cost = cost / output\n",
+    "unit_cost  # still a LazyExpression -- a NonLinearExpressionWarning was also raised"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "67",
+   "metadata": {},
+   "source": [
+    ".evaluate(), .promote() and constraint-building all raise a\n",
+    "linopy.NonLinearOperationError (a TypeError subclass) for such an\n",
+    "expression, pointing at .solution instead:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "68",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "try:\n",
+    "    unit_cost.evaluate()\n",
+    "except Exception as e:\n",
+    "    print(f\"{type(e).__name__}: {e}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "69",
+   "metadata": {},
+   "source": [
+    "unit_cost.is_evaluatable reports this without forcing an evaluation, and\n",
+    "once the model is solved, .solution works like any other expression—\n",
+    "including through m.expressions.solution, since cost and output\n",
+    "are themselves stored on the model:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "70",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "unit_cost.is_evaluatable"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "71",
+   "metadata": {},
+   "source": [
+    "A lazy expression may also read data that only exists post-solve and is not\n",
+    "itself an expression — most notably a constraint’s .dual — and return a\n",
+    "plain DataArray. It is likewise only readable via .solution:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "72",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "m.add_constraints(x >= 2, name=\"x_lower_bound\")\n",
+    "shadow_price = m.add_expressions(\n",
+    "    lambda model: model.constraints[\"x_lower_bound\"].dual, name=\"shadow_price\"\n",
+    ")\n",
+    "shadow_price"
+   ]
   }
  ],
  "metadata": {
@@ -597,7 +820,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.3"
+   "version": "3.11.6"
   }
  },
  "nbformat": 4,

linopy/__init__.py

@@ -19,6 +19,8 @@
     GREATER_EQUAL,
     LESS_EQUAL,
     EvolvingAPIWarning,
+    NonLinearExpressionWarning,
+    NonLinearOperationError,
     PerformanceWarning,
 )
 from linopy.constraints import (
@@ -27,7 +29,12 @@
     Constraints,
     CSRConstraint,
 )
-from linopy.expressions import LinearExpression, QuadraticExpression, merge
+from linopy.expressions import (
+    LazyExpression,
+    LinearExpression,
+    QuadraticExpression,
+    merge,
+)
 from linopy.io import read_netcdf
 from linopy.model import Model, Variable, Variables
 from linopy.objective import Objective
@@ -56,8 +63,11 @@
     "EvolvingAPIWarning",
     "GREATER_EQUAL",
     "LESS_EQUAL",
+    "LazyExpression",
     "LinearExpression",
     "Model",
+    "NonLinearExpressionWarning",
+    "NonLinearOperationError",
     "Objective",
     "OetcHandler",
     "PiecewiseFormulation",