Skip to content

Complete remote HTTPGenerator development and integration - #479

Open
ernestow wants to merge 4 commits into
xopt-org:http-generatorfrom
ernestow:http-generator-complete
Open

ernestow wants to merge 4 commits into
xopt-org:http-generatorfrom
ernestow:http-generator-complete

Conversation

@ernestow

Copy link
Copy Markdown

Overview

This PR continues the remote generator development initiated
by Sara Miskovich in pluflou#26, as part of our effort
to complete #469.

Sara's original commit and authorship have been preserved,
and her changes have been integrated with the upstream
http-generator branch.

Development and validation checklist

  • Integrate Sara's original development work
  • Preserve upstream Git history and authorship
  • Perform detailed source code review
  • Complete remaining development
  • Validate session recovery and data synchronization
  • Validate generator configuration and state management
  • Run unit and regression tests
  • Validate HTTPGenerator and CloudpickleHTTPGenerator
  • Validate CPU and GPU optimization workflows
  • Validate six concurrent independent optimization sessions
  • Validate session lifecycle and error handling
  • Validate documentation and example notebooks
  • Request final review

Related pull requests

The objective is to complete the HTTPGenerator implementation
and prepare PR #469 for final review and merging into main.

pluflou and others added 2 commits October 29, 2026 22:50
…hecks

Client (HTTPGenerator):
- push() sends the local configuration to the server, which keeps its data,
  drops the fitted model and retrains. Editing or replacing the local copy
  without pushing warns once per edit. push() warns when the copy was not
  pulled since the last step, since it carries optimizer state too.
- A lost session (server restart) is recreated from the configuration the
  server last had and the full local data, with a warning that says what it
  resumed from. At most one attempt per request; recover_session=False
  raises instead.
- A new session always receives the local data; the push happens outside
  the recovery path so a failure rolls the session back.
- An ingest whose outcome is unknown (timeout, 5xx) is not retried; the
  client replaces the server's data instead. 4xx rejections are raised.
- A server-side DataError arrives as DataError, so Xopt(strict=False)
  behaves as with a local generator.
- Attribute delegation only resolves names the wrapped class defines,
  never creates a session, and pulls once per step.
- pull(include_model=False) leaves out the fitted model but keeps custom
  kernels and prior means; Xopt.dump() uses it unless serialize_torch.
- Rejects at wrap time: StateOwner generators, custom_objective, custom
  multi-fidelity cost_function (all silently altered by the trip).
- Default timeout 30 s -> 300 s.

Server:
- GET /health (no token), GET /sessions, PUT /sessions/{id}/state,
  ?include_model= on GET state, XOPT_HTTP_MAX_SESSIONS cap (503).
- Construction warnings and a use_cuda-without-CUDA notice are returned
  and re-raised on the client.
- Auth moved to the routers; the cloudpickle server shares the session
  registry, cap and health route.

Core: Generator.sync_for_dump(include_model) and Xopt.dump() passing
serialize_torch through.

Guide: server restarts, deploying the server, limitations and next steps.

Generated with AI

Co-Authored-By: SLAC AI
@codecov

codecov Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.97864% with 11 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...opt/tests/generators/remote/test_http_generator.py 98.62% 6 Missing and 2 partials ⚠️
xopt/generators/remote/server.py 98.30% 0 Missing and 2 partials ⚠️
xopt/tests/generators/test_concurrent_bayesian.py 98.57% 0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

Use a dedicated pytest temporary working directory for each test.

Prevent stale model files from interfering with inline and external model serialization tests.

Remove the deprecated module-scoped cleanup fixture without changing production serialization behavior.
Initialize CUDA linear algebra automatically on first use with thread-safe, per-device initialization.

Prevent concurrent first-use PyTorch CUDA failures without serializing GP model training or acquisition optimization.

Preserve CPU-only execution without triggering CUDA initialization or requiring additional user configuration.

Add unit and integration tests covering six independent local and remote Xopt applications using UCB, BAX, and expected improvement across CPU, CUDA, and mixed workloads.

Validate remote session isolation, selective finalization, continued optimization, and session cleanup.

Improve HTTP server subprocess diagnostics and portable test logging.

Validation: 865 GPU-enabled and 841 CPU-only regression tests passed with zero failures. CUDA initialization utility achieved 100% statement and branch coverage.
@ernestow
ernestow marked this pull request as ready for review November 20, 2026 15:11

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants