From 1a46af5623c7d0d317bc609453520cdd31a83c49 Mon Sep 17 00:00:00 2001 From: Jakhongir Rakhmonov Date: Fri, 17 Jul 2026 15:30:33 +0000 Subject: [PATCH 1/2] firmware docs overhaul Signed-off-by: Jakhongir Rakhmonov --- docs/index.yml | 18 +- docs/operations/firmware-updates.md | 598 ++++-------------- .../firmware-updates/configuration.md | 383 +++++++++++ .../firmware-updates/dpu-firmware.md | 248 ++++++++ .../firmware-updates/host-firmware.md | 382 +++++++++++ .../firmware-updates/pre-ingestion.md | 184 ++++++ .../rack-component-firmware.md | 331 ++++++++++ 7 files changed, 1666 insertions(+), 478 deletions(-) create mode 100644 docs/operations/firmware-updates/configuration.md create mode 100644 docs/operations/firmware-updates/dpu-firmware.md create mode 100644 docs/operations/firmware-updates/host-firmware.md create mode 100644 docs/operations/firmware-updates/pre-ingestion.md create mode 100644 docs/operations/firmware-updates/rack-component-firmware.md diff --git a/docs/index.yml b/docs/index.yml index 6ea13bc029..4951461ffb 100644 --- a/docs/index.yml +++ b/docs/index.yml @@ -152,8 +152,22 @@ navigation: path: manuals/dpu_extension_service.md - page: BlueField DPU Operations path: dpu-management/dpu-operations.md - - page: Firmware Updates - path: operations/firmware-updates.md + - section: Firmware Updates + contents: + - page: Overview + path: operations/firmware-updates.md + - page: Configure Firmware Versions + path: operations/firmware-updates/configuration.md + - page: Pre-ingestion Firmware Updates + path: operations/firmware-updates/pre-ingestion.md + - page: Managed Host Firmware Updates + path: operations/firmware-updates/host-firmware.md + - page: DPU Firmware Updates + path: operations/firmware-updates/dpu-firmware.md + - page: Rack and Tray Firmware Updates + path: operations/firmware-updates/rack-component-firmware.md + - page: Flow Operation Rules + path: ../rest-api/flow/docs/operation-rules-guide.md - page: Monitoring and Health path: operations/monitoring-health.md - page: Tenant Lifecycle Cleanup diff --git a/docs/operations/firmware-updates.md b/docs/operations/firmware-updates.md index 017d21b5a5..7b13639e7b 100644 --- a/docs/operations/firmware-updates.md +++ b/docs/operations/firmware-updates.md @@ -1,479 +1,125 @@ # Firmware Updates -This page covers when NICo updates firmware, how automated updates are -scheduled, how to control update windows, and where NICo hands off to -platform-specific firmware procedures. - -Firmware updates run in two operating moments: - -- **Initial ingestion and pre-ingestion.** NICo can update firmware before a - host is fully ingested when the current firmware is too old for safe - discovery, host-to-DPU pairing, or the configured minimum firmware policy. -- **Approved service windows.** After NICo manages a machine, firmware updates - are scheduled during approved maintenance windows and are limited by health - and concurrency policy. - -For related background, see: - -- [Architecture Overview](../architecture/overview.md#machine-update-manager) -- [Managed Host State Diagrams](../architecture/state_machines/managedhost.md) -- [DPU Lifecycle Management](../dpu-management/dpu-lifecycle-management.md#firmware-upgrades) -- [Redfish Workflow](../architecture/redfish_workflow.md) -- [Core Metrics](../observability/core_metrics.md) - -## Workflow Boundaries - -NICo has several firmware paths. Use the path that matches the component being -updated. - -| Scope | NICo path | Use | -|---|---|---| -| Host BMC, BIOS, UEFI, and platform firmware listed in host firmware metadata | Machine Update Manager and the managed-host state machine | Fleet host firmware drift and host firmware service windows. | -| DPU NIC firmware drift | Machine Update Manager and DPU reprovisioning | Automatic DPU reprovisioning when DPU NIC firmware is outside the configured baseline. | -| DPU BMC, UEFI, CEC, ERoT, and related DPU firmware | DPU reprovisioning | Verification and update as part of the DPU lifecycle. | -| NVIDIA platform procedures | Manual service procedure plus NICo completion gate | NVIDIA-managed platforms, such as GB200, that require a field procedure before NICo resumes automatic checks. | -| OEM platform procedures | OEM instructions plus NICo scheduling or script handoff | Vendor-specific packages, such as SMC/Supermicro platforms, where the OEM procedure is the authority for platform-specific steps. | -| Rack, compute tray, switch, and power shelf firmware | REST firmware APIs for rack and tray updates. Compute tray updates flow through Machine Update Manager and the managed-host lifecycle; switch and power shelf updates use component-manager. | Rack-level and component-level firmware updates. | - -Use `nico-admin-cli` for CLI examples. These commands talk to the NICo Core -gRPC API: - -```bash -nico-admin-cli -a -``` - -## Machine Update Manager - -Machine Update Manager is a scheduler. It does not flash firmware directly. On -each run it: - -1. Clears completed host and DPU update markers. -2. Counts machines already in maintenance or firmware update states. -3. Counts healthy and unhealthy hosts. -4. Computes how many additional updates can start. -5. Asks each enabled update module to start work until the site limit is - reached. - -Automated updates are selected only when the target is eligible. The normal -eligibility gates are: - -- The machine is known to NICo and is in a state that can be updated. -- The relevant update module is enabled. -- The machine needs firmware according to the configured baseline. -- The machine is not already in maintenance or another update flow. -- Site health and concurrency policy allow another host to leave service. -- A required update window is active when the firmware entry requires explicit - start. - -The actual firmware work runs inside the managed-host lifecycle. This keeps -host power, BMC operations, DPU reprovisioning, health reports, and state -transitions under one lifecycle controller. - -## Configuration - -Firmware behavior is controlled by site configuration and firmware metadata. - -### Machine Update Scheduling - -| Field | Default | Meaning | -|---|---:|---| -| `machine_update_run_interval` | `300s` when unset | How often Machine Update Manager runs. | -| `machine_updater.instance_autoreboot_period.start` | unset | Start of the UTC window when machines may automatically reboot for updates. | -| `machine_updater.instance_autoreboot_period.end` | unset | End of the UTC window when machines may automatically reboot for updates. | -| `machine_updater.max_concurrent_machine_updates_absolute` | unset | Hard cap on concurrent machine updates. | -| `machine_updater.max_concurrent_machine_updates_percent` | unset | Percentage cap on concurrent machine updates. When both caps are set, NICo uses the lower effective limit. | - -Example: - -```toml -machine_update_run_interval = 60 - -[machine_updater] -instance_autoreboot_period.start = "2026-05-22T01:00:00Z" -instance_autoreboot_period.end = "2026-05-22T05:00:00Z" -max_concurrent_machine_updates_absolute = 5 -max_concurrent_machine_updates_percent = 10 -``` - -### Host Firmware Settings - -| Field | Default | Meaning | -|---|---:|---| -| `firmware_global.autoupdate` | `false` | Enables automatic host firmware updates. | -| `firmware_global.host_enable_autoupdate` | `[]` | Host models or machine IDs to force-enable for host firmware autoupdate. | -| `firmware_global.host_disable_autoupdate` | `[]` | Host models or machine IDs to force-disable for host firmware autoupdate. | -| `firmware_global.run_interval` | `30s` | Firmware manager polling interval. | -| `firmware_global.max_uploads` | `4` | Maximum concurrent firmware uploads. | -| `firmware_global.concurrency_limit` | `16` | Maximum concurrent firmware flashing operations. | -| `firmware_global.firmware_directory` | `/opt/nico/firmware` | Directory for firmware binaries and `metadata.toml` files. | -| `firmware_global.host_firmware_upgrade_retry_interval` | `60m` | Delay before retrying a failed host firmware upgrade. | -| `firmware_global.instance_updates_manual_tagging` | `true` | Requires manual tagging before host firmware updates are applied. | -| `firmware_global.no_reset_retries` | `false` | Disables retry logic after BMC resets during firmware operations. | -| `firmware_global.hgx_bmc_gpu_reboot_delay` | `30s` | Delay after GPU reboot before the HGX BMC can be accessed. | -| `firmware_global.requires_manual_upgrade` | `false` | Forces firmware upgrades through a manual completion gate. | -| `firmware_global.max_concurrent_bfb_copies` | `10` | Maximum concurrent BFB copies. | - -Example: - -```toml -[firmware_global] -autoupdate = true -host_enable_autoupdate = ["PowerEdge R750"] -host_disable_autoupdate = [] -run_interval = "30s" -max_uploads = 4 -concurrency_limit = 16 -firmware_directory = "/opt/nico/firmware" -host_firmware_upgrade_retry_interval = "60m" -requires_manual_upgrade = false -``` - -### DPU Firmware Settings - -| Field | Default | Meaning | -|---|---:|---| -| `dpu_config.dpu_nic_firmware_initial_update_enabled` | `false` | Enables DPU NIC firmware updates on initial discovery. | -| `dpu_config.dpu_nic_firmware_reprovision_update_enabled` | `true` | Enables DPU NIC firmware updates during reprovisioning. | -| `dpu_config.dpu_models` | BF2/BF3 defaults | DPU firmware definitions. | -| `dpu_config.dpu_nic_firmware_update_versions` | BF2/BF3 NIC versions | Accepted DPU NIC firmware versions. Firmware drift outside this set can trigger automated DPU reprovisioning. | -| `dpu_config.dpu_enable_secure_boot` | `false` | Enables the secure boot flow for DPU provisioning through Redfish. | - -## Firmware Baselines - -NICo compares observed firmware inventory with configured firmware baselines. -Host baselines are loaded from embedded configuration and from -`metadata.toml` files under `firmware_global.firmware_directory`. Metadata can -define vendor and model matching, component ordering, known firmware versions, -default versions, minimum pre-ingestion versions, and whether explicit start is -required. - -List host firmware entries known to NICo: - -```bash -nico-admin-cli -a firmware show -``` - -The output includes vendor, model, component type, inventory-name match, -version, and whether the update needs explicit start. - -Keep live "latest firmware version" tables in a single baseline source and link -to that source from site runbooks. Do not copy live version tables into this -operations page; version tables change independently from the update workflow. -If a site mirrors firmware versions into a wiki or dashboard, make the mirror -point back to the same source-controlled baseline or approved version catalog. - -To update a host firmware baseline: - -1. Add or update the firmware metadata for the target vendor and model. -2. Place the firmware binary where `firmware_global.firmware_directory` points, - or provide the approved URL/script metadata used by the site. -3. Mark the intended version as the default for the component. -4. Verify the new baseline with `firmware show`. -5. Apply the update through a service window or machine-specific update window. - -Example host firmware metadata: - -```toml -[host_models.dell_r750] -vendor = "Dell" -model = "PowerEdge R750" -ordering = ["uefi", "bmc"] - -[host_models.dell_r750.components.bmc] -current_version_reported_as = "^Installed-.*__iDRAC." -preingest_upgrade_when_below = "0.5" -known_firmware = [ - { version = "1.1", filename = "/opt/nico/firmware/dell/r750_bmc_1.1.fw", checksum = "", default = true }, -] - -[host_models.dell_r750.components.uefi] -current_version_reported_as = "^Installed-.*__BIOS.Setup." -preingest_upgrade_when_below = "0.5" -known_firmware = [ - { version = "2.0", filename = "/opt/nico/firmware/dell/r750_uefi_2.0.fw", checksum = "", default = true }, -] -``` - -## Host Firmware Updates - -Host firmware updates use Redfish inventory and firmware metadata. Common host -components include BMC, BIOS, UEFI, HGX BMC, GPU, NIC, and platform-specific -firmware, depending on the platform metadata. - -During a host firmware update, NICo can: - -- Upload firmware through Redfish or run an approved firmware script. -- Poll Redfish tasks and firmware inventory. -- Reset the BMC or host when required by the platform. -- Re-check firmware versions after activation. -- Apply a `HostUpdateInProgress` health report that prevents allocation while - update work is active. - -Enable, disable, or clear machine-specific auto-update behavior: - -```bash -nico-admin-cli -a machine auto-update --machine --enable -nico-admin-cli -a machine auto-update --machine --disable -nico-admin-cli -a machine auto-update --machine --clear -``` - -Set an explicit firmware update window for one or more machines: - -```bash -nico-admin-cli -a managed-host start-updates \ - --machines \ - --start 2026-05-22T01:00:00-0700 \ - --end 2026-05-22T05:00:00-0700 -``` - -Cancel pending start windows: - -```bash -nico-admin-cli -a managed-host start-updates \ - --machines \ - --cancel -``` - -Request host reprovisioning when the host must be put back through the -managed-host firmware path: - -```bash -nico-admin-cli -a host reprovision set \ - --id \ - --update-firmware \ - --update-message "" -``` - -## DPU Firmware Updates - -DPU firmware is managed as part of the managed-host lifecycle. NICo tracks: - -| Component | Inventory name | Meaning | -|---|---|---| -| DPU NIC firmware | `DPU_NIC` | Primary NIC firmware on the BlueField. | -| DPU BMC firmware | `BMC_Firmware` | DPU management controller firmware. | -| DPU UEFI firmware | `DPU_UEFI` | DPU boot firmware. | -| ATF / ERoT firmware | `Bluefield_FW_ERoT` | Arm Trusted Firmware or External Root of Trust. | - -Machine Update Manager uses DPU NIC firmware drift as the automatic trigger. -During DPU reprovisioning, NICo verifies and updates the DPU firmware set -against the configured baseline. - -Inspect DPU firmware status: - -```bash -nico-admin-cli -a dpu versions -``` - -For the full DPU firmware flow, see -[DPU Lifecycle Management](../dpu-management/dpu-lifecycle-management.md#firmware-upgrades). - -## Manual Platform Updates - -Some platforms require a manual field procedure before NICo can complete the -firmware workflow. - -### NVIDIA Platforms - -For NVIDIA-managed platforms, follow the approved NVIDIA service procedure for -the exact platform and firmware package. GB200 is the common example for this -flow. NICo normally runs the automated firmware lifecycle. Use the manual -procedure only when the site runbook or platform support path requires a manual -GB200 update gate. - -A GB200 firmware service procedure can cover several component groups: - -| Component group | Purpose | -|---|---| -| NVSwitch trays | Updates switch firmware before the rack is returned to service. | -| GB200 compute tray BMC | Updates the compute tray management controller used for out-of-band control. | -| HGX firmware | Updates the GPU/HGX firmware package for the compute tray. | -| System BIOS or UEFI | Updates host boot firmware and platform initialization behavior. | -| Auxiliary power or BMC power state | Applies platform power-state changes required for firmware activation. | -| CPU node BMC and BIOS | Updates CPU node management and boot firmware through the applicable OEM procedure. | -| TPM state | Clears or resets TPM state when required by the platform procedure. | -| NVMe SSD firmware and secure erase | Updates storage firmware and returns storage to the expected clean state. | -| CX7 firmware | Updates ConnectX firmware used by the host networking stack. | -| BlueField-3 firmware | Updates DPU NIC and boot-related firmware where the GB200 rack uses BF3 DPUs. | -| Power cycle and validation | Activates pending firmware and verifies platform health, including NVLink status. | - -The exact order, package names, credentials, BMC addresses, and validation -commands are site- and release-specific. Keep those details in the approved -GB200 service procedure. This Operations guide documents where NICo waits and -how to resume the lifecycle after that procedure is complete. - -When manual firmware upgrade is required, NICo moves the managed host to a -manual waiting state. Complete the approved GB200 firmware procedure first. -After the field procedure is complete, mark the manual firmware upgrade -complete so NICo can resume automatic checks: - -```bash -nico-admin-cli -a host reprovision mark-manual-upgrade-complete --id -``` - -### OEM Platforms - -For OEM platforms, consult the OEM support site for the exact platform and -firmware package before starting work. The OEM support site is the final -authority for vendor-specific prerequisites, package selection, activation -steps, and recovery. - -SMC/Supermicro is one example of this path. NICo can use the Redfish firmware -workflow when firmware metadata identifies the target component and package. -The Supermicro Redfish implementation supports: - -| Component or action | NICo behavior | -|---|---| -| Lockdown state | Reads and updates the Supermicro `SysLockdown` setting before firmware work when required. | -| BMC firmware | Uses Redfish multipart upload with the BMC target and Supermicro OEM preserve/backup settings. | -| BIOS or UEFI firmware | Uses Redfish multipart upload with the BIOS target and Supermicro OEM preserve settings. | -| CPLD firmware | Supports Supermicro motherboard and backplane CPLD Redfish inventory targets. | - -NICo can schedule, track, or hand off the update when the site has integrated -an approved firmware package or script. Use the OEM procedure to confirm the -exact package, update order, activation requirements, and recovery steps. - -## Rack and Component Firmware - -Use the NVIDIA Forge REST API for rack and tray firmware updates. The API -accepts a site ID and optional target firmware version, starts the update -workflow, and returns task IDs for tracking. - -REST API entry points: - -| Scope | Method and path | Use | -|---|---|---| -| Single rack | `PATCH /v2/org/{org}/nico/rack/{id}/firmware` | Update firmware for one rack. | -| Rack batch | `PATCH /v2/org/{org}/nico/rack/firmware` | Update firmware for racks selected by filter, or all racks in the site when no filter is supplied. | -| Single tray | `PATCH /v2/org/{org}/nico/tray/{id}/firmware` | Update firmware for one compute tray. | -| Tray batch | `PATCH /v2/org/{org}/nico/tray/firmware` | Update firmware for trays selected by filter, or all trays in the site when no filter is supplied. | - -Request fields: - -| Field | Required | Use | +This guide first gives a high-level view of firmware upgrades in NICo and helps +you choose the workflow that applies to your hardware. The linked workflow +pages then explain their configuration, prerequisites, execution, verification, +and recovery in detail. + +## Overview + +NICo uses a site-wide desired-state process for managed host firmware and DPU +NIC firmware. Operators configure the versions expected at the site. When +automatic updates are enabled, NICo compares those versions with the firmware +reported by managed hardware. Drift makes a device eligible for an upgrade, +which NICo schedules automatically instead of requiring a one-off upgrade +command. + +The drift-driven workflow is: + +```mermaid +flowchart LR + A["Configured host or DPU NIC versions"] --> B["Compare with discovered versions"] + B --> C{"Drift detected?"} + C -- No --> D["No action"] + C -- Yes --> E["Wait until the device is eligible"] + E --> F["Upgrade or reprovision"] + F --> G["Rediscover and verify"] +``` + +Detecting drift does not mean the upgrade starts immediately. NICo may wait for +the machine to enter an eligible state, an update window to open, or site-wide +concurrency limits to allow more work. The responsible controller then performs +the update or reprovisioning and verifies the result from newly discovered +hardware state. + +[Pre-ingestion](firmware-updates/pre-ingestion.md) is a related but separate +path. When automatic updates are enabled, it can update new hardware whose +firmware is below a configured minimum before NICo ingests and manages it. +After ingestion, drift detection keeps the hardware aligned with the configured +baseline. + +Not every firmware workflow begins with drift detection. Rack-scale compute +trays, NVSwitches, and power shelves can be updated through explicit API +requests. Some platforms require a vendor tool, a script, rack maintenance, or +a manually confirmed procedure. See +[Choose the upgrade path](#choose-the-upgrade-path) for the path used by each +component. + +## How automatic work is scheduled + +Machine Update Manager schedules automatic firmware work for managed hosts and +DPUs. It does not install firmware itself; the managed-host state machine owns +the disruptive work and the return to service. + +On each scheduler pass, NICo: + +1. clears bookkeeping for updates that have completed; +2. counts the distinct hosts already in maintenance or an update workflow; +3. calculates the remaining capacity from the site-wide machine-update limit; +4. asks the enabled DPU and host firmware modules for eligible work; and +5. creates requests until the shared capacity is full. + +```mermaid +flowchart LR + A["Periodic scheduler pass"] --> B["Clear completed update markers"] + B --> C["Count hosts already updating"] + C --> D["Calculate shared capacity"] + D --> E["Apply DPU and host eligibility checks"] + E --> F["Create reprovisioning requests"] + F --> G["Managed-host state machine performs the work"] +``` + +Host and DPU work therefore compete for the same capacity, and a host with +several drifted DPUs counts once. Each module still applies its own state, +health, approval, and update-window checks before selecting a host. If the +calculated capacity is zero, drift remains pending and no automatic managed +machine update starts. + +Pre-ingestion firmware is not scheduled by Machine Update Manager. It has a +separate polling interval and concurrency limit because the hardware has not +yet entered the managed-host lifecycle. See the +[Machine Update Manager architecture](../architecture/overview.md#machine-update-manager) +for the broader ownership model. + +## Set the expected versions + +NICo does not discover the latest firmware version on its own. Before NICo can +detect drift, the site must define which firmware versions are expected. + +For host firmware, NICo uses a catalog organized by BMC vendor, host model, and +component. A component can define both: + +- `preingest_upgrade_when_below`: the threshold that triggers firmware work + before ingestion. +- `default`: the desired version used to detect drift after ingestion. + +For example, a host reporting BMC version `7.05.00.00` meets a configured +minimum of `7.00.00.00`, so ingestion can continue. If the default is +`7.10.30.00`, that host is still drifted after ingestion and becomes eligible +for the managed-host firmware workflow. + +Host definitions can come from the +[Host Firmware Config API](api:PUT/v2/org/{org}/nico/firmware-config/host), +legacy `metadata.toml` files, or static `host_models` configuration. NICo merges +these sources into one effective catalog. The API is the current configuration +method; `metadata.toml` is retained for existing deployments. + +DPU NIC firmware uses an accepted-version list in the site configuration +instead of the host catalog. Rack and component workflows receive their target +from an explicit request or their component backend. + +See [Configure firmware versions](firmware-updates/configuration.md) for the +configuration models, setting reference, examples, and source precedence. + +## Choose the upgrade path + +Use the table below to find the path for the hardware and component you are +updating. For hosts, the path also depends on whether ingestion has completed. + +| Your situation | Upgrade path | How it begins | |---|---|---| -| `siteId` | Yes | Site that owns the rack or tray targets. | -| `version` | No | Target firmware version. Omit only when the site workflow selects the target version. | -| `filter` | Batch only | Selects the batch targets. Rack batch filters by rack name. Tray batch filters by rack ID, rack name, tray ID, component ID, or tray type. | - -Batch tray filters follow these constraints: - -- Use either `rackId` or `rackName`, not both. -- Do not combine a rack filter with `ids` or `componentIds`. -- Use `type` when filtering by `componentIds`. - -The API requires provider-admin authorization and returns task IDs for tracking. - -Single-rack example: - -```bash -curl -X PATCH "https:///v2/org//nico/rack//firmware" \ - -H "Authorization: Bearer " \ - -H "Content-Type: application/json" \ - -d '{ - "siteId": "", - "version": "" - }' -``` - -Batch tray example for a rack: - -```bash -curl -X PATCH "https:///v2/org//nico/tray/firmware" \ - -H "Authorization: Bearer " \ - -H "Content-Type: application/json" \ - -d '{ - "siteId": "", - "version": "", - "filter": { - "rackName": "" - } - }' -``` - -The response contains task IDs: - -```json -{ - "taskIds": [ - "" - ] -} -``` - -Use component-manager only for lower-level switch and power shelf firmware -workflows when a REST workflow is not available for that component: - -```bash -nico-admin-cli -a component-manager update-firmware switch \ - --switch-id \ - --target-version \ - --component bmc,cpld,bios,nvos - -nico-admin-cli -a component-manager update-firmware power-shelf \ - --power-shelf-id \ - --target-version \ - --component pmc,psu -``` - -Check component firmware status: - -```bash -nico-admin-cli -a component-manager get-firmware-update-status switch --switch-id -nico-admin-cli -a component-manager get-firmware-update-status power-shelf --power-shelf-id -``` - -## Monitor Progress - -Start with the object being updated, then move to logs and metrics when progress -is unclear. - -| Task | Command | -|---|---| -| Show managed-host state and handler outcome | `nico-admin-cli -a managed-host show ` | -| Show machine details and health reports | `nico-admin-cli -a machine show ` | -| Show host firmware baseline entries | `nico-admin-cli -a firmware show` | -| Show DPU firmware status | `nico-admin-cli -a dpu versions` | -| Start rack or tray firmware update workflow | Use the rack and tray REST firmware endpoints. The response returns `taskIds`. | -| Show switch or power shelf component firmware status | `nico-admin-cli -a component-manager get-firmware-update-status ...` | - -Useful metrics: - -| Metric | Meaning | -|---|---| -| `carbide_pending_host_firmware_update_count` | Host machines that need a host firmware update. | -| `carbide_active_host_firmware_update_count` | Host machines actively updating firmware. | -| `carbide_pending_dpu_nic_firmware_update_count` | Machines with DPU NIC firmware drift. | -| `carbide_unavailable_dpu_nic_firmware_update_count` | Machines with DPU NIC firmware drift that are not currently available for update. | -| `carbide_running_dpu_updates_count` | Machines running DPU firmware update work. | -| `carbide_dpu_firmware_version_count` | DPUs reporting a firmware version. | -| `carbide_preingestion_waiting_download` | Pre-ingestion hosts waiting for firmware downloads on other machines to complete. | -| `carbide_preingestion_waiting_installation` | Pre-ingestion hosts with firmware uploaded and installation in progress. | - -## Recovery - -If a firmware update does not progress: - -1. Inspect the managed-host state, REST task IDs, or component status for the - failing object. -2. Check the health reports and handler outcome for the reason NICo is waiting. -3. Check `nico-api` logs for Redfish task failures, BMC reachability errors, - script failures, REST firmware task failures, RMS backend errors, or - component-manager job errors. -4. Follow the platform or OEM recovery procedure before retrying a failed - firmware operation. -5. Retry through the same NICo workflow after the underlying platform condition - is corrected. - -A host firmware failure can place the host in a failed firmware-upgrade state. -NICo retries according to `firmware_global.host_firmware_upgrade_retry_interval` -where retry is supported. A rack or component firmware failure should be -tracked through the REST task IDs returned by the rack or tray firmware API, -or through component-manager status for switch and power shelf targets. - -After the platform condition is corrected, reset a failed host reprovisioning -flow only when the site runbook calls for it: - -```bash -nico-admin-cli -a managed-host reset-host-reprovisioning --machine -``` +| A host has not completed ingestion and a reported component version is below `preingest_upgrade_when_below`. | [Pre-ingestion](firmware-updates/pre-ingestion.md) | The pre-ingestion manager detects the condition. When global automatic updates are enabled, it selects the default or pre-ingestion-only firmware entry and starts the update. | +| Firmware on a managed host differs from the effective default in the merged firmware catalog. | [Host firmware](firmware-updates/host-firmware.md) | When host automatic updates are enabled, NICo detects the drift and schedules host reprovisioning. If `explicit_start_needed` is enabled, NICo first waits for the host's update window. | +| A managed DPU NIC reports a version outside `dpu_nic_firmware_update_versions`. | [DPU NIC firmware](firmware-updates/dpu-firmware.md#dpu-nic-firmware) | When DPU NIC updates are enabled, NICo detects the drift and schedules DPU reprovisioning. | +| A DPU component other than the NIC needs an update. | [DPU firmware components](firmware-updates/dpu-firmware.md#components-covered) | Request DPU reprovisioning explicitly. Applying the BFB updates the bundled firmware and verifies selected components; non-NIC drift is not an independent automatic trigger. | +| A rack, compute tray, NVSwitch, or power shelf needs an update. | [Rack and component firmware](firmware-updates/rack-component-firmware.md) | An operator requests the update through the appropriate API. NICo routes it to the controller or backend responsible for that component. | diff --git a/docs/operations/firmware-updates/configuration.md b/docs/operations/firmware-updates/configuration.md new file mode 100644 index 0000000000..c51d985b7b --- /dev/null +++ b/docs/operations/firmware-updates/configuration.md @@ -0,0 +1,383 @@ +# Configure Firmware Versions + +NICo needs a site-specific definition of acceptable firmware before it can +detect drift or choose an upgrade target. Host firmware and DPU NIC firmware +use different configuration models. Rack and component updates use a target +supplied by the request or selected by their backend. + +The firmware catalog answers **what firmware NICo recognizes and should +install**. Start there: define host firmware through the +[Host Firmware Config API](api:PUT/v2/org/{org}/nico/firmware-config/host) +or legacy metadata. Later sections cover DPU baselines and the site controls +that determine **when NICo looks for work** and **how much work it may start**. + +## Host firmware catalog + +Each host firmware definition is associated with a BMC vendor and host model. +The definition can come from the Host Firmware Config API, a legacy +`metadata.toml` file, or the static `host_models` configuration. NICo merges +these sources into one effective catalog. + +During site exploration, Redfish reports the BMC vendor and host model. NICo +uses that pair to select the corresponding definition from the catalog. A +definition can contain several components, such as BMC, UEFI, CPLD, HGX BMC, +GPU, NIC, or CX7 firmware. + +A definition contains model-level settings and one or more component entries: + +| Setting | Scope | Meaning | +|---|---|---| +| `vendor` | Model | BMC vendor to which the definition applies. | +| `model` | Model | Host model to which the definition applies. Together, `vendor` and `model` identify the definition. | +| `components` | Model | Groups the component definitions, keyed by component type. | +| `ordering` | Model | Lists the components in the order NICo should update them. | +| `explicit_start_needed` | Model | When enabled, drift alone does not start the update. | +| `type` | Component | Identifies the component as BMC, UEFI, CEC, NIC, CPLD, HGX BMC, GPU, CX7, or a combined BMC and UEFI package. In legacy TOML, the type is the key below `components`. | +| `current_version_reported_as` | Component | Regular expression used to find the component in the firmware inventory reported by the BMC. The API derives this matcher from the vendor, model, and component type. | +| `preingest_upgrade_when_below` | Component | Sets the threshold for pre-ingestion firmware work. A lower reported version triggers an update when global automatic updates are enabled. | +| `known_firmware` | Component | Lists the versions NICo can install and the artifacts used to install them. | +| `version` | Firmware version | Version string NICo expects the firmware inventory to report after installation. | +| `default` | Firmware version | Marks the desired version used for drift detection after ingestion. | +| `preingestion_exclusive_config` | Firmware version | Marks a version for use during pre-ingestion without making it the steady-state default. | +| `files` | Firmware version | Structured list of firmware artifacts. Each entry provides a filename or URL and can include a SHA-256 digest. The API exposes this list as `artifacts`. | +| `install_only_specified` | Firmware version | Tells Redfish to install only the component identified by the firmware definition when the package contains multiple components. | +| `power_drains_needed` | Firmware version | Number of full power-drain cycles required to activate the firmware after installation. | +| `pre_update_resets` | Firmware version | Requests the platform-specific reset sequence before NICo starts the installation. | +| `script` | Firmware version | Legacy path that hands the update to a local script instead of using the normal Redfish upload path. | + +`current_version_reported_as` is matched against firmware inventory entry +identifiers, not against version strings. For example, +`^Installed-.*__iDRAC.` selects the iDRAC entry from a Dell Redfish firmware +inventory. NICo then reads the installed version from that matching entry. + +Legacy metadata also accepts the older artifact fields `filename`, `filenames`, +`url`, and `checksum`; new API entries use URL-based `artifacts`. The data model +still parses `mandatory_upgrade_from_priority` and `scout`, but the current +upgrade workflow does not use them as firmware-policy controls, so they are not +listed as active settings here. + +### Minimum and default versions + +The pre-ingestion minimum is an upgrade trigger, not the upgrade target. When a +host is below the minimum and global automatic updates are enabled, NICo +normally installs the version marked as the default. A firmware version can +instead set `preingestion_exclusive_config = true` when the platform needs a +different target during ingestion. + +For example, assume the BMC minimum is `7.00.00.00` and the default is +`7.10.30.00`: + +| Reported version | Result | +|---|---| +| `6.50.00.00` | Below the minimum. With automatic updates enabled, NICo updates the BMC during pre-ingestion, using `7.10.30.00` as the target. | +| `7.05.00.00` | Meets the minimum, so ingestion can continue. After ingestion it is still drifted from `7.10.30.00`. | +| `7.10.30.00` | Meets the minimum and matches the desired version. No firmware update is needed. | + +See [Pre-ingestion firmware](pre-ingestion.md) for minimum-version enforcement +and [managed host firmware](host-firmware.md) for post-ingestion drift handling. + +### Configure host firmware through the API + +The current way to configure host firmware is the +[Create or Update Host Firmware Config](api:PUT/v2/org/{org}/nico/firmware-config/host) +operation: + +```text +PUT /v2/org/{org}/nico/firmware-config/host +``` + +The request is site-scoped and keyed by `(vendor, model)`. NICo stores the +configuration in `host_firmware_config` and uses it on subsequent discovery, +pre-ingestion, drift detection, and upgrade passes. This allows a site to +change its firmware catalog without changing the NICo deployment or placing +metadata files on the Core filesystem. + +The endpoint requires Provider Admin authorization. The following request +defines one BMC version for a Dell PowerEdge R760: + +```bash +curl -X PUT "https:///v2/org//nico/firmware-config/host" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "siteId": "", + "vendor": "Dell", + "model": "PowerEdge R760", + "ordering": ["BMC"], + "components": [ + { + "type": "BMC", + "preingestUpgradeWhenBelow": "7.00.00.00", + "firmware": [ + { + "version": "7.10.30.00", + "default": true, + "artifacts": [ + { "url": "https:///" } + ] + } + ] + } + ] + }' +``` + +The request does not contain an inventory-matching regular expression. Core +derives that expression from its supported vendor, model, and component +mappings. The request is rejected when no mapping exists. + +`PUT` is an upsert, not a complete replacement: + +- The first request for a vendor and model must provide an ordering that + contains every configured component. +- Each component must have exactly one default firmware version after the + request is merged. +- Later requests can add components or firmware versions. Versions are merged + by version string, and omitted components remain unchanged. +- Marking a new version as the default clears the previous default for that + component. + +The +[Delete Host Firmware Config](api:DELETE/v2/org/{org}/nico/firmware-config/host) +operation removes the runtime entry. If the same vendor and model also exists +in static or legacy metadata, that lower-priority definition becomes effective +again. + +### Legacy `metadata.toml` configuration + +Before the Host Firmware Config API was introduced, host firmware definitions +were deployed as `metadata.toml` files. This is the legacy configuration path. +It remains supported for existing deployments, but new host firmware +definitions should use the API. + +`firmware_global.firmware_directory` points to a directory containing one +subdirectory per metadata entry. NICo looks for a file named `metadata.toml` in +each of those subdirectories. For example: + +```text +/opt/nico/firmware/ +`-- dell-r760-bmc-7.10.30/ + `-- metadata.toml +``` + +Each file describes one vendor and model and can define one or more firmware +components. The equivalent legacy definition for the API example above is: + +```toml +vendor = "Dell" +model = "PowerEdge R760" +ordering = ["bmc"] + +[components.bmc] +current_version_reported_as = "^Installed-.*__iDRAC." +preingest_upgrade_when_below = "7.00.00.00" + +[[components.bmc.known_firmware]] +version = "7.10.30.00" +url = "https:///" +default = true +``` + +NICo also supports host definitions embedded in the static `host_models` +configuration. This is the base catalog supplied with the NICo deployment and +uses the same firmware data model as `metadata.toml`. + +NICo builds the effective host firmware catalog in this order: + +| Applied | Source | +|---|---| +| First | Static `host_models` configuration. | +| Next | Legacy `metadata.toml` files under `firmware_global.firmware_directory`. | +| Last (highest precedence) | Runtime `host_firmware_config` entries created through the API. | + +Higher-priority sources overlay matching vendor, model, and component entries; +they do not discard unrelated lower-priority entries. For example, if a +`metadata.toml` file defines BMC `7.00.00.00` as the default and an API request +adds BMC `7.10.30.00` as the default, both remain known versions, but +`7.10.30.00` becomes the effective target. + +Use the CLI to inspect the effective default versions after all three sources +have been merged: + +```bash +nico-admin-cli -a firmware show +``` + +## DPU firmware + +DPU firmware does not use the Host Firmware Config API. Its site configuration +has two related parts: an accepted-version list that triggers automatic work, +and a model catalog used to verify the result. + +| Setting | Default | Meaning | +|---|---|---| +| `dpu_config.dpu_nic_firmware_initial_update_enabled` | `false` | Compatibility setting for initial DPU setup. The current state machine records it but does not use it to suppress BFB installation or verification. | +| `dpu_config.dpu_nic_firmware_reprovision_update_enabled` | `true` | Enables automatic DPU NIC drift selection in Machine Update Manager. Explicit DPU reprovisioning remains available when disabled. | +| `dpu_config.dpu_nic_firmware_update_versions` | Built-in BF2 and BF3 NIC versions | Accepted NIC versions used by the classic automatic drift detector. | +| `dpu_config.dpu_models` | Built-in BF2 and BF3 definitions | BMC, ERoT/CEC, NIC, and UEFI baselines associated with each DPU model. | +| `dpu_config.dpu_enable_secure_boot` | `false` | Allows classic reprovisioning to install the BFB through Redfish and enable secure boot when every attached DPU supports that path. Otherwise NICo uses UEFI HTTP network installation. | + +### Automatic DPU NIC trigger + +NICo checks the reported DPU NIC version against +`dpu_config.dpu_nic_firmware_update_versions`: + +```toml +[dpu_config] +dpu_nic_firmware_update_versions = ["32.43.1014", "32.44.1030"] +``` + +This is an accepted-version list rather than a single default. A DPU reporting +either version in the example is considered compliant. A version outside the +list is drifted and, when DPU NIC updates are enabled, makes the DPU eligible +for reprovisioning. Allowing more than one version is useful during a staged +rollout where both the old and new versions must temporarily remain valid. + +`dpu_nic_firmware_reprovision_update_enabled` enables this automatic drift +module. It does not prevent an operator from requesting DPU reprovisioning. + +### Post-reprovision baseline + +The `dpu_models` catalog describes the component versions expected after a +classic DPU reprovision. Entries are keyed by NICo's normalized DPU model, +currently `bluefield2` or `bluefield3`, and use the same firmware structures as +the static host catalog: + +```toml +[dpu_config.dpu_models.bluefield3] +vendor = "Nvidia" +model = "Bluefield 3 SmartNIC Main Card" +ordering = ["bmc", "cec"] + +[dpu_config.dpu_models.bluefield3.components.nic] +current_version_reported_as = "DPU_NIC" + +[[dpu_config.dpu_models.bluefield3.components.nic.known_firmware]] +version = "32.44.1030" +default = true +``` + +The built-in catalog defines BMC, CEC/ERoT, NIC, and UEFI entries for each +model. The current reprovisioning verifier compares BMC, CEC/ERoT, and NIC. It +selects the last non-pre-ingestion-only known version for each component; in +normal configurations, keep one steady-state entry for each component. + +NICo applies these versions by installing or booting the deployed BlueField +Bundle, not by uploading the `known_firmware` entries as independent packages. +The accepted NIC list, DPU model catalog, and firmware carried by `forge.bfb` +must therefore be updated together. For example, if the BFB installs NIC +version `32.44.1030`, that version must be accepted by +`dpu_nic_firmware_update_versions` and should be the NIC baseline in the +matching `dpu_models` entry. + +See [DPU firmware upgrades](dpu-firmware.md) for how NICo acts on this +configuration. + +## Operational controls + +The settings in this section are fields in NICo Core's site configuration. They +control the automatic paths; they do not define component target versions. + +### Managed-machine scheduling + +Machine Update Manager uses one capacity budget for managed-host and DPU +firmware work: + +| Setting | Default | Meaning | +|---|---|---| +| `machine_update_run_interval` | `300` | Seconds between Machine Update Manager passes. | +| `machine_updater.max_concurrent_machine_updates_absolute` | unset | Maximum number of distinct hosts that may be in machine-update workflows. Set this to a positive value to allow automatic managed-host or DPU work. | +| `machine_updater.max_concurrent_machine_updates_percent` | unset | Intended percentage-based capacity. The current implementation calculates this incorrectly; leave it unset and use the absolute limit. | +| `machine_updater.instance_autoreboot_period.start` | unset | Start of a one-time UTC window in which an assigned host may consume an existing update request without tenant approval. | +| `machine_updater.instance_autoreboot_period.end` | unset | End of that one-time UTC window. | + +For example, this allows at most five managed hosts to be in update workflows +and gives assigned hosts a four-hour automatic reboot window: + +```toml +machine_update_run_interval = 300 + +[machine_updater] +max_concurrent_machine_updates_absolute = 5 +instance_autoreboot_period.start = "2026-08-15T01:00:00Z" +instance_autoreboot_period.end = "2026-08-15T05:00:00Z" +``` + +The autoreboot period is not the per-host firmware window used by models with +`explicit_start_needed`. Operators open that window through the managed-host +API. See [Scheduling gates](host-firmware.md#scheduling-gates). + +### Firmware execution + +`firmware_global` controls host firmware execution and the separate +pre-ingestion manager: + +| Setting | Default | Meaning | +|---|---|---| +| `firmware_global.autoupdate` | `false` | Enables pre-ingestion upgrades and supplies the default automatic-update policy for managed hosts. A per-machine policy can override the managed-host default. | +| `firmware_global.run_interval` | `30s` | Interval between pre-ingestion manager passes. | +| `firmware_global.max_uploads` | `4` | Shared limit for concurrent host firmware uploads. | +| `firmware_global.concurrency_limit` | `16` | Maximum number of pre-ingestion endpoints processed concurrently. This is separate from Machine Update Manager capacity. | +| `firmware_global.firmware_directory` | `/opt/nico/firmware`, otherwise `/opt/carbide/firmware` | Directory containing legacy metadata and local firmware artifacts. | +| `firmware_global.firmware_download_cache_directory` | `/mnt/persistence/fw/download-cache` | Writable cache for downloaded firmware artifacts. | +| `firmware_global.host_firmware_upgrade_retry_interval` | `60m` | Delay before the managed-host state machine retries a failed host firmware upgrade. | +| `firmware_global.instance_updates_manual_tagging` | See below | When `true`, automatic host selection is limited to unassigned hosts in top-level `Ready`. When `false`, assigned hosts can also receive a pending request. | +| `firmware_global.no_reset_retries` | `false` | Disables the normal retry handling for BMC reset failures during pre-ingestion and managed-host firmware work. | +| `firmware_global.hgx_bmc_gpu_reboot_delay` | `30s` | Delay after a GPU reboot before NICo accesses the HGX BMC again. | +| `firmware_global.requires_manual_upgrade` | `false` | Adds a manually confirmed upgrade gate for MNNVL-capable hosts. | +| `firmware_global.max_concurrent_bfb_copies` | `10` | Maximum concurrent BFB copies performed by the pre-ingestion manager. | + +Set `instance_updates_manual_tagging` explicitly. When a present +`firmware_global` table omits the field, deserialization uses `true`; when the +entire table is omitted, the current whole-structure default uses `false`. + +A staging configuration can make the policy explicit while leaving site-wide +automatic selection disabled: + +```toml +[firmware_global] +autoupdate = false +run_interval = "30s" +max_uploads = 4 +concurrency_limit = 16 +host_firmware_upgrade_retry_interval = "60m" +instance_updates_manual_tagging = true +requires_manual_upgrade = false +max_concurrent_bfb_copies = 10 +``` + +The legacy `host_enable_autoupdate` and `host_disable_autoupdate` fields do not +provide a reliable model allowlist. Use the +[per-machine automatic-update policy](host-firmware.md#enable-or-disable-one-host) +instead. + +## Roll out a new baseline + +Treat a baseline change as a site-wide rollout, even when the first update is a +canary: + +1. Update the canonical configuration source. Do not maintain a second live + version table in this guide or a site runbook. +2. Make every referenced host artifact available. For DPU firmware, update the + BFB, accepted NIC versions, and `dpu_models` together. +3. Check host component ordering, defaults, pre-ingestion thresholds, and + inventory matchers. Then inspect the merged host catalog with `firmware + show`. +4. Configure a nonzero shared capacity, explicit-start windows, and assigned-host + approval policy. Keep site-wide automatic selection disabled while validating + one host through a per-machine host policy or an explicit DPU reprovisioning + request. +5. After verification, enable the intended site-wide automatic policy. +6. Monitor the applicable workflow until inventory reports the configured + versions and the host has returned to service. + +Changing a pre-ingestion threshold while `firmware_global.autoupdate` is enabled +can affect every matching endpoint on the next pre-ingestion pass. Validate the +catalog and artifact availability before applying that change. + +## Rack and component firmware + +Rack and component firmware do not use either of the configuration models +above. Their target version comes from the update request or from the default +selected by the rack or component backend. See +[Rack and component firmware upgrades](rack-component-firmware.md). diff --git a/docs/operations/firmware-updates/dpu-firmware.md b/docs/operations/firmware-updates/dpu-firmware.md new file mode 100644 index 0000000000..fbe4ce88cb --- /dev/null +++ b/docs/operations/firmware-updates/dpu-firmware.md @@ -0,0 +1,248 @@ +# DPU Firmware Updates + +DPU firmware is updated as part of DPU reprovisioning. NICo does not upload a +separate package for each DPU component. Instead, it installs or boots the +site's BlueField Bundle (BFB), power-cycles the platform, and then checks the +firmware reported by the DPU BMC. + +Keep two kinds of configuration separate: + +- `dpu_nic_firmware_update_versions` determines whether DPU NIC drift should + automatically start reprovisioning. +- `dpu_models` contains the BMC, ERoT/CEC, and NIC versions that NICo expects + after the BFB has been applied. + +These values and the BFB must describe the same release. See +[Configure firmware versions](configuration.md#dpu-firmware) before enabling +automatic updates. + +## Components covered + +The classic, non-DPF reprovisioning path uses the following Redfish inventory: + +| Component | Redfish inventory match | Role in the workflow | +|---|---|---| +| DPU OS and bundled firmware | `DPU_OS` / BFB | NICo installs `forge.bfb` through Redfish when supported, or boots the DPU into the network-install path. The BFB carries the DPU software and associated firmware payloads. | +| DPU NIC | `DPU_NIC` | Its reported version is the automatic drift trigger. NICo also verifies it after reprovisioning. | +| DPU BMC | An inventory ID containing `BMC_Firmware` | Verified after reprovisioning against the DPU model catalog. | +| ERoT/CEC | `Bluefield_FW_ERoT` | Verified after reprovisioning against the DPU model catalog. | +| DPU UEFI | `DPU_UEFI` | Read for inventory and topology data, but not compared in the current post-reprovision verification loop. | + +The distinction matters: BMC or ERoT/CEC drift alone does not make Machine +Update Manager select a DPU. A NIC mismatch starts the workflow; applying the +BFB is expected to bring the other verified components to the corresponding +baseline. + +## DPU NIC firmware + +### Drift detection + +Machine Update Manager periodically reads the DPU NIC version discovered for +each managed DPU. It trims surrounding whitespace and checks for an exact +match in `dpu_config.dpu_nic_firmware_update_versions`. + +For example: + +```toml +[dpu_config] +dpu_nic_firmware_reprovision_update_enabled = true +dpu_nic_firmware_update_versions = ["32.43.1014", "32.44.1030"] +``` + +A DPU reporting either listed version is compliant. A DPU reporting +`32.42.1000` is drifted. The list does not identify which version NICo will +install; the deployed BFB determines the resulting firmware versions. + +Setting `dpu_nic_firmware_reprovision_update_enabled = false` removes the DPU +NIC module from Machine Update Manager. NICo then stops detecting and +scheduling classic DPU NIC drift automatically. It does not disable DPU +reprovisioning requested through the API. + +### Eligibility and scheduling + +Drift is evaluated per DPU, but scheduling and disruption are managed per +host. If several DPUs attached to one host are drifted, NICo groups them into +one host update and counts that host once against the shared machine-update +limit. + +NICo automatically selects a host only when all of the following are true: + +| Check | Requirement | +|---|---| +| Managed-host state | The host is in top-level `Ready`. An assigned host is not automatically selected. | +| Health | The aggregate host health report has no alerts. | +| Existing work | The host is not already in another machine-update workflow, and none of its DPUs already has a reprovisioning request. | +| Site capacity | The shared `machine_updater.max_concurrent_machine_updates_*` policy has room for another host. DPU and host firmware work use the same budget. | + +When the host is eligible, Machine Update Manager creates a reprovisioning +request on each drifted DPU. The managed-host controller adds a +`HostUpdateInProgress` health alert with `PreventAllocations`, restarts the +selected DPUs, and enters DPU reprovisioning. + +```mermaid +flowchart TD + A["Read discovered DPU NIC version"] --> B{"Version is accepted?"} + B -- Yes --> C["No action"] + B -- No --> D{"Host is healthy, Ready, and within the update budget?"} + D -- No --> E["Report as unavailable and retry later"] + D -- Yes --> F["Create DPU reprovisioning request"] + F --> G["Mark host unavailable for allocation"] + G --> H["Reprovision DPU and verify firmware"] +``` + +### Reprovisioning and verification + +The installation mechanism depends on how the DPU is managed and on platform +capabilities: + +```mermaid +flowchart TD + A["DPU reprovisioning starts"] --> B{"DPF-managed DPU?"} + B -- Yes --> C["Ask DPF to reprovision the DPU"] + B -- No --> D{"Secure boot enabled and all DPUs support Redfish BFB install?"} + D -- Yes --> E["Install forge.bfb through Redfish UpdateService"] + D -- No --> F["Boot the DPU into the UEFI HTTP network-install path"] + C --> G["Wait for DPU readiness"] + E --> H["Power-cycle the host"] + F --> H + H --> I["Read BMC, ERoT/CEC, and NIC inventory"] + I --> J{"All versions match the DPU model baseline?"} + J -- No --> K["Wait and retry verification"] + J -- Yes --> G + G --> L["Wait for network configuration and health"] + L --> M["Clear reprovisioning request and return host to service"] +``` + +`dpu_config.dpu_enable_secure_boot` controls the classic installation branch +and defaults to `false`: + +- When it is `false`, NICo uses the UEFI HTTP network-install path. +- When it is `true`, NICo uses Redfish BFB installation only when every attached + DPU reports BMC firmware `24.10` or newer. If one DPU does not meet that + requirement or has no reported BMC version, NICo uses network installation. +- DPF-managed provisioning takes precedence and does not use this classic + secure-boot decision. + +This setting chooses how NICo applies the BFB. It does not enable automatic DPU +selection and does not choose the BFB or component target versions. + +For the classic path, version verification uses exact string equality. NICo +checks BMC, ERoT/CEC, and NIC in that order. It waits when an inventory entry +is missing, has no version, or reports a different version. DPU UEFI is not in +this verification loop. + +For DPF-managed DPUs, Machine Update Manager asks DPF for devices whose +installed BFB or DPU flavor differs from the active `DPUDeployment`. DPF drift +can therefore trigger the same host-level scheduling flow without consulting +the classic NIC accepted-version list. DPF owns reprovisioning, and NICo skips +the classic Redfish component comparison afterward. + +## Assigned hosts and operator requests + +An assigned host is not eligible for automatic DPU NIC selection. Its drift +contributes to the unavailable count until the host returns to top-level +`Ready`. + +An operator can +[request DPU reprovisioning explicitly](api:PATCH/v2/org/{org}/nico/machine/{machineId}/dpu/reprovision). +The API requires a `HostUpdateInProgress` health alert with +`PreventAllocations`; the CLI's `--update-message` option creates that alert +before sending the request: + +```bash +nico-admin-cli -a dpu reprovision set \ + --id \ + --update-message "" +``` + +Passing a host ID requests reprovisioning for all attached DPUs. Passing a DPU +ID requests it only for that DPU. On an unassigned `Ready` host, the controller +can start the request directly. On an assigned host, the request waits for +tenant reboot approval or the configured `instance_autoreboot_period`, as +described in +[Assigned hosts need reboot approval](host-firmware.md#assigned-hosts-need-reboot-approval). + +The CLI still accepts `--update-firmware`, but the current DPU state machine +does not branch on that value. Every classic DPU reprovision follows the same +BFB installation or network-install path and firmware verification. + +## Monitor an update + +Show the versions reported for all DPUs: + +```bash +nico-admin-cli -a dpu versions +``` + +The output includes DPU NIC, BMC, BIOS/UEFI, HBN, and DPU agent versions. Do +not use `dpu versions --updates-only` as the authoritative drift list: the +current implementation reads an empty legacy expected-version map and can +return all DPUs instead of only drifted ones. + +List pending and active DPU reprovisioning requests, then inspect the +managed-host substate for a particular host: + +```bash +nico-admin-cli -a dpu reprovision list +nico-admin-cli -a managed-host show +``` + +The following metrics provide the site-wide view: + +| Metric | Meaning | +|---|---| +| `carbide_pending_dpu_nic_firmware_update_count` | Drifted DPUs attached to hosts currently eligible for automatic selection. | +| `carbide_unavailable_dpu_nic_firmware_update_count` | Drifted classic DPUs whose host is not currently eligible. | +| `carbide_running_dpu_updates_count` | DPU reprovisioning requests marked as firmware updates that have started. | +| `carbide_dpu_firmware_version_count{firmware_version="..."}` | DPUs grouped by the firmware version reported in their DPU hardware information. | +| `carbide_firmware_updates_total{target="dpu_nic",phase="started"}` | Automatic DPU NIC update starts. This target does not emit a `completed` phase. | +| `carbide_firmware_update_failures_total{target="dpu_nic",cause="wrong_version_after_update"}` | Reprovisioning returned to `Ready`, but the reported NIC version is still outside the accepted list. | + +## Recover a stalled update + +Start with `dpu reprovision list` and `managed-host show`. The substate usually +separates an installation problem from a post-install health or configuration +problem. + +| Symptom | Check | +|---|---| +| `InstallDpuOs/WaitForInstallComplete` does not advance | Inspect the Redfish task and `nico-api` logs. Verify that the DPU BMC can reach `carbide-pxe.forge` and that `forge.bfb` is available. | +| `WaitingForNetworkInstall` does not advance | Check the DPU's UEFI HTTP boot, PXE requests in `nico-pxe`, and whether the DPU agent starts from the installed image. | +| `VerifyFirmareVersions` keeps waiting | Compare BMC, ERoT/CEC, and NIC Redfish inventory with `dpu_models`. A mismatch usually means that the deployed BFB and configured baseline do not describe the same release. | +| `WaitingForNetworkConfig` does not advance | Check DPU agent health and whether the managed-host network configuration versions are synchronized. | +| The host returned to `Ready`, but the update marker remains | Check the NIC version against `dpu_nic_firmware_update_versions`. NICo deliberately retains the update health marker when the resulting NIC version is not accepted. | + +Restart an active or failed reprovisioning flow for all requested DPUs on a +host: + +```bash +nico-admin-cli -a dpu reprovision restart \ + --id +``` + +Clear a request only when it has not started: + +```bash +nico-admin-cli -a dpu reprovision clear \ + --id +``` + +For DPU OS installation, networking, and health details beyond firmware, see +[DPU lifecycle management](../../dpu-management/dpu-lifecycle-management.md). + +## Current limitations + +- `dpu_nic_firmware_initial_update_enabled` does not currently suppress BFB + installation or firmware verification during initial DPU setup; the machine + controller only logs the value. +- The machine controller stores + `dpu_nic_firmware_reprovision_update_enabled`, but does not consult that field + while executing a reprovisioning request. The setting controls automatic + selection through Machine Update Manager. +- The `update_firmware` value stored on a DPU reprovisioning request is used by + status and metrics, but does not select a different execution path. +- Automatic completion emits no `carbide_firmware_updates_total` event with + `phase="completed"` for the `dpu_nic` target. Use reprovisioning state, the + running gauge, and resulting inventory for completion monitoring. +- Classic post-reprovision verification does not compare `DPU_UEFI`, even + though the DPU model catalog contains a UEFI entry. diff --git a/docs/operations/firmware-updates/host-firmware.md b/docs/operations/firmware-updates/host-firmware.md new file mode 100644 index 0000000000..58cfba3421 --- /dev/null +++ b/docs/operations/firmware-updates/host-firmware.md @@ -0,0 +1,382 @@ +# Managed Host Firmware Updates + +Managed host firmware upgrades keep an ingested host aligned with the default +versions in the +[effective host firmware catalog](configuration.md#host-firmware-catalog). +They are normally triggered by drift, scheduled by Machine Update Manager, and +executed by the managed-host state machine. + +This is different from [pre-ingestion](pre-ingestion.md). Pre-ingestion uses a +minimum-version threshold before the host is managed. The managed-host path +compares a managed host with its steady-state defaults and performs the work as +host reprovisioning. + +Three parts of NICo cooperate in this workflow: + +| Responsibility | NICo component | +|---|---| +| Determine whether reported firmware differs from the defaults. | Site Explorer and the host firmware catalog. | +| Decide whether another host may start an update. | Machine Update Manager. | +| Install, activate, and verify each component. | Managed-host state machine, with Redfish, Scout, or a local script. | + +## How an automatic update starts + +```mermaid +flowchart TD + A["Configured desired versions"] --> C["Compare versions"] + B["Latest reported versions"] --> C + C --> D{"Exact match?"} + D -- Yes --> Z["No update request"] + D -- No --> E{"Automatic update enabled for this host?"} + E -- No --> Z + E -- Yes --> F{"Explicit start required?"} + F -- No --> G{"Site concurrency budget available?"} + F -- Yes --> V{"Per-host update window active?"} + V -- No --> W["Keep pending"] + V -- Yes --> G + G -- No --> W + G -- Yes --> H["Create host reprovisioning request"] + H --> I{"Host has an assigned instance?"} + I -- No --> K["Begin managed-host firmware workflow"] + I -- Yes --> J{"Tenant approval or autoreboot period active?"} + J -- No --> W + J -- Yes --> L["Boot discovery environment"] + L --> K +``` + +### Drift detection + +Machine Update Manager snapshots the default version of each configured +component by vendor and model. Site Explorer independently stores the versions +reported by the host's matching Redfish inventory entries. A host is drifted +when these two normalized `Versions` maps are not exactly equal. + +This comparison is equality-based, not an ordering comparison. A version is +drifted whether it is older, newer, or formatted differently from the +configured default. `preingest_upgrade_when_below` is not used after ingestion. + +The catalog can change through the +[Host Firmware Config API](api:PUT/v2/org/{org}/nico/firmware-config/host) +or legacy/static configuration. Machine Update Manager refreshes its +desired-version snapshot when the runtime configuration changes or the legacy +firmware directory is modified. + +### Scheduling gates + +A drifted host becomes eligible only after the relevant gates pass: + +| Gate | Behavior | +|---|---| +| Automatic update policy | `firmware_global.autoupdate` supplies the site default. A per-machine override can enable or disable one host independently. | +| Host state | With `firmware_global.instance_updates_manual_tagging = true`, automatic selection is limited to unassigned hosts in `Ready`. When false, assigned hosts can also be selected. Unassigned hosts are considered first. | +| Existing work | A host with an existing reprovisioning request is not selected again. Machines already in maintenance or another update workflow count against the shared budget. | +| Explicit start | When the model has `explicit_start_needed = true`, the current time must be inside that machine's firmware update window. | +| Site capacity | `machine_updater.max_concurrent_machine_updates_absolute` defines the usable site-wide update ceiling. Machines already in maintenance consume this budget. | + +Automatic machine updates do not start when both concurrency limits are +unset, because the resulting update ceiling is zero. Machine Update Manager +runs every `machine_update_run_interval` seconds; the default is 300 seconds. + +> **Current limitation:** +> `machine_updater.max_concurrent_machine_updates_percent` is intended to set a +> percentage-based ceiling and account for unhealthy hosts. The current +> implementation passes the total and unhealthy host counts to that calculation +> in reverse, so setting the percentage option reduces the calculated ceiling +> to zero. Until that is corrected, configure +> `machine_updater.max_concurrent_machine_updates_absolute` and leave the +> percentage option unset. + +When a host is selected, NICo creates a host reprovisioning request. The +managed-host state machine then adds the `host-fw-update` health report. This +report prevents a currently unassigned host from being allocated and suppresses +external health alerting caused by expected update operations. + +### Enable or disable one host + +The per-machine policy overrides the global setting: + +```bash +nico-admin-cli -a machine auto-update \ + --machine --enable + +nico-admin-cli -a machine auto-update \ + --machine --disable + +nico-admin-cli -a machine auto-update \ + --machine --clear +``` + +`--clear` returns the host to the `firmware_global.autoupdate` policy. + +The static `firmware_global.host_enable_autoupdate` and +`firmware_global.host_disable_autoupdate` fields are legacy controls. The +current scheduler does not read `host_enable_autoupdate`, and it compares +`host_disable_autoupdate` entries only with machine IDs. A matching disabled +entry also ends host selection for that scheduler pass. Use the per-machine +command for new operational workflows. + +### Open an explicit-start window + +For a model with `explicit_start_needed = true`, set a window on the affected +machines: + +```bash +nico-admin-cli -a managed-host start-updates \ + --machines \ + --start \ + --end +``` + +Use `YYYY-MM-DDTHH:MM:SS+0000` for an explicit UTC offset, or +`YYYY-MM-DDTHH:MM:SS` for local time. +Omitting `--start` starts the window now. Omitting `--end` makes it last 24 +hours. Cancel a pending window with: + +```bash +nico-admin-cli -a managed-host start-updates \ + --machines --cancel +``` + +This command opens a scheduler gate; it does not create an update request by +itself. Models without `explicit_start_needed` do not require this window. + +### Assigned hosts need reboot approval + +An assigned host can have a pending reprovisioning request without immediately +interrupting its instance. The request starts when either: + +- the tenant approves updates while rebooting the instance; or +- the current time is inside the site-level + `machine_updater.instance_autoreboot_period`. + +Tenant approval is expressed as: + +```bash +nico-admin-cli -a instance reboot \ + --instance --apply-updates-on-reboot +``` + +The autoreboot period is separate from the per-machine firmware window. The +firmware window lets Machine Update Manager create the request for a model that +requires explicit start. The autoreboot period lets an assigned host consume an +existing request without waiting for tenant approval. + +After approval, NICo moves the assigned host through its platform configuration +and discovery-boot states before entering the same component-update loop used +for an unassigned host. `HostReprovision` is the internal state name; this is +not a return to pre-ingestion. + +### Request host reprovisioning directly + +An operator can bypass automatic drift selection and scheduling by creating a +host reprovisioning request: + +```bash +nico-admin-cli -a host reprovision set \ + --id \ + --update-message "" +``` + +The state machine still compares inventory with the catalog and returns the +host to its prior steady state if nothing needs an update. A direct request is +not constrained by Machine Update Manager's concurrency calculation, although +an assigned host still needs reboot approval. Use it only as part of an +operator-controlled maintenance procedure. + +The host reprovisioning command always enters firmware checking; its legacy +`--update-firmware` flag is not needed. A request that has not started can be +removed with `host reprovision clear --id `. + +## Upgrade sequence + +After the request can proceed, the managed-host state machine performs the +following sequence: + +1. **Protect the host.** NICo records the update health report and, where + needed, moves an assigned host into its discovery environment and disables + platform lockdown. +2. **Refresh inventory.** Site Explorer revisits the BMC. The state machine + selects the effective catalog definition by BMC vendor and host model. +3. **Select a component.** NICo walks `ordering` and finds the first component + for which any matching inventory entry differs from the default version. + Pre-ingestion-only firmware entries are excluded. +4. **Install it.** NICo chooses the installation route described below. + Multi-artifact entries are applied in sequence. +5. **Activate and verify.** For Scout and Redfish updates, NICo performs + component-specific resets, reboots, or configured power drains. A legacy + local script owns its own activation procedure. NICo then requests fresh + inventory and checks the reported versions. +6. **Repeat.** NICo returns to the beginning of `ordering` and selects the next + mismatched component. +7. **Restore service.** When no mismatch remains, NICo restores lockdown when + required, returns the host to `Ready` or its assigned-instance state, clears + the reprovisioning request, and removes the update health report. + +### Installation route + +The installation mechanism is selected per vendor, model, and component. It is +not a separate top-level workflow. + +```mermaid +flowchart TD + A["Ordered component differs from default"] --> B{"Packaged Scout script exists?"} + B -- Yes --> C["Send verified task and artifacts to Scout"] + B -- No --> D{"Firmware entry has legacy script?"} + D -- Yes --> E["Run local script from NICo Core"] + D -- No --> F["Download artifact and upload through Redfish"] + C --> G["Collect Scout result"] + E --> J["Refresh inventory and recheck component"] + F --> H["Poll Redfish task"] + H --> G + G --> I["Run NICo activation and verify through fresh inventory"] +``` + +The route precedence is: + +1. **Scout script.** NICo first looks for a packaged + `/opt/carbide/scout-firmware-scripts////upgrade.sh` + and its `metadata.toml`. If present, NICo sends Scout the verified script, + artifact URLs and digests, and execution timeouts. Scout downloads and + verifies the files, runs the script on the host, and reports the result. +2. **Core-local script.** If no Scout script exists and the firmware entry has + `script`, NICo runs that legacy script on the Core service with the BMC + address and credentials in its environment. The script is responsible for + installation and activation; after it succeeds, NICo returns to inventory + checking. +3. **Redfish.** Otherwise NICo resolves the configured artifact, waits for a + shared `firmware_global.max_uploads` slot, uploads asynchronously, and polls + the returned Redfish task. + +All successful routes eventually return to component selection. Scout and +Redfish updates first pass through NICo's activation and inventory-verification +states; the legacy script path returns directly to inventory checking. + +### Platform and OEM procedures + +NICo selects the configured component, starts the integrated installation +route, and verifies the resulting inventory. The approved platform or OEM +procedure remains authoritative for package selection, prerequisites, +preservation settings, activation requirements, and recovery. + +Before adding an artifact or script to the catalog, verify that it implements +the procedure for the exact platform and release. A successful Redfish task or +script exit only shows that the installation mechanism completed; NICo still +needs fresh inventory to confirm the configured version. See the +[Redfish workflow](../../architecture/redfish_workflow.md) for the request and +task ownership boundaries. + +### Manual platform gate + +When `firmware_global.requires_manual_upgrade` is enabled for an MNNVL-capable +host, NICo pauses in `WaitingForManualUpgrade` before selecting components. +GB200 is the common example. Complete the approved platform procedure, then +confirm it with: + +```bash +nico-admin-cli -a host reprovision \ + mark-manual-upgrade-complete --id +``` + +NICo then refreshes inventory and runs the normal component loop for anything +that still differs from the catalog. Marking the manual work complete does not +assert that every component is compliant; verification still comes from +discovered inventory. + +## Monitor and verify + +List pending or active host reprovisioning requests: + +```bash +nico-admin-cli -a host reprovision list +``` + +Inspect the current state and recent state history: + +```bash +nico-admin-cli -a managed-host show +nico-admin-cli -a machine show --history-count 10 +``` + +Use the host BMC address to inspect the firmware inventory last stored by Site +Explorer: + +```bash +nico-admin-cli -a site-explorer get-report \ + managed-host +``` + +The principal host firmware substates are: + +| State | Meaning | +|---|---| +| `CheckingFirmwareV2`, `CheckingFirmwareRepeatV2` | Refresh inventory and select the next mismatched component. | +| `WaitingForManualUpgrade` | Wait for completion of the approved platform procedure. | +| `WaitingForScoutUpgrade` | Wait for Scout to execute the packaged upgrade task and report its result. | +| `WaitingForScript` | Wait for the legacy Core-local script. | +| `WaitingForUpload` | A Redfish upload is running asynchronously. | +| `WaitingForFirmwareUpgrade` | The upload completed and NICo is polling the Redfish task. | +| `ResetForNewFirmware` | NICo is performing activation resets, reboots, or power drains. | +| `NewFirmwareReportedWait` | NICo is refreshing inventory until all matching entries report the target. | +| `FailedFirmwareUpgrade` | The attempt failed; NICo is waiting to retry or has exhausted its retry budget. | + +See the [managed-host state machine](../../architecture/state_machines/managedhost.md) +for these firmware substates in the complete host lifecycle. + +Completion requires more than leaving an installation state. Verify that the +host returned to `Ready` or `Assigned/Ready`, the reprovisioning request is no +longer listed, and Site Explorer reports the configured default versions. + +Useful metrics are: + +| Metric | What it shows | +|---|---| +| `carbide_pending_host_firmware_update_count` | Hosts currently selected by the drift query as needing firmware work. | +| `carbide_active_host_firmware_update_count` | Hosts whose top-level state is `HostReprovision`. Assigned-host firmware substates are not included by this query. | +| `carbide_exhausted_reprovision_retry_count` | Unassigned hosts whose top-level firmware retry budget is exhausted. Assigned-host update substates are not included. | +| `carbide_firmware_updates_total{target="host"}` | Host updates started and completed, separated by the `phase` label. | +| `carbide_host_reprovision_retries_total` | Automatic retries of failed host firmware reprovisioning. | +| `carbide_machines_in_maintenance_count` | Machines consuming the shared maintenance budget. | +| `carbide_concurrent_machine_updates_available` | The calculated site-wide concurrency ceiling for machine updates. | + +See [Core metrics](../../observability/core_metrics.md) for the common metric +labels and collection endpoint. + +## Failures and recovery + +Transient upload failures and interrupted asynchronous uploads return to +firmware checking and are attempted again. A failed Scout result, failed local +script, terminal Redfish task, or firmware that exhausts its activation-reset +retries moves the host to `FailedFirmwareUpgrade`. + +NICo waits `firmware_global.host_firmware_upgrade_retry_interval` before +retrying that workflow. The default interval is 60 minutes. An unassigned host +in the top-level `HostReprovision` state has a budget of five retries. After it +is exhausted, the host remains failed and is counted by +`carbide_exhausted_reprovision_retry_count`. + +> **Current limitation:** Assigned-host firmware work stores its substate under +> `Assigned` without a persisted host retry counter. It therefore retries at +> the configured interval without reaching the five-retry limit or the +> exhausted-retry metric. + +Correct the artifact, script, BMC, inventory matcher, or platform condition +before forcing another attempt. Then reset host reprovisioning to firmware +checking and, for an unassigned host, clear the retry count: + +```bash +nico-admin-cli -a managed-host \ + reset-host-reprovisioning --machine +``` + +This command does not repair the platform or change the catalog. It only asks +the state machine to start firmware checking again. + +A Core-local script interrupted by a NICo restart is not resumed automatically +and can remain in `WaitingForScript`; use the same reset command after verifying +that the old script is no longer running. + +If a host repeatedly appears as pending but immediately leaves the firmware +workflow, compare the catalog's `current_version_reported_as` matcher with the +latest Redfish inventory. A missing matching inventory entry makes the desired +and reported `Versions` maps differ, but leaves the execution path without a +component it can install or verify. diff --git a/docs/operations/firmware-updates/pre-ingestion.md b/docs/operations/firmware-updates/pre-ingestion.md new file mode 100644 index 0000000000..5298c69553 --- /dev/null +++ b/docs/operations/firmware-updates/pre-ingestion.md @@ -0,0 +1,184 @@ +# Pre-ingestion Firmware Updates + +Pre-ingestion firmware upgrades run against a discovered host BMC before NICo +creates the managed host. Their purpose is to update firmware that is too old +for the normal ingestion and management workflow. + +This path is driven by the pre-ingestion manager. It does not use Machine +Update Manager, host update windows, `explicit_start_needed`, or per-host +automatic-update overrides. A machine ID might not exist yet, so the only +automatic-update switch used by this path is `firmware_global.autoupdate`. + +This page covers host firmware selected from the +[host firmware catalog](configuration.md#host-firmware-catalog). DPU BFB +recovery also runs in the pre-ingestion state machine, but installs a DPU boot +image to recover the DPU; it is not a host firmware-catalog workflow. + +## When this path applies + +A host firmware update starts during pre-ingestion when all of the following +are true: + +1. Site exploration has produced a BMC vendor, host model, and firmware + inventory for an endpoint that has not completed pre-ingestion. +2. The effective host firmware catalog contains a definition for that vendor + and model. +3. At least one reported component has + `preingest_upgrade_when_below` configured and its reported version is lower + than that threshold. +4. `firmware_global.autoupdate` is enabled. + +The reported version must come from an inventory entry matched by +`current_version_reported_as`. A missing catalog definition, missing inventory +match, or missing version cannot trigger the update. If no other component +triggers it, NICo completes the firmware portion of pre-ingestion and continues +with ingestion. + +When `firmware_global.autoupdate` is disabled, NICo still checks for versions +below the threshold, but it does not install firmware or block ingestion. It +marks pre-ingestion complete. The host model and machine ID allowlists used by +managed-host updates are not consulted here. + +## Trigger and target + +`preingest_upgrade_when_below` decides whether firmware work starts; it does not +specify the version NICo installs. NICo selects an installable entry from +`known_firmware` that is marked `preingestion_exclusive_config` or `default`. +A pre-ingestion-only entry allows the ingestion target to differ from the +steady-state default. + +Once any component is below its threshold, NICo walks the model's `ordering` +and updates every ordered component whose reported version differs from its +pre-ingestion target. A component does not need to be below its own threshold +at this point. If no ordering is configured, the legacy default is BMC followed +by UEFI. + +For example: + +| Component | Minimum | Target | Reported on host A | Reported on host B | +|---|---|---|---|---| +| BMC | `7.00` | `7.10` | `6.50` | `7.05` | +| UEFI | `1.50` | `1.80` | `1.60` | `1.60` | + +- Host A triggers pre-ingestion firmware work because its BMC is below `7.00`. + NICo updates both BMC and UEFI because both differ from their targets. +- Host B does not trigger pre-ingestion firmware work because both components + meet their minimums. After ingestion, both can still be treated as drifted + from their steady-state targets. + +## Upgrade sequence + +```mermaid +flowchart TD + A["Fresh BMC exploration report"] --> B["BMC reset and time checks"] + B --> C{"Any reported component below its minimum?"} + C -- No --> J["Complete pre-ingestion"] + C -- Yes --> D{"Global autoupdate enabled?"} + D -- No --> J + D -- Yes --> E["Select next mismatched component in ordering"] + E --> F["Run upgrade script or upload artifacts through Redfish"] + F --> G["Wait for installation and perform activation resets"] + G --> H{"Fresh inventory reports the target?"} + H -- No --> K["Wait for inventory or retry activation"] + K --> H + H -- Yes --> I{"Another ordered component needs an update?"} + I -- Yes --> E + I -- No --> J +``` + +The state machine performs the following work: + +1. **Prepare the BMC.** NICo attempts an initial BMC reset, waits for a fresh + exploration report, configures the site's NTP servers, and checks BMC time. + Firmware comparison uses the refreshed inventory. +2. **Evaluate minimums.** NICo selects the effective catalog definition by BMC + vendor and host model, then compares matched inventory versions with each + configured pre-ingestion threshold. +3. **Select one component.** If a minimum is breached, NICo follows `ordering` + and selects the first component whose reported version differs from its + pre-ingestion target. Components are handled serially for each endpoint. +4. **Install the firmware.** A firmware entry with `script` runs the configured + local upgrade script. Otherwise NICo resolves the artifact, waits for a + shared upload slot, and uploads it through Redfish. Multi-artifact entries + are installed in sequence. +5. **Activate and verify.** NICo waits for the Redfish task, performs the + component-specific reset, reboot, or configured power drains, and requests + a fresh exploration report. When the matching inventory entry is present, + NICo waits until its reported version matches the target. A missing entry + cannot be verified or selected for another update. +6. **Continue ingestion.** NICo repeats target selection for the remaining + ordered components. When none need an update, it marks pre-ingestion + complete and the normal ingestion process can continue. + +`firmware_global.max_uploads` limits concurrent uploads across firmware +workflows. `firmware_global.concurrency_limit` limits how many endpoints the +pre-ingestion manager processes concurrently; it does not allow parallel +component updates on one endpoint. + +## Monitor progress + +Use the BMC address to inspect the latest exploration report and pre-ingestion +state: + +```bash +nico-admin-cli -a site-explorer get-report endpoint +``` + +The states most relevant to host firmware are: + +| State | Meaning | +|---|---| +| `InitialBMCReset`, `SetNtpServers`, `TimeSyncReset` | NICo is preparing the BMC and obtaining reliable inventory before comparing versions. | +| `UpgradeFirmwareWait` | The firmware was uploaded and NICo is polling the Redfish task. | +| `ScriptRunning` | The configured firmware upgrade script is running. | +| `ResetForNewFirmware` | Installation completed and NICo is performing activation resets or power drains. | +| `NewFirmwareReportedWait` | NICo is waiting for refreshed inventory to report the target version. | +| `RecheckVersions`, `RecheckVersionsAfterFailure` | NICo is checking the current component again or selecting the next one. | +| `Complete` | No applicable pre-ingestion update remains, or firmware autoupdate was disabled. Ingestion can continue. | +| `Failed` | Pre-ingestion stopped and requires operator action. The state includes the reason. | + +`Complete` does not mean every component matches its steady-state default. It +means no pre-ingestion update remains after applying the rules above. + +The following metrics provide site-wide visibility: + +| Metric | What it shows | +|---|---| +| `carbide_preingestion_total` | Endpoints currently being evaluated by the full pre-ingestion state machine. | +| `carbide_preingestion_waiting_download` | Endpoints whose firmware work was deferred because no upload slot was available. | +| `carbide_preingestion_waiting_installation` | Endpoints waiting for a Redfish firmware task to complete. | +| `carbide_preingestion_firmware_upload_total` | Upload attempts by method and outcome. | +| `carbide_preingestion_firmware_upgrade_tasks_total` | Completed and failed Redfish tasks by component and final task state. | +| `carbide_preingestion_power_control_total` | Host, BMC, and chassis power operations by operation and outcome. | + +## Failures and retries + +NICo retries most transient conditions on later pre-ingestion passes. These +include an unavailable BMC, a busy upload limit, artifact download still in +progress, Redfish upload errors, and inventory that has not yet refreshed. A +failed Redfish task causes NICo to refresh inventory and evaluate the component +again. If an installed version is still not reported after 30 minutes, NICo can +repeat the activation reset unless `firmware_global.no_reset_retries` is set. + +Some failures, including an unsuccessful upgrade script or exhausted BMC time +synchronization attempts, move the endpoint to `Failed`. Correct the underlying +problem, then clear the error to restart pre-ingestion from `Initial`: + +```bash +nico-admin-cli -a site-explorer clear-error +``` + +The command resets only a terminal `Failed` pre-ingestion state; it does not +interrupt an update that is still progressing. + +An upgrade script interrupted by a NICo restart is not resumed automatically +and can remain in `ScriptRunning`. After confirming that the script and firmware +operation are no longer running, an operator can delete the unpaired explored +endpoint as a last resort and allow it to be discovered again: + +```bash +nico-admin-cli -a site-explorer delete --address +``` + +NICo rejects explored-endpoint deletion after the endpoint belongs to a managed +host. diff --git a/docs/operations/firmware-updates/rack-component-firmware.md b/docs/operations/firmware-updates/rack-component-firmware.md new file mode 100644 index 0000000000..1ee7056948 --- /dev/null +++ b/docs/operations/firmware-updates/rack-component-firmware.md @@ -0,0 +1,331 @@ +# Rack and Tray Firmware Updates + +Firmware updates for racks, compute trays, NVSwitches, and power shelves are +explicit operations. NICo does not start them from the host or DPU drift +detectors described elsewhere in this guide. A Provider Admin submits a REST +request, and NICo Flow creates one or more asynchronous tasks to sequence the +work. + +Use this path when an operator needs Flow to update a rack or selected tray. +For desired-state updates on a conventional managed host, use +[Host firmware upgrades](host-firmware.md) instead. + +## How the request runs + +The REST response confirms that the task was created; it does not mean that +the firmware update has completed. + +```mermaid +flowchart LR + A["Provider Admin submits rack or tray request"] --> B["Flow resolves targets and an operation rule"] + B --> C["Wait for affected hosts to be ready"] + C --> D["Start update through NICo Core"] + D --> E["Component backend applies firmware"] + E --> F["Flow polls component status"] + F --> G{"Terminal result?"} + G -- No --> F + G -- Success --> H["Task succeeds"] + G -- Failure or timeout --> I["Task fails with a report"] +``` + +Before dispatching a disruptive operation, Flow checks persisted component +readiness. A compute-tray update checks the targeted hosts. An NVSwitch or +power-shelf update checks the hosts in the owning rack, because those devices +can affect every tenant in that rack. Flow waits for up to 30 minutes and +checks every 5 seconds by default. + +If readiness information is missing, the current implementation logs a warning +and allows the operation to proceed. Do not treat the readiness check as a +substitute for confirming the maintenance scope and tenant impact. + +## Choose the endpoint + +All endpoints require the Provider Admin role. `siteId` is required in every +request body and must identify a site owned by the provider organization. + +| Scope | Endpoint | Selection | +|---|---|---| +| One rack | [`PATCH /v2/org/{org}/nico/rack/{rack-id}/firmware`](api:PATCH/v2/org/{org}/nico/rack/{id}/firmware) | The rack in the URL. | +| Several or all racks | [`PATCH /v2/org/{org}/nico/rack/firmware`](api:PATCH/v2/org/{org}/nico/rack/firmware) | Optional `filter.names`. No filter means every rack in the site. | +| One tray | [`PATCH /v2/org/{org}/nico/tray/{tray-id}/firmware`](api:PATCH/v2/org/{org}/nico/tray/{id}/firmware) | The tray in the URL. | +| Several or all trays | [`PATCH /v2/org/{org}/nico/tray/firmware`](api:PATCH/v2/org/{org}/nico/tray/firmware) | Optional `filter`. No filter means all compute, NVSwitch, and power-shelf trays in every site rack. | + +The batch tray filter supports: + +| Filter | Meaning | +|---|---| +| `rackId` or `rackName` | Select trays in one rack. These fields are mutually exclusive. | +| `type` | Select exactly `Compute`, `NVSwitch`, or `PowerShelf`. | +| `componentIds` | Select component IDs of the specified `type`; `type` is required. | +| `ids` | Select tray UUIDs. | +| `slotId` | Select a rack slot; requires `rackId` or `rackName`. | + +Rack selection cannot be combined with `ids` or `componentIds`. Prefer a +single-rack or single-tray request for the first update of a new firmware +bundle. An unfiltered batch request can affect the entire site. + +## Describe the update + +The request has four controls in addition to `siteId`: + +| Field | Purpose | +|---|---| +| `version` | Target passed to the component backend. For current rack-scale RMS paths, this is a complete SOT firmware-object JSON document serialized as a string. Legacy backends can accept a plain version string. | +| `targets` | Optional component subset for tray requests. When present, `version` must also be present. Rack handlers do not forward this field, so do not send it with a rack request. | +| `ruleId` | Pins the task to a custom Flow operation rule. When omitted, Flow resolves a rule and falls back to its built-in firmware rule. | +| `overrideReadinessCheck` | Bypasses Flow's readiness gate and tells Core to bypass its state controller where supported. Use only during supervised maintenance after tenant impact has been accepted. | + +Although the API permits `version` to be omitted when `targets` is empty, that +is not portable across component backends. Rack-scale RMS updates require SOT +JSON. Supply an explicit version unless the selected backend and operation rule +are known to resolve one. + +### SOT firmware-object JSON + +RMS firmware is described by a source-of-truth (SOT) firmware object: a JSON +document containing the bundle identity and the artifacts RMS must apply. The +document comes from the platform's firmware release process; it is not the +same as the host firmware catalog described in +[Configure firmware versions](configuration.md). + +The REST `version` field is a string, so the complete JSON document must be +serialized into that string. Build the request body with a JSON tool instead +of escaping the document by hand: + +```sh +SOT_JSON=$(jq -c . compute-firmware-object.json) + +jq -n \ + --arg siteId "$SITE_ID" \ + --arg version "$SOT_JSON" \ + '{siteId: $siteId, version: $version, targets: ["bmc", "bios"]}' +``` + +For a rack request that needs a different value for each component type, +`version` can contain a layered JSON document with `compute`, `nvswitch`, and +`powershelf` keys. Flow extracts the relevant value before calling each +component manager. For example, this builds a layered value from two complete +SOT documents: + +```sh +COMPUTE_SOT=$(jq -c . compute-firmware-object.json) +SWITCH_SOT=$(jq -c . switch-firmware-object.json) + +LAYERED_VERSION=$(jq -cn \ + --argjson compute "$COMPUTE_SOT" \ + --argjson nvswitch "$SWITCH_SOT" \ + '{compute: $compute, nvswitch: $nvswitch}') +``` + +If a layered document omits a component-type key, Flow passes an empty target +to that component manager. Use an operation rule that excludes the component +instead of relying on an empty value to skip it. + +The REST surface does not accept an RMS artifact access token. Core sends +`NOAUTH` when no token is available, so artifacts referenced by these requests +must be reachable without a separate token. + +## Submit an update + +This request updates only the BMC and BIOS targets on one compute tray: + +```json +{ + "siteId": "2b88bb63-9a21-4bad-b113-68a54aa6e3dd", + "version": "", + "targets": ["bmc", "bios"] +} +``` + +The response contains the task IDs to monitor: + +```json +{ + "taskIds": ["c5e00a88-9b42-4e2b-a237-63e787f698ef"] +} +``` + +To update all racks named `A01` and `A02`, submit a batch rack request: + +```json +{ + "siteId": "2b88bb63-9a21-4bad-b113-68a54aa6e3dd", + "filter": { + "names": ["A01", "A02"] + }, + "version": "" +} +``` + +## Default sequencing + +When no site-specific rule or `ruleId` applies, Flow uses this built-in rule: + +| Stage | Component type | Status polling | Attempts | +|---|---|---|---| +| 1 | Compute | Every 2 minutes, for up to 45 minutes | 2 | +| 2 | NVSwitch | Every 2 minutes, for up to 45 minutes | 2 | + +The stages run in order. A failed or timed-out stage fails the task. A stage +that has no matching components is reported as skipped. + +The built-in rule deliberately excludes power shelves and does not perform an +AC power cycle after flashing. Use an approved custom operation rule for power +shelves. If firmware activation requires a power cycle, submit the appropriate +power-recycle task separately or include it in a custom rule. See the +[Flow operation rules guide](../../../rest-api/flow/docs/operation-rules-guide.md). + +## Component behavior + +### Compute trays + +Supported `targets` are: + +`bmc`, `bios`, `cec`, `nic`, `cpld_mb`, `cpld_pdb`, `hgx_bmc`, +`combined_bmc_uefi`, `gpu`, and `cx7`. + +Omitting `targets` asks Core to update all components represented by the +selected firmware bundle. It does not include DPU reprovisioning. + +`dpu` is a special, explicit-only compute target. NICo first submits any +compute-tray update, then reprovisions the DPU on each selected host serially. +The request's `version` is ignored for the DPU branch; its target comes from +site configuration. Use `targets: ["dpu"]` for a DPU-only request. See +[DPU firmware upgrades](dpu-firmware.md#assigned-hosts-and-operator-requests) +before using this path. + +Core routes conventional standalone machines through the managed-host firmware +workflow and rack-scale compute trays through the configured rack state +controller or component backend. Consequently, a single REST shape can start +different internal workflows depending on the hardware model. + +### NVSwitches + +Supported `targets` are `bmc`, `cpld`, `bios`, and `nvos`. Omitting `targets` +passes an empty component list to Core, which means all supported switch +components for the selected backend. + +When `version` is omitted, Flow first compares the switches' inventory with +Core's desired switch versions and skips the call if every switch is already +current. If an update is needed, the backend must still be able to resolve an +empty target. Supply a target explicitly for predictable behavior. + +### Power shelves + +Supported `targets` are `pmc` and `psu`. If `targets` is omitted, the current +Flow component manager updates only `pmc`. + +Power shelves are not present in the built-in firmware rule, so a power-shelf +tray request needs an operation rule that contains a `PowerShelf` firmware +step. Core's power-shelf state-controller path is not implemented; updates are +currently dispatched through the configured direct backend. + +## Monitor and cancel tasks + +Use [Retrieve a Task](api:GET/v2/org/{org}/nico/task/{id}) +to read each returned task ID until it reaches `Succeeded`, `Failed`, or +`Terminated`: + +```http +GET /v2/org/{org}/nico/task/{task-id}?siteId={site-id} +``` + +The task report records each rule stage and component step as `pending`, +`running`, `completed`, `failed`, or `skipped`. On failure, inspect both the +top-level `message` and the report's stage or step `error`. + +You can also +[list tasks for a rack](api:GET/v2/org/{org}/nico/rack/{id}/task) +or [list tasks for a tray](api:GET/v2/org/{org}/nico/tray/{id}/task). + +The `activeOnly=true` query parameter restricts the result to non-terminal +tasks, and the `includeReport=true` query parameter includes the stage and +step report. + +```http +GET /v2/org/{org}/nico/rack/{rack-id}/task?siteId={site-id}&activeOnly=true&includeReport=true +GET /v2/org/{org}/nico/tray/{tray-id}/task?siteId={site-id}&activeOnly=true&includeReport=true +``` + +[Cancel a Task](api:POST/v2/org/{org}/nico/task/{id}/cancel) +is best effort. It terminates a pending, running, or waiting task, but it cannot +undo firmware work already accepted by a hardware backend. Completed and +failed tasks cannot be cancelled. + +```text +POST /v2/org/{org}/nico/task/{task-id}/cancel + +{"siteId":""} +``` + +After cancellation or failure, inspect component status before retrying. Do not +assume that every component remained on its previous version. + +## Inspect Core component status + +The `component-manager` commands call NICo Core directly. Use them to inspect +the backend's view of a running REST task or for platform-specific recovery. +They do not create a Flow task or apply its operation rule. + +```sh +nico-admin-cli component-manager get-firmware-update-status rack \ + --rack-id + +nico-admin-cli component-manager get-firmware-update-status compute-tray \ + --machine-id + +nico-admin-cli component-manager get-firmware-update-status switch \ + --switch-id + +nico-admin-cli component-manager get-firmware-update-status power-shelf \ + --power-shelf-id + +nico-admin-cli component-manager get-firmware-versions switch \ + --switch-id +``` + +Core also exposes direct update commands. Prefer the REST workflow for normal +rack operations because it provides readiness checks, ordering, task reports, +and cancellation. Direct commands require the operator to provide those +safeguards. + +For RMS-backed compute trays or switches, pass a SOT file rather than embedding +it on the command line: + +```sh +nico-admin-cli component-manager update-firmware compute-tray \ + --machine-id \ + --sot-json-file ./compute-firmware-object.json + +nico-admin-cli component-manager update-firmware switch \ + --switch-id \ + --sot-json-file ./switch-firmware-object.json \ + --component bmc,cpld,bios,nvos + +nico-admin-cli component-manager update-firmware power-shelf \ + --power-shelf-id \ + --target-version \ + --component pmc,psu +``` + +Run `nico-admin-cli component-manager update-firmware --help` for the complete +command options. See the +[rack state machine](../../architecture/state_machines/rackstatemachine.md) for +lower-level execution details. + +## Troubleshooting + +| Symptom | Check | +|---|---| +| No work starts after the REST response | Read the returned task. It may be waiting at the readiness gate or for an earlier rule stage. | +| Task fails after about 30 minutes | Inspect the error for component IDs blocked by the readiness gate. Confirm tenant state and the persisted component operation status. | +| Stage times out | Check Core and backend status. The built-in firmware rule polls for 45 minutes per attempt; a backend job can still be running when Flow times out. | +| Rack-scale update rejects `version` | Confirm that `version` contains a valid SOT JSON object, serialized as a string, and that referenced artifacts are reachable without a REST-supplied access token. | +| Power-shelf request succeeds without updating a shelf | Confirm that the resolved operation rule contains a `PowerShelf` step. The built-in rule excludes power shelves. | +| Firmware was flashed but is not active | Determine whether the platform requires an AC cycle. The built-in firmware rule does not include one. | +| Retry begins from an uncertain state | Inspect per-component status and inventory first. A Flow task failure or cancellation does not roll hardware back. | + +Set `overrideReadinessCheck: true` only after diagnosing a readiness block and +confirming that the affected hardware is in a supervised maintenance window. +The override bypasses a tenant-safety guard and may also send the update +directly to the component backend instead of through Core's state controller. From c4a06d463027efd38e4e3e0be90863f177311df7 Mon Sep 17 00:00:00 2001 From: Alex Ball Date: Fri, 24 Jul 2026 16:11:24 -0700 Subject: [PATCH 2/2] last style/formatting tweaks Signed-off-by: Alex Ball --- docs/operations/firmware-updates.md | 30 +++++++++---------- .../firmware-updates/configuration.md | 30 +++++++++---------- .../firmware-updates/dpu-firmware.md | 10 +++---- .../firmware-updates/host-firmware.md | 26 ++++++++-------- .../firmware-updates/pre-ingestion.md | 20 ++++++------- .../rack-component-firmware.md | 12 ++++---- 6 files changed, 64 insertions(+), 64 deletions(-) diff --git a/docs/operations/firmware-updates.md b/docs/operations/firmware-updates.md index 7b13639e7b..a681047e89 100644 --- a/docs/operations/firmware-updates.md +++ b/docs/operations/firmware-updates.md @@ -1,6 +1,6 @@ # Firmware Updates -This guide first gives a high-level view of firmware upgrades in NICo and helps +This guide first gives a high-level view of firmware updates in NICo and helps you choose the workflow that applies to your hardware. The linked workflow pages then explain their configuration, prerequisites, execution, verification, and recovery in detail. @@ -10,8 +10,8 @@ and recovery in detail. NICo uses a site-wide desired-state process for managed host firmware and DPU NIC firmware. Operators configure the versions expected at the site. When automatic updates are enabled, NICo compares those versions with the firmware -reported by managed hardware. Drift makes a device eligible for an upgrade, -which NICo schedules automatically instead of requiring a one-off upgrade +reported by managed hardware. Drift makes a device eligible for an update, +which NICo schedules automatically instead of requiring a one-off update command. The drift-driven workflow is: @@ -22,11 +22,11 @@ flowchart LR B --> C{"Drift detected?"} C -- No --> D["No action"] C -- Yes --> E["Wait until the device is eligible"] - E --> F["Upgrade or reprovision"] + E --> F["Update or reprovision"] F --> G["Rediscover and verify"] ``` -Detecting drift does not mean the upgrade starts immediately. NICo may wait for +Detecting drift does not mean the update starts immediately. NICo may wait for the machine to enter an eligible state, an update window to open, or site-wide concurrency limits to allow more work. The responsible controller then performs the update or reprovisioning and verifies the result from newly discovered @@ -41,8 +41,8 @@ baseline. Not every firmware workflow begins with drift detection. Rack-scale compute trays, NVSwitches, and power shelves can be updated through explicit API requests. Some platforms require a vendor tool, a script, rack maintenance, or -a manually confirmed procedure. See -[Choose the upgrade path](#choose-the-upgrade-path) for the path used by each +a manually confirmed procedure. Refer to +[Choose the update path](#choose-the-update-path) for the path used by each component. ## How automatic work is scheduled @@ -54,10 +54,10 @@ the disruptive work and the return to service. On each scheduler pass, NICo: 1. clears bookkeeping for updates that have completed; -2. counts the distinct hosts already in maintenance or an update workflow; -3. calculates the remaining capacity from the site-wide machine-update limit; -4. asks the enabled DPU and host firmware modules for eligible work; and -5. creates requests until the shared capacity is full. +1. counts the distinct hosts already in maintenance or an update workflow; +1. calculates the remaining capacity from the site-wide machine-update limit; +1. asks the enabled DPU and host firmware modules for eligible work; and +1. creates requests until the shared capacity is full. ```mermaid flowchart LR @@ -77,7 +77,7 @@ machine update starts. Pre-ingestion firmware is not scheduled by Machine Update Manager. It has a separate polling interval and concurrency limit because the hardware has not -yet entered the managed-host lifecycle. See the +yet entered the managed-host lifecycle. Refer to the [Machine Update Manager architecture](../architecture/overview.md#machine-update-manager) for the broader ownership model. @@ -108,15 +108,15 @@ DPU NIC firmware uses an accepted-version list in the site configuration instead of the host catalog. Rack and component workflows receive their target from an explicit request or their component backend. -See [Configure firmware versions](firmware-updates/configuration.md) for the +Refer to [Configure Firmware Versions](firmware-updates/configuration.md) for the configuration models, setting reference, examples, and source precedence. -## Choose the upgrade path +## Choose the update path Use the table below to find the path for the hardware and component you are updating. For hosts, the path also depends on whether ingestion has completed. -| Your situation | Upgrade path | How it begins | +| Your situation | Update path | How it begins | |---|---|---| | A host has not completed ingestion and a reported component version is below `preingest_upgrade_when_below`. | [Pre-ingestion](firmware-updates/pre-ingestion.md) | The pre-ingestion manager detects the condition. When global automatic updates are enabled, it selects the default or pre-ingestion-only firmware entry and starts the update. | | Firmware on a managed host differs from the effective default in the merged firmware catalog. | [Host firmware](firmware-updates/host-firmware.md) | When host automatic updates are enabled, NICo detects the drift and schedules host reprovisioning. If `explicit_start_needed` is enabled, NICo first waits for the host's update window. | diff --git a/docs/operations/firmware-updates/configuration.md b/docs/operations/firmware-updates/configuration.md index c51d985b7b..908bae6d2f 100644 --- a/docs/operations/firmware-updates/configuration.md +++ b/docs/operations/firmware-updates/configuration.md @@ -7,7 +7,7 @@ supplied by the request or selected by their backend. The firmware catalog answers **what firmware NICo recognizes and should install**. Start there: define host firmware through the -[Host Firmware Config API](api:PUT/v2/org/{org}/nico/firmware-config/host) +[Host Firmware Config API](api:PUT/v2/org/:org/nico/firmware-config/host) or legacy metadata. Later sections cover DPU baselines and the site controls that determine **when NICo looks for work** and **how much work it may start**. @@ -73,17 +73,17 @@ For example, assume the BMC minimum is `7.00.00.00` and the default is | `7.05.00.00` | Meets the minimum, so ingestion can continue. After ingestion it is still drifted from `7.10.30.00`. | | `7.10.30.00` | Meets the minimum and matches the desired version. No firmware update is needed. | -See [Pre-ingestion firmware](pre-ingestion.md) for minimum-version enforcement -and [managed host firmware](host-firmware.md) for post-ingestion drift handling. +Refer to [Pre-ingestion Firmware Updates](pre-ingestion.md) for minimum-version enforcement +and [Managed Host Firmware Updates](host-firmware.md) for post-ingestion drift handling. ### Configure host firmware through the API The current way to configure host firmware is the -[Create or Update Host Firmware Config](api:PUT/v2/org/{org}/nico/firmware-config/host) +[Create or Update Host Firmware Config](api:PUT/v2/org/:org/nico/firmware-config/host) operation: ```text -PUT /v2/org/{org}/nico/firmware-config/host +PUT /v2/org/:org/nico/firmware-config/host ``` The request is site-scoped and keyed by `(vendor, model)`. NICo stores the @@ -138,7 +138,7 @@ mappings. The request is rejected when no mapping exists. component. The -[Delete Host Firmware Config](api:DELETE/v2/org/{org}/nico/firmware-config/host) +[Delete Host Firmware Config](api:DELETE/v2/org/:org/nico/firmware-config/host) operation removes the runtime entry. If the same vendor and model also exists in static or legacy metadata, that lower-priority definition becomes effective again. @@ -270,7 +270,7 @@ version `32.44.1030`, that version must be accepted by `dpu_nic_firmware_update_versions` and should be the NIC baseline in the matching `dpu_models` entry. -See [DPU firmware upgrades](dpu-firmware.md) for how NICo acts on this +Refer to [DPU firmware updates](dpu-firmware.md) for how NICo acts on this configuration. ## Operational controls @@ -305,7 +305,7 @@ instance_autoreboot_period.end = "2026-08-15T05:00:00Z" The autoreboot period is not the per-host firmware window used by models with `explicit_start_needed`. Operators open that window through the managed-host -API. See [Scheduling gates](host-firmware.md#scheduling-gates). +API. Refer to [Scheduling gates](host-firmware.md#scheduling-gates). ### Firmware execution @@ -358,17 +358,17 @@ canary: 1. Update the canonical configuration source. Do not maintain a second live version table in this guide or a site runbook. -2. Make every referenced host artifact available. For DPU firmware, update the +1. Make every referenced host artifact available. For DPU firmware, update the BFB, accepted NIC versions, and `dpu_models` together. -3. Check host component ordering, defaults, pre-ingestion thresholds, and +1. Check host component ordering, defaults, pre-ingestion thresholds, and inventory matchers. Then inspect the merged host catalog with `firmware show`. -4. Configure a nonzero shared capacity, explicit-start windows, and assigned-host +1. Configure a nonzero shared capacity, explicit-start windows, and assigned-host approval policy. Keep site-wide automatic selection disabled while validating one host through a per-machine host policy or an explicit DPU reprovisioning request. -5. After verification, enable the intended site-wide automatic policy. -6. Monitor the applicable workflow until inventory reports the configured +1. After verification, enable the intended site-wide automatic policy. +1. Monitor the applicable workflow until inventory reports the configured versions and the host has returned to service. Changing a pre-ingestion threshold while `firmware_global.autoupdate` is enabled @@ -379,5 +379,5 @@ catalog and artifact availability before applying that change. Rack and component firmware do not use either of the configuration models above. Their target version comes from the update request or from the default -selected by the rack or component backend. See -[Rack and component firmware upgrades](rack-component-firmware.md). +selected by the rack or component backend. Refer to +[Rack and Tray Firmware Updates](rack-component-firmware.md). diff --git a/docs/operations/firmware-updates/dpu-firmware.md b/docs/operations/firmware-updates/dpu-firmware.md index fbe4ce88cb..345d895515 100644 --- a/docs/operations/firmware-updates/dpu-firmware.md +++ b/docs/operations/firmware-updates/dpu-firmware.md @@ -12,8 +12,8 @@ Keep two kinds of configuration separate: - `dpu_models` contains the BMC, ERoT/CEC, and NIC versions that NICo expects after the BFB has been applied. -These values and the BFB must describe the same release. See -[Configure firmware versions](configuration.md#dpu-firmware) before enabling +These values and the BFB must describe the same release. Refer to +[DPU firmware](configuration.md#dpu-firmware) before enabling automatic updates. ## Components covered @@ -144,7 +144,7 @@ contributes to the unavailable count until the host returns to top-level `Ready`. An operator can -[request DPU reprovisioning explicitly](api:PATCH/v2/org/{org}/nico/machine/{machineId}/dpu/reprovision). +[request DPU reprovisioning explicitly](api:PATCH/v2/org/:org/nico/machine/:machineId/dpu/reprovision). The API requires a `HostUpdateInProgress` health alert with `PreventAllocations`; the CLI's `--update-message` option creates that alert before sending the request: @@ -227,8 +227,8 @@ nico-admin-cli -a dpu reprovision clear \ --id ``` -For DPU OS installation, networking, and health details beyond firmware, see -[DPU lifecycle management](../../dpu-management/dpu-lifecycle-management.md). +For DPU OS installation, networking, and health details beyond firmware, refer to +[DPU Lifecycle Management](../../dpu-management/dpu-lifecycle-management.md). ## Current limitations diff --git a/docs/operations/firmware-updates/host-firmware.md b/docs/operations/firmware-updates/host-firmware.md index 58cfba3421..ca6da9d9d8 100644 --- a/docs/operations/firmware-updates/host-firmware.md +++ b/docs/operations/firmware-updates/host-firmware.md @@ -1,6 +1,6 @@ # Managed Host Firmware Updates -Managed host firmware upgrades keep an ingested host aligned with the default +Managed host firmware updates keep an ingested host aligned with the default versions in the [effective host firmware catalog](configuration.md#host-firmware-catalog). They are normally triggered by drift, scheduled by Machine Update Manager, and @@ -187,7 +187,7 @@ The host reprovisioning command always enters firmware checking; its legacy `--update-firmware` flag is not needed. A request that has not started can be removed with `host reprovision clear --id `. -## Upgrade sequence +## Update workflow After the request can proceed, the managed-host state machine performs the following sequence: @@ -195,20 +195,20 @@ following sequence: 1. **Protect the host.** NICo records the update health report and, where needed, moves an assigned host into its discovery environment and disables platform lockdown. -2. **Refresh inventory.** Site Explorer revisits the BMC. The state machine +1. **Refresh inventory.** Site Explorer revisits the BMC. The state machine selects the effective catalog definition by BMC vendor and host model. -3. **Select a component.** NICo walks `ordering` and finds the first component +1. **Select a component.** NICo walks `ordering` and finds the first component for which any matching inventory entry differs from the default version. Pre-ingestion-only firmware entries are excluded. -4. **Install it.** NICo chooses the installation route described below. +1. **Install it.** NICo chooses the installation route described below. Multi-artifact entries are applied in sequence. -5. **Activate and verify.** For Scout and Redfish updates, NICo performs +1. **Activate and verify.** For Scout and Redfish updates, NICo performs component-specific resets, reboots, or configured power drains. A legacy local script owns its own activation procedure. NICo then requests fresh inventory and checks the reported versions. -6. **Repeat.** NICo returns to the beginning of `ordering` and selects the next +1. **Repeat.** NICo returns to the beginning of `ordering` and selects the next mismatched component. -7. **Restore service.** When no mismatch remains, NICo restores lockdown when +1. **Restore service.** When no mismatch remains, NICo restores lockdown when required, returns the host to `Ready` or its assigned-instance state, clears the reprovisioning request, and removes the update health report. @@ -238,12 +238,12 @@ The route precedence is: and its `metadata.toml`. If present, NICo sends Scout the verified script, artifact URLs and digests, and execution timeouts. Scout downloads and verifies the files, runs the script on the host, and reports the result. -2. **Core-local script.** If no Scout script exists and the firmware entry has +1. **Core-local script.** If no Scout script exists and the firmware entry has `script`, NICo runs that legacy script on the Core service with the BMC address and credentials in its environment. The script is responsible for installation and activation; after it succeeds, NICo returns to inventory checking. -3. **Redfish.** Otherwise NICo resolves the configured artifact, waits for a +1. **Redfish.** Otherwise NICo resolves the configured artifact, waits for a shared `firmware_global.max_uploads` slot, uploads asynchronously, and polls the returned Redfish task. @@ -261,7 +261,7 @@ preservation settings, activation requirements, and recovery. Before adding an artifact or script to the catalog, verify that it implements the procedure for the exact platform and release. A successful Redfish task or script exit only shows that the installation mechanism completed; NICo still -needs fresh inventory to confirm the configured version. See the +needs fresh inventory to confirm the configured version. Refer to the [Redfish workflow](../../architecture/redfish_workflow.md) for the request and task ownership boundaries. @@ -319,7 +319,7 @@ The principal host firmware substates are: | `NewFirmwareReportedWait` | NICo is refreshing inventory until all matching entries report the target. | | `FailedFirmwareUpgrade` | The attempt failed; NICo is waiting to retry or has exhausted its retry budget. | -See the [managed-host state machine](../../architecture/state_machines/managedhost.md) +Refer to [Managed Host State Diagrams](../../architecture/state_machines/managedhost.md) for these firmware substates in the complete host lifecycle. Completion requires more than leaving an installation state. Verify that the @@ -338,7 +338,7 @@ Useful metrics are: | `carbide_machines_in_maintenance_count` | Machines consuming the shared maintenance budget. | | `carbide_concurrent_machine_updates_available` | The calculated site-wide concurrency ceiling for machine updates. | -See [Core metrics](../../observability/core_metrics.md) for the common metric +Refer to [Core Metrics](../../observability/core_metrics.md) for the common metric labels and collection endpoint. ## Failures and recovery diff --git a/docs/operations/firmware-updates/pre-ingestion.md b/docs/operations/firmware-updates/pre-ingestion.md index 5298c69553..19709dd225 100644 --- a/docs/operations/firmware-updates/pre-ingestion.md +++ b/docs/operations/firmware-updates/pre-ingestion.md @@ -1,6 +1,6 @@ # Pre-ingestion Firmware Updates -Pre-ingestion firmware upgrades run against a discovered host BMC before NICo +Pre-ingestion firmware updates run against a discovered host BMC before NICo creates the managed host. Their purpose is to update firmware that is too old for the normal ingestion and management workflow. @@ -21,12 +21,12 @@ are true: 1. Site exploration has produced a BMC vendor, host model, and firmware inventory for an endpoint that has not completed pre-ingestion. -2. The effective host firmware catalog contains a definition for that vendor +1. The effective host firmware catalog contains a definition for that vendor and model. -3. At least one reported component has +1. At least one reported component has `preingest_upgrade_when_below` configured and its reported version is lower than that threshold. -4. `firmware_global.autoupdate` is enabled. +1. `firmware_global.autoupdate` is enabled. The reported version must come from an inventory entry matched by `current_version_reported_as`. A missing catalog definition, missing inventory @@ -66,7 +66,7 @@ For example: meet their minimums. After ingestion, both can still be treated as drifted from their steady-state targets. -## Upgrade sequence +## Update sequence ```mermaid flowchart TD @@ -91,22 +91,22 @@ The state machine performs the following work: 1. **Prepare the BMC.** NICo attempts an initial BMC reset, waits for a fresh exploration report, configures the site's NTP servers, and checks BMC time. Firmware comparison uses the refreshed inventory. -2. **Evaluate minimums.** NICo selects the effective catalog definition by BMC +1. **Evaluate minimums.** NICo selects the effective catalog definition by BMC vendor and host model, then compares matched inventory versions with each configured pre-ingestion threshold. -3. **Select one component.** If a minimum is breached, NICo follows `ordering` +1. **Select one component.** If a minimum is breached, NICo follows `ordering` and selects the first component whose reported version differs from its pre-ingestion target. Components are handled serially for each endpoint. -4. **Install the firmware.** A firmware entry with `script` runs the configured +1. **Install the firmware.** A firmware entry with `script` runs the configured local upgrade script. Otherwise NICo resolves the artifact, waits for a shared upload slot, and uploads it through Redfish. Multi-artifact entries are installed in sequence. -5. **Activate and verify.** NICo waits for the Redfish task, performs the +1. **Activate and verify.** NICo waits for the Redfish task, performs the component-specific reset, reboot, or configured power drains, and requests a fresh exploration report. When the matching inventory entry is present, NICo waits until its reported version matches the target. A missing entry cannot be verified or selected for another update. -6. **Continue ingestion.** NICo repeats target selection for the remaining +1. **Continue ingestion.** NICo repeats target selection for the remaining ordered components. When none need an update, it marks pre-ingestion complete and the normal ingestion process can continue. diff --git a/docs/operations/firmware-updates/rack-component-firmware.md b/docs/operations/firmware-updates/rack-component-firmware.md index 1ee7056948..6c10b3d511 100644 --- a/docs/operations/firmware-updates/rack-component-firmware.md +++ b/docs/operations/firmware-updates/rack-component-firmware.md @@ -8,7 +8,7 @@ work. Use this path when an operator needs Flow to update a rack or selected tray. For desired-state updates on a conventional managed host, use -[Host firmware upgrades](host-firmware.md) instead. +[Host firmware updates](host-firmware.md) instead. ## How the request runs @@ -172,8 +172,8 @@ that has no matching components is reported as skipped. The built-in rule deliberately excludes power shelves and does not perform an AC power cycle after flashing. Use an approved custom operation rule for power shelves. If firmware activation requires a power cycle, submit the appropriate -power-recycle task separately or include it in a custom rule. See the -[Flow operation rules guide](../../../rest-api/flow/docs/operation-rules-guide.md). +power-recycle task separately or include it in a custom rule. Refer to the +Flow [Operation Rules Guide](../../../rest-api/flow/docs/operation-rules-guide.md). ## Component behavior @@ -190,8 +190,8 @@ selected firmware bundle. It does not include DPU reprovisioning. `dpu` is a special, explicit-only compute target. NICo first submits any compute-tray update, then reprovisions the DPU on each selected host serially. The request's `version` is ignored for the DPU branch; its target comes from -site configuration. Use `targets: ["dpu"]` for a DPU-only request. See -[DPU firmware upgrades](dpu-firmware.md#assigned-hosts-and-operator-requests) +site configuration. Use `targets: ["dpu"]` for a DPU-only request. Refer to +[Assigned hosts and operator requests](dpu-firmware.md#assigned-hosts-and-operator-requests) before using this path. Core routes conventional standalone machines through the managed-host firmware @@ -309,7 +309,7 @@ nico-admin-cli component-manager update-firmware power-shelf \ ``` Run `nico-admin-cli component-manager update-firmware --help` for the complete -command options. See the +command options. Refer to the [rack state machine](../../architecture/state_machines/rackstatemachine.md) for lower-level execution details.