Merge pull request #1112 from jmpsec/windows-upgrades

Windows osquery upgrades

Author: javuto

ARCHITECTURE.md

@@ -140,9 +140,9 @@ Request controls are composed at route registration:
 - `osctrld` routes in `osctrl-tls` (`rateLimits.osctrldAuth`, `osctrld`, `osctrldDownload`; `cmd/tls/handlers/osctrld_limits.go`): every request is limited per client IP (default 300/min, extension downloads also 30/min), and wrong environment secrets are charged to a per-IP-and-environment budget (default 10/hour). Once that budget is spent the IP gets 429 with `Retry-After` on that environment's `osctrld` routes, even with the right secret: checked before the body is read, and again atomically with the secret comparison (`ratelimit.Limiter.Attempt`), so concurrent requests cannot exceed it. Correct secrets never charge it, so hosts sharing a NAT are unaffected. The client IP is `utils.GetIP`: behind a reverse proxy or load balancer, `service.trustedProxies` must list it, otherwise all clients share the proxy's budgets and one bad client can block the fleet. Past `maxBuckets` tracked keys, new keys share one overflow budget and are blocked together. Every wrong secret is audited (while blocked, one rejection per budget period per IP and environment), capped at 60 audit entries per hour per environment; drops are counted. Metrics: `osctrld_auth_failures_total{environment}`, `osctrld_rate_limited_total{route,limit}`, `osctrld_auth_audit_dropped_total{environment}`, and request duration/status by route pattern. Limits are per process.
 - `cmd/tls/handlers/handlers.go`: `PrometheusMiddleware(...)` records request duration/status for enabled osquery-facing endpoints; handlers also apply endpoint-specific body-size caps.
 
-`osctrl-tls`, under `osctrld.enabled` and when per-node flags overrides are wired, also serves `POST /{env}/osctrld-status` (body <= 4 KiB, secret in the body, through the same rate-limited middleware). osctrld reports whether osqueryd is running, since when it is down, and its last flags and extensions rollback (`stage`, `event_id`); the report is stored in `node_osctrld_status`. A malformed body, a bad UUID or a UUID that is not a node of the environment gets 400 (never 404, which osctrld reads as "route not served"); invalid stages or event ids are dropped for that kind. Each new rollback event id is counted and audited once (compare-and-set, so replicas agree): `osctrld_rollbacks_total{environment,kind,stage}` and a warning audit entry (`osctrld rollback on node <uuid> in env <env>: <kind> <stage>`). A scrape-time collector exports `osctrld_osquery_down{environment}` for every environment (0 when none), counting nodes whose report is fresher than `nodeconfig.ReportFreshness` (15 min) and says osqueryd is down; on a database error or timeout it exports nothing for that scrape and backs off for a minute. Operator semantics: `osctrld_osquery_down` is derived from the database, so every `osctrl-tls` replica exports the same value; aggregate with `max by (environment)`, never `sum`. `osctrld_rollbacks_total` is counted by one replica only (compare-and-set), so `sum` is correct. The report may also carry an `upgrade` object (the result of osctrld's last osquery upgrade: `result` `ok|rejected|skipped`, `version`, `reason`, `request_id`, `event_id`); a valid one overwrites the `upgrade_*` columns of `node_osctrld_status`, an invalid one is dropped without touching the rest of the report. A `rejected` result is a rollback: its `event_id` is deduplicated by the same compare-and-set (on `upgrade_event_id`) and counted once as `osctrld_osquery_upgrade_rollbacks_total{environment}` plus a warning audit entry (`osctrld osquery upgrade rolled back on node <uuid> in env <env>: <version> (<reason>)`); one replica counts each event, so `sum` is correct for it too. A second scrape-time collector, `osctrld_osquery_behind{environment}`, counts nodes whose report is fresh and whose stored `osquery_version` is lower than their effective target (an unexpired pin of the node's environment, else the environment target; no target, an empty or unparseable version, or a `windows` node, which osctrld does not upgrade, is not counted; the report and the node row are joined on UUID and environment, so a stale row left by re-enrolling into another environment is ignored); it is exported for every environment (0 when none), has the same cache, timeout and backoff behavior, is registered only when the osquery version library is enabled, and is aggregated with `max by (environment)` like the down gauge. A third collector, `osctrld_osquery_rollout_paused{environment}`, is 1 for an environment whose latest rollout row is stored as paused (a newer running, aborted or completed row replaces it) and 0 otherwise, for every environment, with the same cache, timeout, backoff and registration conditions; it is database-derived, so aggregate with `max by (environment)` across replicas. The node's stored `osquery_version` comes from the `osquery_version` decoration of its logs or, when a batch carries none (a default environment has an empty schedule and may send undecorated status logs only), from the top-level `version` of its status logs (the most recent valid one; anything that is not a valid version is ignored); until one arrives the stored version is stale, so an upgraded host can still count as behind. Pair an alert on the gauge with an `absent()`/staleness rule if a database outage must not resolve it. A host counts as down only while its latest report is under 15 minutes old. A `rejected` result carrying a 32-hex `request_id`, from a node of the environment, also pauses the environment's osquery rollout when that id is the current running rollout's (compare-and-set in `PauseRollout`, so a pin's or an older target's id, a rollout already paused, aborted or replaced is a no-op), whether or not the rollback event was new; the first host to pause is recorded (display name hostname, else localname, else UUID) and a warning audit entry (`osquery rollout paused in env <env>: <version> rolled back on <host> (<reason>)`) is written once, only when this report paused it. A rollback only pauses if it was first recorded (server time, `node_osctrld_status.upgrade_event_at`, set in the same compare-and-set as the event id; host clocks are never used) after the rollout was last resumed (`osquery_rollouts.resumed_at`), so an osctrld that keeps re-reporting the old rejected result cannot undo a Resume; a genuinely new rollback after the resume pauses again. A failed pause write answers 500; the rollback's event id is already stored, so the next report of the same result repairs it. `skipped` results never pause. The reasons `rollout_pending` and `rollout_paused` are valid for `skipped` (and by the generic rules `rejected`) results: the host is not yet eligible in the open stage, or the rollout is paused and the host has not upgraded. An older `osctrl-tls` answers 405 (not 404) to unknown POST routes; osctrld treats both as "not supported".
+`osctrl-tls`, under `osctrld.enabled` and when per-node flags overrides are wired, also serves `POST /{env}/osctrld-status` (body <= 4 KiB, secret in the body, through the same rate-limited middleware). osctrld reports whether osqueryd is running, since when it is down, and its last flags and extensions rollback (`stage`, `event_id`); the report is stored in `node_osctrld_status`. A malformed body, a bad UUID or a UUID that is not a node of the environment gets 400 (never 404, which osctrld reads as "route not served"); invalid stages or event ids are dropped for that kind. Each new rollback event id is counted and audited once (compare-and-set, so replicas agree): `osctrld_rollbacks_total{environment,kind,stage}` and a warning audit entry (`osctrld rollback on node <uuid> in env <env>: <kind> <stage>`). A scrape-time collector exports `osctrld_osquery_down{environment}` for every environment (0 when none), counting nodes whose report is fresher than `nodeconfig.ReportFreshness` (15 min) and says osqueryd is down; on a database error or timeout it exports nothing for that scrape and backs off for a minute. Operator semantics: `osctrld_osquery_down` is derived from the database, so every `osctrl-tls` replica exports the same value; aggregate with `max by (environment)`, never `sum`. `osctrld_rollbacks_total` is counted by one replica only (compare-and-set), so `sum` is correct. The report may also carry an `upgrade` object (the result of osctrld's last osquery upgrade: `result` `ok|rejected|skipped`, `version`, `reason`, `request_id`, `event_id`); a valid one overwrites the `upgrade_*` columns of `node_osctrld_status`, an invalid one is dropped without touching the rest of the report. A `rejected` result is a rollback: its `event_id` is deduplicated by the same compare-and-set (on `upgrade_event_id`) and counted once as `osctrld_osquery_upgrade_rollbacks_total{environment}` plus a warning audit entry (`osctrld osquery upgrade rolled back on node <uuid> in env <env>: <version> (<reason>)`); one replica counts each event, so `sum` is correct for it too. A second scrape-time collector, `osctrld_osquery_behind{environment}`, counts nodes whose report is fresh and whose stored `osquery_version` is lower than their effective target (an unexpired pin of the node's environment, else the environment target; no target, an empty or unparseable version, or a node on a platform without a package (`freebsd`), is not counted; Windows nodes count; the report and the node row are joined on UUID and environment, so a stale row left by re-enrolling into another environment is ignored); it is exported for every environment (0 when none), has the same cache, timeout and backoff behavior, is registered only when the osquery version library is enabled, and is aggregated with `max by (environment)` like the down gauge. A third collector, `osctrld_osquery_rollout_paused{environment}`, is 1 for an environment whose latest rollout row is stored as paused (a newer running, aborted or completed row replaces it) and 0 otherwise, for every environment, with the same cache, timeout, backoff and registration conditions; it is database-derived, so aggregate with `max by (environment)` across replicas. The node's stored `osquery_version` comes from the `osquery_version` decoration of its logs or, when a batch carries none (a default environment has an empty schedule and may send undecorated status logs only), from the top-level `version` of its status logs (the most recent valid one; anything that is not a valid version is ignored); until one arrives the stored version is stale, so an upgraded host can still count as behind. Pair an alert on the gauge with an `absent()`/staleness rule if a database outage must not resolve it. A host counts as down only while its latest report is under 15 minutes old. A `rejected` result carrying a 32-hex `request_id`, from a node of the environment, also pauses the environment's osquery rollout when that id is the current running rollout's (compare-and-set in `PauseRollout`, so a pin's or an older target's id, a rollout already paused, aborted or replaced is a no-op), whether or not the rollback event was new; the first host to pause is recorded (display name hostname, else localname, else UUID) and a warning audit entry (`osquery rollout paused in env <env>: <version> rolled back on <host> (<reason>)`) is written once, only when this report paused it. A rollback only pauses if it was first recorded (server time, `node_osctrld_status.upgrade_event_at`, set in the same compare-and-set as the event id; host clocks are never used) after the rollout was last resumed (`osquery_rollouts.resumed_at`), so an osctrld that keeps re-reporting the old rejected result cannot undo a Resume; a genuinely new rollback after the resume pauses again. A failed pause write answers 500; the rollback's event id is already stored, so the next report of the same result repairs it. `skipped` results never pause. The reasons `rollout_pending` and `rollout_paused` are valid for `skipped` (and by the generic rules `rejected`) results: the host is not yet eligible in the open stage, or the rollout is paused and the host has not upgraded. The reason `rollback_unavailable` is valid for `rejected` results: osctrld on Windows reports it when an upgrade failed and the previous version could not be restored, so the host needs manual repair; it counts as a rollback (event counted and audited once) and pauses the rollout like any other rejected rollback. An older `osctrl-tls` answers 405 (not 404) to unknown POST routes; osctrld treats both as "not supported".
 
-`osctrl-tls`, under `osctrld.enabled`, also serves `POST /{env}/osctrld-osquery` (body <= 4 KiB, secret in the body, through the same rate-limited `OsctrldMiddleware`) so `osctrld` can ask which osquery version to run. Request: `{secret, uuid, platform, arch, format, installed_version}`; `platform`, `arch` and `format` must match `^[a-z0-9_]{1,16}$` and `installed_version` must be empty or a valid version. Response `200`: `{reason, source, request_id, target, previous, reported_version}`. `reason` is always present and empty means upgrade (`target` and `previous`, each `{version, sha256}`, are then present); otherwise it is `no_target`, `ahead_of_target`, `no_digest_for_target`, `no_digest_for_installed` or `unsupported_platform`. `source` (`environment` or `pin`) and `request_id` are present whenever a target exists and omitted for `no_target` and `unsupported_platform`. `reported_version` is the node's stored `osquery_version`. The route never answers 404: a malformed body, a bad UUID, malformed tokens or a UUID that is not a node of the environment (identical empty body for unknown and other-environment nodes) get 400, a wrong secret 403, a database error 500; a well-formed but unsupported combination (for example `windows`/`msi`) is a 200 with `unsupported_platform`. The decision is `osqueryversions.Manager.Resolve`; the route is registered only when that manager is wired.
+`osctrl-tls`, under `osctrld.enabled`, also serves `POST /{env}/osctrld-osquery` (body <= 4 KiB, secret in the body, through the same rate-limited `OsctrldMiddleware`) so `osctrld` can ask which osquery version to run. Request: `{secret, uuid, platform, arch, format, installed_version}`; `platform`, `arch` and `format` must match `^[a-z0-9_]{1,16}$` and `installed_version` must be empty or a valid version. Response `200`: `{reason, source, request_id, target, previous, reported_version}`. `reason` is always present and empty means upgrade (`target`, `{version, sha256}`, is then present, and so is `previous` except for Windows, which gets the target only and never `no_digest_for_installed`); otherwise it is `no_target`, `ahead_of_target`, `no_digest_for_target`, `no_digest_for_installed` or `unsupported_platform`. `source` (`environment` or `pin`) and `request_id` are present whenever a target exists and omitted for `no_target` and `unsupported_platform`. `reported_version` is the node's stored `osquery_version`. The route never answers 404: a malformed body, a bad UUID, malformed tokens or a UUID that is not a node of the environment (identical empty body for unknown and other-environment nodes) get 400, a wrong secret 403, a database error 500; a well-formed but unsupported combination (for example `freebsd`/`pkg`) is a 200 with `unsupported_platform`. The decision is `osqueryversions.Manager.Resolve`; the route is registered only when that manager is wired.
 
 A new `request_id` tells osctrld to try a target again, including one the host rejected. Setting an environment target always gets a new one. A node pin gets a new one unless only its expiry changed (same environment and version, an `expires_at` at least a second apart), so extending a pin does not retry a rejected version; re-saving the same version with the same expiry is how an operator retries.
 

cmd/api/handlers/osquery_versions.go

@@ -149,7 +149,7 @@ func (h *HandlersApi) OsqueryVersionsListHandler(w http.ResponseWriter, r *http.
 
 // OsqueryVersionPutHandler - PUT Handler to add or replace the digests of a version
 // @Summary Set osquery version digests
-// @Description Adds or replaces the package SHA-256 digests of a version (linux deb/rpm amd64/arm64, darwin pkg universal), each combination at most once. Unknown body fields are refused. Windows is not supported. Changing a digest of a version used by a target or pin is refused with 409. Super admins only.
+// @Description Adds or replaces the package SHA-256 digests of a version (linux deb/rpm amd64/arm64, darwin pkg universal, windows msi amd64), each combination at most once. Unknown body fields are refused. Changing a digest of a version used by a target or pin is refused with 409. Super admins only.
 // @Tags osquery
 // @Accept json
 // @Produce json
@@ -342,6 +342,12 @@ func (h *HandlersApi) serveEnvOsquery(w http.ResponseWriter, envID uint) {
 	for _, c := range counts {
 		resp.Counts = append(resp.Counts, types.OsqueryVersionHostCount{Version: c.Version, Count: c.Count})
 	}
+	win, err := h.Nodes.CountWindowsByEnv(envID)
+	if err != nil {
+		osqueryError(w, err, http.StatusInternalServerError)
+		return
+	}
+	resp.WindowsHosts = int(win)
 	utils.HTTPResponse(w, utils.JSONApplicationUTF8, http.StatusOK, resp)
 }
 
@@ -505,15 +511,15 @@ func (h *HandlersApi) EnvOsqueryRolloutAbortHandler(w http.ResponseWriter, r *ht
 }
 
 // serveNodeOsquery writes the node's pin, effective target, installed version,
-// whether its platform is unsupported (windows), its place in the environment
+// whether its platform is unsupported (no package, e.g. freebsd), its place in the environment
 // rollout (only when the target comes from it) and the last stored upgrade
 // report. A pin or report recorded for another
 // environment is not applied by osctrl-tls and is treated as absent.
 func (h *HandlersApi) serveNodeOsquery(w http.ResponseWriter, env environments.TLSEnvironment, node nodes.OsqueryNode) {
 	uuid := node.UUID
 	resp := types.NodeOsqueryResponse{
 		InstalledVersion:    node.OsqueryVersion,
-		PlatformUnsupported: strings.EqualFold(node.Platform, "windows"),
+		PlatformUnsupported: !osqueryversions.NodePlatformSupported(node.Platform),
 	}
 	var err error
 	if resp.Available, err = h.OsqueryVersions.VersionNames(); err != nil {
@@ -565,7 +571,7 @@ func (h *HandlersApi) serveNodeOsquery(w http.ResponseWriter, env environments.T
 
 // NodeOsqueryGetHandler - GET Handler for a node's osquery upgrade state
 // @Summary Get node osquery upgrade state
-// @Description Returns the node's pin, the version it is rolled out to (the unexpired pin, else the environment target), the osquery version it reports as installed, whether its platform is unsupported by osctrld upgrades (windows), its stage in the environment rollout (rollout, null unless the environment target is a running or paused rollout; eligible_at is when that stage opens and is null when paused or already open), the names (no digests) of the library versions it can be pinned to and the last upgrade report of its osctrld. The installed version and the upgrade result, version, reason and request_id are reported by the host and are untrusted; reported_at is the server's receive time. Environment admins.
+// @Description Returns the node's pin, the version it is rolled out to (the unexpired pin, else the environment target), the osquery version it reports as installed, whether its platform is unsupported by osctrld upgrades (no package, e.g. freebsd), its stage in the environment rollout (rollout, null unless the environment target is a running or paused rollout; eligible_at is when that stage opens and is null when paused or already open), the names (no digests) of the library versions it can be pinned to and the last upgrade report of its osctrld. The installed version and the upgrade result, version, reason and request_id are reported by the host and are untrusted; reported_at is the server's receive time. Environment admins.
 // @Tags nodes
 // @Produce json
 // @Param env path string true "Environment name or UUID"