Update installation tutorial

content/en/docs/getting-started/_index.md

@@ -16,7 +16,7 @@ instructions below.
 
 To build `cc-backend` you need:
 
-- A [go compiler](https://go.dev/dl/), version 1.24 or newer. Most recent os
+- A [go compiler](https://go.dev/dl/), version 1.25 or newer. Most recent os
   environments should have a package with a recent enough version. On MacOS we
   recommend to use [Homebrew](https://brew.sh) to install on.
 - A node.js environment including the `npm` package manager.

content/en/docs/tutorials/prod-arch.md

@@ -17,7 +17,8 @@ need to be made:
    cc-backend vs external standalone)
 
 This guide helps you understand the trade-offs to make informed decisions based
-on your cluster size, administrative capabilities, and requirements.
+on your cluster size, administrative capabilities, and requirements. You may
+also use a mix of above options.
 
 ## Transport: REST API vs NATS
 
@@ -75,7 +76,7 @@ subscribes to receive them.
 **Advantages:**
 
 - Decoupled architecture: collectors and metric store are independent
-- Built-in buffering and message persistence (with JetStream)
+- Built-in buffering and message persistence
 - Better scalability for large clusters (1000+ nodes)
 - Supports multiple subscribers (e.g., external metric store for redundancy)
 - Collectors continue working even if metric store is temporarily down
@@ -142,8 +143,8 @@ configuration and lifecycle.
 **Disadvantages:**
 
 - Metric store restart requires cc-backend restart and the other way around
-- A cc-backend restart can take very long since the metric store checkpoints
-  have to loaded on startup
+- A cc-backend restart can take more then a minute since the metric store checkpoints
+  have to be loaded on startup
 - Resource contention between web serving and metric ingestion
 - No horizontal scaling of metric ingestion
 - Single point of failure for entire system
@@ -176,7 +177,7 @@ its REST API.
 
 **Disadvantages:**
 
-- Two processes to deploy and manage
+- Two services to deploy and manage
 - Separate configuration files
 - Additional network communication between components
 - More complex setup and monitoring
@@ -201,9 +202,9 @@ its REST API.
 | NATS Server  | N/A                            | Exposed to all collectors and metric stores |
 | cc-backend   | Exposed to users               | Exposed to users                            |
 
-With NATS, the metric store can be isolated from the compute network, reducing
+With NATS, the metric store can be isolated from the cluster network, reducing
 attack surface. The NATS server becomes the single point of ingress for metrics.
-Another option to isolate the web backend from the compute network is to setup
+Another option to isolate the web backend from the cluster network is to setup
 [cc-metric-collector proxies]({{< ref hierarchical-collection >}}).
 
 ### Authentication
@@ -270,6 +271,8 @@ Checkpoints are written periodically. For large deployments:
 - Use fast storage (SSD) for checkpoint directory
 - Consider separate disks for checkpoints and archives
 - Monitor disk space for archive growth
+- ClusterCockpit supports to store archives on an external S3 compatible object
+  store
 
 ## Example Configurations
 

content/en/docs/tutorials/prod-ccms.md

@@ -29,7 +29,7 @@ Minimal example of a configuration file:
 ```json
 {
   "main": {
-    "addr": "localhost:8080",
+    "addr": "0.0.0.0:8082",
     "jwt-public-key": "kzfYrYy+TzpanWZHJ5qSdMj5uKUWgq74BWhQG6copP0="
   },
   "metrics": {
@@ -62,6 +62,7 @@ Minimal example of a configuration file:
     "retention-in-memory": "48h",
     "memory-cap": 100,
     "checkpoints": {
+      "file-format": "wal",
       "directory": "./var/checkpoints"
     },
     "cleanup": {
@@ -78,9 +79,10 @@ The `main` section specifies the address and port on which the server should
 listen (`addr`). Optionally, for HTTPS, paths to TLS cert and key files can be
 specified via `https-cert-file` and `https-key-file`. If using a privileged
 port (e.g., 443), you can specify `user` and `group` to drop root permissions
-after binding. The `backend-url` option allows connecting to cc-backend for
-querying job information. The REST API uses JWT token based authentication.
-The option `jwt-public-key` provides the Ed25519 public key to verify signed
+after binding. The `backend-url` option allows connecting to `cc-backend` for
+querying job information. This is required to enable the dynamic memory
+retention in `cc-metric-store`. The REST API uses JWT token based authentication.
+The option `jwt-public-key` provides the ED25519 public key to verify signed
 JWT tokens.
 
 ### Metrics Section
@@ -100,19 +102,67 @@ Metrics that are only available at node scope should set aggregation to `null`.
 
 The most important configuration option is the `retention-in-memory` setting. It
 specifies for which duration back in time metrics should be provided. This
-should be long enough to cover common job durations plus a safety margin. The
-`memory-cap` option sets the maximum memory capacity in GB to use.
-The memory footprint scales with the number of nodes, the number of native
-metric scopes (cores, sockets), the number of metrics, and the memory retention
-time divided by the frequency.
+should be long enough to cover common job durations. The `memory-cap` option
+sets the approximate maximum memory capacity in GB to use. The memory footprint
+scales with the number of nodes, the number of native metric scopes (cores,
+sockets), the number of metrics, and the memory retention time divided by the
+frequency.
+
+The `num-workers` option controls the number of parallel workers used for
+checkpoint and archive I/O. Setting it to `0` (the default) enables automatic
+setup, capped at 10.
 
 The `cc-metric-store` supports checkpoints and cleanup/archiving. Checkpoints
 are always performed on shutdown. To not lose data on a crash or other failure,
-checkpoints are written regularly in fixed intervals. Checkpoints that are not
-needed anymore can either be archived (moved and compressed) or deleted,
-controlled by the `cleanup.mode` setting (`archive` or `delete`). The cleanup
-happens at the interval specified in `cleanup.interval`. You may want to set up
-a cron job to delete older archive files.
+checkpoints are written at regular intervals configured via
+`checkpoints.interval` (e.g. `"12h"`). The `checkpoints.file-format` option
+selects the persistence format: `"json"` (human-readable) or `"wal"` (binary
+Write-Ahead Log, crash-safe, default). See [Checkpoint
+Formats](#checkpoint-formats) below. Checkpoints that are not needed anymore can
+either be archived or deleted, controlled by the `cleanup.mode` setting
+(`archive` or `delete`). The cleanup runs at the interval specified in
+`cleanup.interval`.
+
+## Checkpoint Formats
+
+The `checkpoints.file-format` field controls how in-memory data is persisted to
+disk.
+
+**`"json"`** — Human-readable JSON snapshots written
+periodically. Each snapshot is stored as
+`<dir>/<cluster>/<host>/<timestamp>.json`. Easy to inspect and recover manually,
+but larger on disk and slower to write.
+
+**`"wal"` (recommended)** — Binary Write-Ahead Log format designed for crash
+safety. Two file types are used per host:
+
+- `current.wal` — append-only binary log; every incoming data point is appended
+  immediately. Truncated trailing records from unclean shutdowns are silently
+  skipped on restart.
+- `<timestamp>.bin` — binary snapshot written at each checkpoint interval,
+  containing the complete metric state. Written atomically via a `.tmp` rename.
+
+On startup the most recent `.bin` snapshot is loaded, then remaining WAL entries
+are replayed on top. The WAL is rotated after each successful snapshot. The
+`"wal"` format will become the only supported option in a future release. If you
+are migrating from an older installation using JSON checkpoints, switch to `"wal"`
+after a clean restart.
+
+## Parquet Archive
+
+When `cleanup.mode` is `"archive"`, data that ages out of the in-memory
+retention window is written to [Apache Parquet](https://parquet.apache.org/)
+files before being freed, organized as:
+
+```
+<cleanup.directory>/
+  <cluster>/
+    <timestamp>.parquet
+```
+
+One Parquet file is produced per cluster per cleanup run, consolidating all
+hosts. Files are compressed with Zstandard and sorted by cluster, hostname,
+metric, and timestamp for efficient columnar reads.
 
 ## Authentication
 
@@ -158,4 +208,4 @@ and `nats-subscriptions` in the `metric-store` section:
 The `nats` section configures the NATS server connection with address and
 credentials. The `nats-subscriptions` within `metric-store` define which
 subjects to subscribe to and how to tag incoming metrics with cluster
-information.
+information in case this is not already done.

content/en/docs/tutorials/prod-deploy.md

@@ -21,10 +21,10 @@ It is recommended to install all ClusterCockpit components in a common
 directory, e.g. `/opt/monitoring`, `var/monitoring` or `var/clustercockpit`. In
 the following we use `/opt/monitoring`.
 
-Two Systemd services run on the central monitoring server:
+A Systemd service runs on the central monitoring server:
 
-- clustercockpit : binary cc-backend in `/opt/monitoring/cc-backend`.
-- cc-metric-store : Binary cc-metric-store in `/opt/monitoring/cc-metric-store`.
+- clustercockpit : Binary cc-backend in `/opt/monitoring/cc-backend`.
+- (Optional with external metric-store) cc-metric-store : Binary cc-metric-store in `/opt/monitoring/cc-metric-store`.
 
 ClusterCockpit is deployed as a single binary that embeds all static assets.
 We recommend keeping all `cc-backend` binary versions in a folder `archive` and
@@ -37,19 +37,25 @@ This allows for easy roll-back in case something doesn't work.
 which case `cc-backend` will drop root permissions once the ports are taken.
 You have to take care, that the ownership of the `./var` folder and
 its contents are set accordingly.
+You also can run `cc-backend` behind a reverse proxy. In this case it can be
+started with an unprivileged user and the reverse proxy takes care of TLS
+encryption. This also enables to automatically show a maintenance page in case
+ClusterCockpit is not reachable.
 {{< /alert >}}
 
 ### Workflow to deploy new version
 
-This example assumes the DB and job archive versions did not change.
+This example assumes you are deploying ClusterCockpit for the first time or the
+DB and job archive versions did not change between versions.
 
 - Stop systemd service:
 
 ```sh
 sudo systemctl stop clustercockpit.service
 ```
 
-- Backup the sqlite DB file! This is as simple as to copy it.
+- Backup the sqlite DB file! This is as simple as to copy it. You can also use a
+  continuous replication service as e.g. litestream.
 - Copy new `cc-backend` binary to `/opt/monitoring/cc-backend/archive` (Tip: Use a
   date tag like `YYYYMMDD-cc-backend`). Here is an example:
 
@@ -75,7 +81,7 @@ sudo systemctl start clustercockpit.service
 sudo systemctl status clustercockpit.service
 ```
 
-- Check log for issues:
+- Check log for errors:
 
 ```sh
 sudo journalctl -u clustercockpit.service

content/en/docs/tutorials/prod-intro.md

@@ -18,10 +18,10 @@ ClusterCockpit requires the following components:
   database and ensure the metric data is send or forwarded to `cc-backend`.
 - The **api and web interface backend** `cc-backend`. Only one instance of
   `cc-backend` is required. This will provide the HTTP server at the desired
-  monitoring URL for serving the web interface. It also integrates a in-memory
+  monitoring URL for serving the web interface. It also integrates an in-memory
   metric store.
 - A **SQL database**. The only supported option is to use the builtin sqlite
-  database for ClusterCockpit. You can setup [LiteStream](https://litestream.io/)
+  database for ClusterCockpit. It is recommended to setup [LiteStream](https://litestream.io/)
   as a service which performs a continuous replication of the sqlite database to
   multiple storage backends.
 - (Optional) **Metric store**: One or more `cc-metric-store` instances.
@@ -49,7 +49,7 @@ ClusterCockpit requires the following components:
   - Slurm: Golang based solution
     ([cc-slurm-adapter](https://github.com/ClusterCockpit/cc-slurm-adapter)) maintained
     by NHR@FAU. This is the recommended option in case you use Slurm. All
-    options in `cc-backend` are supported.
+    functionality in `cc-backend` is supported.
   - Slurm: Python based solution
     ([cc-slurm-sync](https://github.com/ClusterCockpit/cc-slurm-sync)) maintained
     by PC2 Paderborn
@@ -72,7 +72,9 @@ and around 1.4TB for the job archive. In case you have very high job counts, we
 recommend to use a retention policy to keep the database and the job archive at
 a manageable size. In case you archive old jobs the database can be easily
 restored using cc-backend.
-We run `cc-backend` as systemd services.
+It is recommended to run `cc-backend` as a systemd service. Example systemd unit
+files are available in the ClusterCockpit component repositories and in the
+[cc-examples](https://github.com/ClusterCockpit/cc-examples) repository.
 
 ## Planning and initial configuration
 
@@ -92,8 +94,8 @@ installation:
 1. (Optional) [Configure and deploy]({{< ref prod-ccms >}}) `cc-metric-store`
 1. [Configure and deploy]({{< ref prod-ccmc >}}) `cc-metric-collector`
 1. [Configure and deploy]({{< ref prod-cc-backend >}}) `cc-backend`
-1. [Configure and deploy](/docs/reference/cc-slurm-adapter/) `cc-slurm-adapter` or
-   another job scheduler adapter of your choice
+1. [Configure and deploy](/docs/reference/cc-slurm-adapter/) `cc-slurm-adapter`
+   or another job scheduler adapter of your choice
 
 You can find complete example production configurations in the
 [cc-examples](https://github.com/ClusterCockpit/cc-examples) repository.
@@ -132,8 +134,9 @@ will not be found.
 
 If you have GPU accelerators `cc-slurm-adapter` should use the PCI-E device
 addresses as ids. The option `gpuPciAddrs` for the nvidia and
-rocm-smi collectors in the collector configuration must be configured.
-To validate and debug problems you can use the `cc-backend` debug endpoint:
+`rocm-smi` collectors in the collector configuration must be configured.
+To validate and debug problems you can use the `cc-backend` or `cc-metric-store`
+debug endpoint:
 
 ```bash
 curl -H "Authorization: Bearer $JWT" -D - "http://localhost:8080/api/debug"