diff --git a/docs/get-started/best-practices/best-practices-for-harvard.md b/docs/get-started/best-practices/best-practices-for-harvard.md index b853e7bb1..e3afc7069 100644 --- a/docs/get-started/best-practices/best-practices-for-harvard.md +++ b/docs/get-started/best-practices/best-practices-for-harvard.md @@ -55,7 +55,7 @@ Crowdstrike can be downloaded from our repository at: for all devices owned by Harvard staff/faculty and available for all operating systems. -!!! note "Please note" +!!! note "Please Note" To acess this repository you need to be in **Harvard Campus Network**. diff --git a/docs/migration-guides/index.md b/docs/migration-guides/index.md new file mode 100644 index 000000000..66d1b9060 --- /dev/null +++ b/docs/migration-guides/index.md @@ -0,0 +1,15 @@ +# Migration Guides + +This section provides guidance on migrating your workloads and data off the NERC +platform. + +- [Migrate from NERC OpenStack](openstack.md) — Data migration and decommissioning + for NERC OpenStack resources, including VM data transfer, and object storage + export. + +- [Migrate from NERC OpenShift (OCP) & OpenShift AI (RHOAI)](ocp-and-rhoai.md) + — Data migration, configuration export, image backup, and decommissioning for + NERC OpenShift and RHOAI resources, including PVC data transfer, MinIO + object storage export, and workbench notebook backup. + +--- diff --git a/docs/migration-guides/ocp-and-rhoai.md b/docs/migration-guides/ocp-and-rhoai.md new file mode 100644 index 000000000..9fbdd82ab --- /dev/null +++ b/docs/migration-guides/ocp-and-rhoai.md @@ -0,0 +1,143 @@ +# Migrate from NERC OpenShift and OpenShift AI (RHOAI) + +This guide covers the process of migrating your workloads and data off the NERC +OpenShift (OCP) and Red Hat OpenShift AI (RHOAI) platforms. + +## Overview + +Migrating from NERC OpenShift and/or RHOAI involves four main phases: + +1. **Bulk-Export All Project Configurations** — Export all application setups, + configurations, and secrets into a reusable YAML file. + +2. **Data Migration** — Transfer your data from OpenShift persistent storage, + object storage (MinIO), and RHOAI workbenches/notebooks to your target platform. + +3. **Backup Internal Container Images** — Pull images from the internal OpenShift + registry and push them to an external registry. + +4. **Decommissioning** — Delete all resources and archive your ColdFront project. + +## Bulk-Export All Project Configurations + +Export all application setups, configurations, and secrets within a specific +project into a single reusable YAML file: + +```sh +oc get all,pod,deployment,deploymentconfig,pvc,route,service,build,buildconfig, +statefulset,replicaset,replicationcontroller,job,cronjob,imagestream,revision, +configuration,notebook -o yaml > openshift_backup.yaml +``` + +## Data Migration + +Before decommissioning your resources, ensure all critical data has been migrated +to your target environment. NERC provides several methods for transferring data +from OCP and RHOAI. + +### Persistent Storage (PVCs) + +Copy data from PVCs to a pod's filesystem and then transfer it +to your target system. Refer to the +[Use Persistent Storage section](../openshift/storage/data-transfer-from-to-openshift.md#use-persistent-storage) +of the Data Transfer guide for detailed instructions on the following methods: + +- **`oc rsync`** — Recommended for transferring directories between your local + machine and a pod. Efficient for repeat transfers as it only copies changed + files. See [Using `oc rsync`](../openshift/storage/data-transfer-from-to-openshift.md#using-oc-rsync). + +- **`oc cp`** — Convenient for copying individual files or small directories + to/from a container. See + [Copying a Single File](../openshift/storage/data-transfer-from-to-openshift.md#copying-a-single-file). + +- **`tar` with `oc exec`** — Stream a tar archive through a pod for efficient + directory transfers. See + [Using `tar` with `oc exec`](../openshift/storage/data-transfer-from-to-openshift.md#using-tar-with-oc-exec). + +- **Transfer data directly to a PVC** — Run a temporary pod that mounts the PVC + and transfer data using `oc rsync` or `oc cp`. See + [Transferring Data Directly to a PVC](../openshift/storage/data-transfer-from-to-openshift.md#transferring-data-directly-to-a-pvc). + +- **Transfer between two PVCs** — Run a pod that mounts both PVCs and copy data + between them. See + [Transferring Between Two PVCs](../openshift/storage/data-transfer-from-to-openshift.md#transferring-between-two-pvcs). + +For help choosing the right method, see the +[Choosing a Transfer Method](../openshift/storage/data-transfer-from-to-openshift.md#choosing-a-transfer-method) +table. + +### Object Storage (MinIO) + +If you use [MinIO](../openshift/storage/minio.md) for object storage on your +OpenShift project, export your data using the following approaches documented +in the +[For Object Storage Setup on NERC OCP](../openshift/storage/data-transfer-from-to-openshift.md#for-object-storage-setup-on-nerc-ocp) +section: + +- **MinIO Web Console** — Upload and download data through the browser-based + interface. See [Using MinIO](../openshift/storage/data-transfer-from-to-openshift.md#using-minio). + +- **Rclone** — Sync, copy, or mount object storage for transfer to any supported + destination. See [Using Rclone](../openshift/storage/data-transfer-from-to-openshift.md#using-rclone). + +- **Rclone workbench (RHOAI)** — Deploy an Rclone-based workbench through RHOAI + to manage transfers via a web interface. See + [Using RHOAI Rclone Workbench](../openshift/storage/Rclone.md). + +### RHOAI Workbenches, Notebooks, and Cluster Storage + +For users of Red Hat OpenShift AI (RHOAI): + +1. **Notebook data** — Connect to your JupyterLab environment (see + [Explore the JupyterLab Environment](../openshift-ai/data-science-project/explore-the-jupyterlab-environment.md)) + and download any data, models, or notebooks from the workbench's filesystem. + +2. **Data Science Projects** — Review your data science project resources (see + [Using Your Data Science Project (DSP)](../openshift-ai/data-science-project/using-projects-the-rhoai.md)) + and export any stored artifacts, trained models, or pipelines. + +3. **Cluster storage** — All RHOAI cluster storage is backed by PVCs in your + OpenShift project to store your Jupyter notebooks and associated data, ensuring + that your work remains persistent. You can download the data as described in + [Persistent Storage (PVCs)](#persistent-storage-pvcs) above. + +!!! note "Important" + + The PVC backing a workbench's cluster storage includes all files uploaded to + that workbench. When you download the PVC data, you will also get all + notebooks, applications, and data stored on that workbench. + +## Backup Internal Container Images + +If you have images stored in the internal OpenShift registry (ImageStreams), +pull them to your local machine and push them to an external registry: + +```sh +# Log in to the OpenShift registry via docker or podman +podman login -u $(oc whoami) -p $(oc whoami -t) $(oc registry info) + +# Pull the image to your local machine +podman pull $(oc registry info)//: + +# Tag and push to an external registry (e.g., Quay or Docker Hub) +podman tag $(oc registry info)//: quay.io//: +podman push quay.io//: +``` + +## Decommissioning + +Once your data has been safely migrated, follow the +[Decommission OpenShift Resources](../openshift/decommission/decommission-openshift-resources.md) +guide to: + +1. Delete all resources (pods, deployments, PVCs, routes, services, builds, etc.) + +2. Remove users from your ColdFront project (optional) + +3. Reduce all resource quotas to zero via a ColdFront change request + +4. Review project resource quotas + +5. Archive your ColdFront project + +--- diff --git a/docs/migration-guides/openstack.md b/docs/migration-guides/openstack.md new file mode 100644 index 000000000..9c0a20bf5 --- /dev/null +++ b/docs/migration-guides/openstack.md @@ -0,0 +1,120 @@ +# Migrate from NERC OpenStack + +This guide covers migrating your workloads and data off the NERC +OpenStack platform, including data migration, resource cleanup, and project decommissioning. + +## Overview + +Migrating from NERC OpenStack involves two main phases: + +1. **Data Migration** — Transfer your data from NERC OpenStack resources (volumes, +object storage, VM filesystems) to your target platform. + +2. **Decommissioning** — Clean up all OpenStack resources and archive your ColdFront +project. + +## Data Migration + +Before decommissioning your resources, ensure all critical data has been migrated +to your target environment. + +### Data Transfer from VMs + +To transfer data from your virtual machine's filesystem to an external location, +use one of the following methods documented in the +[Data Transfer To/From NERC VM](../openstack/data-transfer/data-transfer-from-to-vm.md) +guide: + +- **Globus** — Preferred method for large datasets; handles retries, error recovery, + and provides status updates. Set up a personal Globus endpoint on your VM. + +- **SCP** — Secure copy for smaller files (<~10 GB) using SSH. + +- **tar+ssh** — Stream a tar archive over SSH for efficient directory transfers. + +- **rsync** — Fast, delta-transfer synchronization; ideal for repeat or incremental + transfers. + +- **Rclone** — Sync, copy, or mount the VM's filesystem via SFTP for transfer to + any supported destination. + +- **WinSCP** (Windows) — GUI-based SFTP/SCP client for drag-and-drop transfers. + +- **Cyberduck** (macOS/Windows) — GUI-based SFTP/FTP client. + +- **Filezilla** (cross-platform) — Open-source SFTP/FTP client. + +!!! note "Data on Detached Volumes" + + If you have a detached volume with data to migrate, attach it to a running VM + as described in the + [Attach a Volume to an Instance guide](../openstack/persistent-storage/attach-the-volume-to-an-instance.md), + then use any of the clients listed above to transfer the data from the VM to + your target location. + +### Object Storage Data + +Data stored in OpenStack Object Storage (Swift) containers can be migrated +using any of the clients documented in the +[Object Storage guide](../openstack/persistent-storage/object-storage.md): + +- **Horizon Dashboard** — Download files through the web interface. + +- **OpenStack CLI** — Use `openstack object store` commands to download objects. + +- **Swift Interface** — Use the `swift` command-line client (`python-swiftclient`) + to download or sync data. + +- **AWS CLI** — Use S3-compatible `aws s3` commands with the NERC endpoint. + +- **s3cmd** — Sync or copy data from containers to your local system or another + S3 target. + +- **Rclone** — Sync, copy, or mount object storage for transfer to any supported + destination. + +- **Python libraries (Boto3, Minio)** — Programmatically download objects using + the S3 API. + +- **GUI tools (WinSCP, Cyberduck)** — Connect via S3 protocol for a file-browser + experience. + +!!! note "Mount Object Storage to a VM" + + You can also mount your object storage container to a running instance as + described in the + [Mount The Object Storage to an Instance guide](../openstack/persistent-storage/mount-the-object-storage.md), + then use any of the clients listed in [Data Transfer from VMs](#data-transfer-from-vms) + to download the data from the VM, just like a mounted volume. + +## Decommissioning + +Once your data has been safely migrated, follow the +[Decommission Your NERC OpenStack Resources](../openstack/decommission/decommission-openstack-resources.md) +guide to: + +1. Delete all VMs + +2. Delete volumes and snapshots + +3. Delete custom images + +4. Delete private networks, routers, and interfaces + +5. Release floating IPs + +6. Clean up security groups + +7. Delete key pairs + +8. Delete buckets and objects + +9. Remove users from your ColdFront project (optional) + +10. Reduce storage quotas to zero + +11. Review project resource quotas + +12. Archive your ColdFront project + +--- diff --git a/docs/openshift-ai/other-projects/deploying-a-llama-model-with-kserve.md b/docs/openshift-ai/other-projects/deploying-a-llama-model-with-kserve.md index 957578c83..0e65ebfb2 100644 --- a/docs/openshift-ai/other-projects/deploying-a-llama-model-with-kserve.md +++ b/docs/openshift-ai/other-projects/deploying-a-llama-model-with-kserve.md @@ -84,7 +84,7 @@ with your operational, security, and performance requirements. - Once successfully initiated, click on the **minio** deployment and select the "Resources" tab to review created *Pods*, *Services*, and *Routes*. - ![MinIO Deployemnt Resources](images/minio-deployment-resources.png) + ![MinIO Deployment Resources](images/minio-deployment-resources.png) The **`minio-s3`** route URL (found under "Routes" -> `minio-s3` -> _Location_ path) is used to interact with the MinIO API **programmatically** and will diff --git a/docs/openshift-ai/other-projects/fraud-detection-predictive-ai-app.md b/docs/openshift-ai/other-projects/fraud-detection-predictive-ai-app.md index bf635f3e3..42cb5b32c 100644 --- a/docs/openshift-ai/other-projects/fraud-detection-predictive-ai-app.md +++ b/docs/openshift-ai/other-projects/fraud-detection-predictive-ai-app.md @@ -482,7 +482,7 @@ are created and grouped under "minio" application grouping on the **Workloads** Once successfully initiated, click on the **minio** deployment and select the "Resources" tab to review created *Pods*, *Services*, and *Routes*. -![MinIO Deployemnt Resources](images/minio-deployment-resources.png) +![MinIO Deployment Resources](images/minio-deployment-resources.png) Please note the **minio-console** route URL by navigating to the "Routes" section under the _Location_ path. When you click on the **minio-console** route URL, it diff --git a/docs/openshift-ai/other-projects/object-detection-app-using-yolo5.md b/docs/openshift-ai/other-projects/object-detection-app-using-yolo5.md index f72f4fa3b..99469c3a3 100644 --- a/docs/openshift-ai/other-projects/object-detection-app-using-yolo5.md +++ b/docs/openshift-ai/other-projects/object-detection-app-using-yolo5.md @@ -453,7 +453,7 @@ are created and grouped under "minio" application grouping on the **Workloads** Once successfully initiated, click on the **minio** deployment and select the "Resources" tab to review created *Pods*, *Services*, and *Routes*. -![MinIO Deployemnt Resources](images/minio-deployment-resources.png) +![MinIO Deployment Resources](images/minio-deployment-resources.png) Please note the **minio-console** route URL by navigating to the "Routes" section under the _Location_ path. When you click on the **minio-console** route URL, it diff --git a/docs/openshift/decommission/decommission-openshift-resources.md b/docs/openshift/decommission/decommission-openshift-resources.md index e7a2bc122..7e509bfa4 100644 --- a/docs/openshift/decommission/decommission-openshift-resources.md +++ b/docs/openshift/decommission/decommission-openshift-resources.md @@ -7,7 +7,7 @@ below. - **Backup**: Back up any critical data or configurations stored on the resources that going to be decommissioned. This ensures that important information is not - lost during the process. + lost during the process as [described here](../storage/data-transfer-from-to-openshift.md). - **Kubernetes Objects (Resources)**: Please review all OpenShift Kubernetes Objects (Resources) to ensure they are not actively used and ready to be decommissioned. diff --git a/docs/openshift/index.md b/docs/openshift/index.md index 5008d646d..03efd5874 100644 --- a/docs/openshift/index.md +++ b/docs/openshift/index.md @@ -61,6 +61,8 @@ the list below. - [Rclone](storage/Rclone.md) +- [Data Transfer](storage/data-transfer-from-to-openshift.md) + ## Deleting Applications - [Deleting your applications](applications/deleting-applications.md) diff --git a/docs/openshift/storage/data-transfer-from-to-openshift.md b/docs/openshift/storage/data-transfer-from-to-openshift.md new file mode 100644 index 000000000..c98942826 --- /dev/null +++ b/docs/openshift/storage/data-transfer-from-to-openshift.md @@ -0,0 +1,753 @@ +# Data Transfer To and From NERC OpenShift + +OpenShift containers are [ephemeral](storage-overview.md#ephemeral-storage). +Files written only to a container's local filesystem can be lost when +the pod is restarted, replaced, or deleted. + +Store important data on a [PersistentVolumeClaim (PVC)](storage-overview.md#persistent-volume-claims-pvcs) +or in [object storage using Minio](minio.md). + +See [Storage Overview](storage-overview.md) for more information about OCP storage. + +OpenShift users normally transfer data through the OpenShift API to a pod. Durable +data should be placed on a PVC or in object storage rather than in the pod's +container filesystem. + +For most transfers between your computer and OpenShift, use `oc rsync`. For large +datasets or transfers that must continue after your computer disconnects, use NERC +object storage, `rclone`, or another managed data-transfer service. + +## **Prerequisites** + +Before proceeding, confirm that you have: + +1. Set up the OpenShift CLI (`oc`) tools locally and configure the OpenShift CLI + to enable `oc` commands. Refer to [this user guide](../logging-in/setup-the-openshift-cli.md). + +2. From a terminal on your laptop/desktop, log in to the NERC OpenShift cluster and + switch to your project namespace: + + ```sh + oc login --token= --server=https://api.shift.nerc.mghpcc.org:6443 + ``` + + For example: + + ```sh + oc login --token= --server=https://api.shift.nerc.mghpcc.org:6443 + Logged into "https://api.shift.nerc.mghpcc.org:6443" as "" using + the token provided. + ``` + +3. Select your project. + + !!! info "Information" + + Some users may have access to multiple projects. Run the following command + to switch to a specific project space: `oc project `. + +4. Confirm you have selected the correct project by running `oc project`, + as shown below: + + ```sh + oc project + Using project "" on server "https://api.shift.nerc.mghpcc.org:6443". + ``` + +5. Identify the pod, container, PVC, source directory, and destination directory. + + ```sh + oc get pods + ``` + + To list the containers in a pod: + + ```sh + oc get pod \ + -o jsonpath='{.spec.containers[*].name}{"\n"}' + ``` + + To see the volumes and mount locations used by a pod: + + ```sh + oc describe pod + ``` + +## Use Persistent Storage + +A container filesystem is not persistent. Unless a file is needed only temporarily, +transfer data into a directory backed by a PVC as [described here](storage-overview.md#persistent-storage). + +For example, an application might mount its PVC at: + +```sh +/data +``` + +Confirm the correct mount path before transferring files: + +```sh +oc exec -- df -h +``` + +You can also examine the pod configuration: + +```sh +oc get pod -o yaml +``` + +OpenShift mounts storage requested through a PVC into one or more containers in +a pod. The PVC remains available independently of a particular pod, subject to +its [storage class](storage-overview.md#storage-class) and +[access-mode](storage-overview.md#access-modes) configuration. + +### Using `oc rsync` + +[Rsync](https://linux.die.net/man/1/rsync) is a fast, versatile, remote (and local) +file-copying tool. It is famous for its delta-transfer algorithm, which reduces +the amount of data sent over the network by sending only the differences between +the source files and the existing files in the destination. This can often lead +to efficiencies in repeat-transfer scenarios, as rsync only copies files that are +different between the source and target locations (and can even transfer partial +files when only part of a file has changed). This can be very useful in reducing +the number of copies you may perform when synchronizing two datasets. + +`oc rsync` is the recommended method for transferring directories between your +computer and a running OpenShift pod and vice-versa. + +The basic syntax is: + +```sh +oc rsync +``` + +A pod path is written as: + +```sh +: +``` + +`oc rsync` transfers directories rather than individual files. + +When native `rsync` is unavailable in the container, the command may fall back to +a tar-based copy, provided that `tar` is available. The fallback does not provide +all normal `rsync` behavior, such as transferring only changed files. + +#### Copy a Local Directory to OpenShift + +```sh +oc rsync ./local-data/ :/data/ +``` + +For example: + +```sh +oc rsync ./research-data/ analysis-workbench-0:/data/research-data/ +``` + +The trailing slash on `./research-data/` means to copy the directory's contents. + +Without the trailing slash: + +```sh +oc rsync ./research-data analysis-workbench-0:/data/ +``` + +the `research-data` directory itself is copied into `/data`. + +#### Copy Data from OpenShift to Your Computer + +```sh +oc rsync :/data/results/ ./results/ +``` + +For example: + +```sh +oc rsync analysis-workbench-0:/data/results/ ./results/ +``` + +#### Select a Container + +A pod can contain more than one container. Specify the container using `-c`: + +```sh +oc rsync ./local-data/ :/data/ \ + -c +``` + +For example: + +```sh +oc rsync ./local-data/ analysis-workbench-0:/data/ \ + -c workbench +``` + +#### Select a Different Project + +Use `-n` when the pod is in a project other than your currently selected project: + +```sh +oc rsync ./local-data/ :/data/ \ + -n +``` + +#### Synchronize Changes + +When both the local computer and container have `rsync` installed, subsequent runs +generally transfer only changed data: + +```sh +oc rsync ./local-data/ :/data/ +``` + +To remove files from the destination that no longer exist in the source: + +```sh +oc rsync ./local-data/ :/data/ --delete +``` + +!!! danger "Be careful with this option!" + + Use `--delete` carefully. Files in the destination directory that are not + present in the source directory can be permanently deleted. + +To continuously synchronize local changes: + +```sh +oc rsync ./local-data/ :/data/ --watch +``` + +### Copying a Single File + +Although `oc rsync` operates on directories, `oc cp` can be convenient for copying +a single file. + +#### Copy a File to OpenShift + +```sh +oc cp ./input.csv /:/data/input.csv +``` + +When using the currently selected project: + +```sh +oc cp ./input.csv :/data/input.csv +``` + +#### Copy a File from OpenShift + +```sh +oc cp /:/data/results.csv ./results.csv +``` + +For a multi-container pod: + +```sh +oc cp ./input.csv :/data/input.csv \ + -c +``` + +!!! note "Important Note" + + `oc cp` requires the `tar` command to be available inside the container. It + can fail with minimal or distroless container images that do not contain `tar`. + +### Using `tar` with `oc exec` + +For directories containing many small files, streaming a compressed tar archive +can be more efficient than transferring each file individually. + +#### Copy a Directory to OpenShift + +```sh +tar czf - ./local-directory \ + | oc exec -i -- \ + tar xzf - -C /data +``` + +For a particular container: + +```sh +tar czf - ./local-directory \ + | oc exec -i -c -- \ + tar xzf - -C /data +``` + +#### Copy a Directory from OpenShift + +```sh +oc exec -- \ + tar czf - -C /data results \ + | tar xzf - -C ./local-destination +``` + +Create the local destination first if necessary: + +```sh +mkdir -p ./local-destination +``` + +Both the local computer and the container must have `tar` available. + +### Transferring Data Directly to a PVC + +Sometimes an application container does not contain `tar`, `rsync`, or other +transfer utilities. In that case, create a temporary transfer pod and mount the +application's PVC into it. + +First identify the PVC: + +```sh +oc get pvc +``` + +Note the name of your PVC, i.e., ``. + +Create a file named `data-transfer-pod.yaml`: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: data-transfer +spec: + restartPolicy: Never + containers: + - name: data-transfer + image: registry.access.redhat.com/ubi9/ubi:latest + command: + - /bin/sh + - -c + - sleep infinity + volumeMounts: + - name: data + mountPath: /data + volumes: + - name: data + persistentVolumeClaim: + claimName: +``` + +Replace `` with the PVC name and create the pod: + +```sh +oc apply -f data-transfer-pod.yaml +oc wait --for=condition=Ready pod/data-transfer --timeout=120s +``` + +Copy the data: + +```sh +oc rsync ./local-data/ data-transfer:/data/ +``` + +Verify the transfer: + +```sh +oc exec data-transfer -- ls -lah /data +``` + +Delete the transfer pod when finished: + +```sh +oc delete pod data-transfer +``` + +Deleting the transfer pod does not delete the PVC. + +!!! note "Important Note" + + A `ReadWriteOnce` PVC might not be mountable by the application pod and transfer + pod simultaneously, depending on the storage system and where the pods are scheduled. + + You may need to stop or scale down the application before starting the transfer + pod. + + For example: + + ```sh + oc scale deployment/ --replicas=0 + ``` + + After the transfer: + + ```sh + oc scale deployment/ --replicas=1 + ``` + + **NOTE:** Stop applications before copying live database files or other data + that must remain internally consistent. + +### Transferring Between Two PVCs + +Create a temporary pod that mounts both PVCs: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: pvc-transfer +spec: + restartPolicy: Never + containers: + - name: transfer + image: registry.access.redhat.com/ubi9/ubi:latest + command: + - /bin/sh + - -c + - sleep infinity + volumeMounts: + - name: source + mountPath: /source + readOnly: true + - name: destination + mountPath: /destination + volumes: + - name: source + persistentVolumeClaim: + claimName: + - name: destination + persistentVolumeClaim: + claimName: +``` + +Create the pod: + +```sh +oc apply -f pvc-transfer-pod.yaml +oc wait --for=condition=Ready pod/pvc-transfer --timeout=120s +``` + +Copy the files: + +```sh +oc exec pvc-transfer -- \ + cp -a /source/. /destination/ +``` + +For directories containing many small files, use `tar`: + +```sh +oc exec pvc-transfer -- \ + sh -c 'tar cf - -C /source . | tar xf - -C /destination' +``` + +Verify the destination: + +```sh +oc exec pvc-transfer -- \ + du -sh /source /destination +``` + +Delete the pod after verification: + +```sh +oc delete pod pvc-transfer +``` + +!!! tip "Important Note" + + This procedure copies the contents. It does not transfer ownership of the source + PVC or delete its data. + + To delete the PVC completely, run: + + ```sh + oc delete pvc + ``` + +## For Object Storage Setup on NERC OCP + +### Using Minio + +If you are using object storage set up with [MinIO](https://min.io/) as +[explained here](minio.md), you can transfer data through it. + +Once successfully initiated, click on the **MinIO** deployment and select the +"Resources" tab to review created *Pods*, *Services*, and *Routes*. + +![MinIO Deployment Resources](images/minio-deployment-resources-all.png) + +The **`minio-s3`** route URL (found under "Routes" -> `minio-s3` -> _Location_ +path) is used to interact with the MinIO API **programmatically** and will +serve as the `S3_ENDPOINT`. Make sure to note this **S3_ENDPOINT**, as it +will be required when configuring `Rclone` later. + +Please note the **minio-console** route URL, also under "Routes" section +under the _Location_ path. When you click on the **minio-console** route URL, it +will open the MinIO web console as shown below: + +![MinIO Web Console](images/minio-web-console.png) + +!!! info "MinIO Web Console Login Credential" + + For this, you need to install and configure the OpenShift CLI by following + the [setup instructions](../logging-in/setup-the-openshift-cli.md#installing-the-openshift-cli). + + Once the OpenShift CLI is set up, the username and password for the MinIO web + console can be retrieved by running the following `oc` commands: + + i. To get *Access key* run: + + `oc get secret minio-root-user -o template --template '{{.data.MINIO_ROOT_USER}}' | base64 --decode` + + ii. And to get *Secret key* run: + + `oc get secret minio-root-user -o template --template '{{.data.MINIO_ROOT_PASSWORD}}' | base64 --decode` + +Return to the **MinIO Web Console** using the provided URL. Enter the **Access Key** +as the **Username** and the **Secret Key** as the **Password**. + +![MinIO Web Console](images/minio-web-console.png) + +This will open the **Object Browser**, from where you will be able to upload/download +data using a web browser. + +### Using Rclone + +[Rclone](https://rclone.org/) is a convenient and performant command-line tool +for transferring files and synchronizing directories directly between your local +file systems and NERC's containers. Rclone is mature, open-source software +originally inspired by **rsync** and written in Go. + +#### Using RHOAI Rclone Workbench + +You can also set up an Rclone-based RHOAI workbench and connect +it to your object storage as [explained here](Rclone.md). + +#### Using Rclone CLI + +Alternatively, you can use the `rclone` command line tool locally. + +**Prerequisites**: + +To run the `rclone` commands, you need to have: + +- `rclone` installed, see + [Downloading and Installing the latest version of the Rclone](https://rclone.org/downloads/) + for more information. + +##### Configuring Rclone + +First, you'll need to configure `rclone`. Since object storage systems +have quite complicated authentication, these credentials are kept in a config file. + +If you run `rclone config file` you will see where the default location is +for you. + +!!! note "Note" + + For **Windows** users, you may need to specify the full path to the `rclone` + executable file if it is not included in your system's PATH variable. + +The **S3_ENDPOINT**, **Access Key**, and **Secret Key** that you previously noted +from MinIO deployed routes can then be plugged into `rclone` config file. + +Edit the config file at the path shown by `rclone config file` and +add the following entry with the remote name `nerc`: + + [nerc] + type = s3 + env_auth = false + provider = Other + endpoint = + acl = public-read + access_key_id = + secret_access_key = + location_constraint = + server_side_encryption = + +See [rclone S3 configuration](https://rclone.org/s3/) for more information. + +!!! note "Important Information" + + If `env_auth` is set to `true`, rclone will read credentials from environment + variables, so you should not insert them directly in the config. + +Or you can copy this content to a new config file and then use the +`--config` flag to override the config location, e.g. `rclone --config=FILE` + +!!! note "Interactive Configuration" + + Run `rclone config` to setup. See [rclone config docs](https://rclone.org/docs/) + for more details. + +##### Using Rclone Subcommands + +`rclone` supports many subcommands (see +[the complete list of Rclone subcommands](https://rclone.org/docs/#subcommands)). +A few commonly used subcommands (assuming you configured the NERC Object Storage +as `nerc`): + +Once your object storage has been configured in rclone, you can list all +buckets with the `lsd` command + + rclone lsd "nerc:" + +Once an object-storage is configured with the name "nerc:", common `rclone` +commands include the following. + +List available buckets or directories: + +```sh +rclone lsd "nerc:" +``` + +Upload a directory: + +```sh +rclone copy /data/results/ :/results/ \ + --progress +``` + +Download a directory: + +```sh +rclone copy :/input-data/ /data/input-data/ \ + --progress +``` + +Preview a synchronization: + +```sh +rclone sync /data/results/ :/results/ \ + --dry-run +``` + +Review the dry-run output, then run: + +```sh +rclone sync /data/results/ :/results/ \ + --progress +``` + +!!! warning "Very Important" + + `rclone sync` can delete destination objects that do not exist in the source. + Use `rclone copy` when you do not want deletion behavior. + +Do not place object-storage credentials directly in a container image or commit +them to Git. Store them in an OpenShift Secret and mount the Secret into the transfer +pod. + +## Verifying a Transfer + +Compare directory sizes: + +```sh +du -sh ./local-data +oc exec -- du -sh /data/local-data +``` + +Compare file counts: + +```sh +find ./local-data -type f | wc -l +``` + +```sh +oc exec -- \ + sh -c 'find /data/local-data -type f | wc -l' +``` + +For important datasets, create checksums locally: + +```sh +find ./local-data -type f -print0 \ + | sort -z \ + | xargs -0 sha256sum > SHA256SUMS +``` + +Transfer the checksum file with the data and verify it in the pod: + +```sh +oc exec -- \ + sh -c 'cd /data/local-data && sha256sum -c SHA256SUMS' +``` + +## Troubleshooting + +### Permission Denied + +Check the destination permissions: + +```sh +oc exec -- \ + ls -ld /data /data/ +``` + +Check the identity used by the container: + +```sh +oc exec -- id +``` + +OpenShift containers commonly run with a dynamically assigned, non-root user ID. +Do not assume that the container can write to every directory. + +Use the application's designated writable directory or PVC mount. Do not change +the container to run as root merely to complete a transfer. + +### Pod Not Found + +Confirm the active project and pod name: + +```sh +oc project +oc get pods +``` + +Or specify the project explicitly: + +```sh +oc get pods -n +``` + +### Multiple Containers + +List the containers: + +```sh +oc get pod \ + -o jsonpath='{.spec.containers[*].name}{"\n"}' +``` + +Then add the following option to the transfer command: + +```sh +-c +``` + +### `rsync` or `tar` Is Missing + +Use a temporary transfer pod with the same PVC mounted, as described in +[Transferring Data Directly to a PVC](#transferring-data-directly-to-a-pvc). + +### Transfer Stops When the Pod Restarts + +Transfer data to a PVC, not to the container's local filesystem. + +If the application pod is frequently replaced, use a dedicated transfer pod. + +### Transfer Is Too Slow or Frequently Interrupted + +Avoid transferring a very large dataset through `oc rsync` or `oc cp`. + +Use object storage, `rclone`, or another managed transfer mechanism that supports +retries and resumable transfers. + +## Choosing a Transfer Method + +| Situation | Recommended method | +| -------------------------------------- | ------------------------------------------------------------------- | +| One small file | `oc cp` | +| Small or medium directory | `oc rsync` | +| Repeated synchronization | `oc rsync` with native `rsync` | +| Many small files | `tar` through `oc exec` | +| Application image lacks transfer tools | Temporary transfer pod | +| Copy between PVCs | Pod mounting both PVCs | +| Large or resumable transfer | Object storage, `rclone` | +| Database backup | Database-native backup tool followed by transfer of the backup file | + +--- diff --git a/docs/openshift/storage/images/minio-deployment-resources-all.png b/docs/openshift/storage/images/minio-deployment-resources-all.png new file mode 100644 index 000000000..f5af0d48c Binary files /dev/null and b/docs/openshift/storage/images/minio-deployment-resources-all.png differ diff --git a/docs/openshift/storage/minio.md b/docs/openshift/storage/minio.md index f972090bd..9574b8794 100644 --- a/docs/openshift/storage/minio.md +++ b/docs/openshift/storage/minio.md @@ -285,7 +285,7 @@ are created and grouped under "minio" application grouping on the **Workloads** Once successfully initiated, click on the **minio** deployment and select the "Resources" tab to review created *Pods*, *Services*, and *Routes*. -![MinIO Deployemnt Resources](images/minio-deployment-resources.png) +![MinIO Deployment Resources](images/minio-deployment-resources.png) Please note the **minio-console** route URL by navigating to the "Routes" section under the _Location_ path. When you click on the **minio-console** route URL, it diff --git a/mkdocs.yaml b/mkdocs.yaml index 742a43761..ae20e3a38 100644 --- a/mkdocs.yaml +++ b/mkdocs.yaml @@ -40,6 +40,9 @@ nav: - Best Practices for My Institution: get-started/best-practices/best-practices-for-my-institution.md - Best Practices for Harvard University: get-started/best-practices/best-practices-for-harvard.md - Best Practices for Boston University: get-started/best-practices/best-practices-for-bu.md + - NERC Migration Guides: + - OpenStack: migration-guides/openstack.md + - OpenShift & OpenShift AI: migration-guides/ocp-and-rhoai.md - OpenStack: - OpenStack: openstack/index.md - Logging In: @@ -120,6 +123,7 @@ nav: - Storage Overview: openshift/storage/storage-overview.md - Minio: openshift/storage/minio.md - Rclone: openshift/storage/Rclone.md + - Data Transfer: openshift/storage/data-transfer-from-to-openshift.md - Deleting Applications: - Deleting your applications: openshift/applications/deleting-applications.md - Decommission OpenShift Resources: openshift/decommission/decommission-openshift-resources.md