Agustina data transfer

The Agustina supercomputer is a high-performance computing (HPC) system used for scientific and technical computing tasks. Efficient data transfer to and from Agustina is essential for preparing jobs, analyzing results, and managing large datasets.
This document provides step-by-step instructions for securely transferring files between your local machine and the Agustina supercomputer using several methods, including SFTP, Filezilla, SSHFS, SCP, and Rclone. Each method is described with practical examples to help you choose the best approach for your workflow.

Prerequisites

Before you can transfer data, you will need the following:

  1. Username and Password: You will need a valid username and password to access the Agustina supercomputer.
  2. Secure Shell (SSH) Client: You will need an SSH client, such as PuTTY (Windows) or Terminal (macOS/Linux), to connect to the supercomputer.
  3. Filezilla: You will need Filezilla to copy files on graphical UI.

Transfer Methods

SFTP (Secure File Transfer Protocol)

SFTP is a secure way to transfer files to and from the Agustina supercomputer. Here's how to use it:

  1. Open your SSH client and connect to the Agustina supercomputer using your username and password.
  2. Once connected, you can use the SFTP client to transfer files. For example, in PuTTY, you can type sftp username@agustina.bifi.unizar.es and press Enter.
  3. Navigate to the desired directories on both your local machine and the Agustina supercomputer using the cd command.
  4. Use the put command to upload files from your local machine to the Agustina supercomputer, and the get command to download files from the Agustina supercomputer to your local machine.

To upload a file from your local machine to Agustina:

sftp username@agustina.bifi.unizar.es
# After entering your password and connecting:
cd /path/to/remote/directory
put /path/to/local/file

To download a file from Agustina to your local machine:

sftp username@agustina.bifi.unizar.es
# After entering your password and connecting:
cd /path/to/remote/directory
get remote_file.txt /path/to/local/directory/

Filezilla (Graphical SFTP Client)

Filezilla is a free, cross-platform graphical SFTP client that makes it easy to transfer files between your local machine and the Agustina supercomputer.

  1. Download and Install Filezilla
    Download Filezilla from https://filezilla-project.org/ and install it on your computer.

  2. Open Filezilla and Set Up a New Connection

    • Open Filezilla.
    • At the top, enter the following details:
      • Host: sftp://agustina.bifi.unizar.es
      • Username: your Agustina username
      • Password: your Agustina password
      • Port: 22
    • Click "Quickconnect".

  3. Transfer Files

    • The left pane shows your local files; the right pane shows files on Agustina.
    • Drag and drop files between panes to upload or download.

  4. Security Note

    • The first time you connect, you may be prompted to trust the server's host key. Accept to proceed.

Filezilla provides a user-friendly way to manage file transfers without using the command line.

SSHFS (Secure Shell Filesystem)

SSHFS allows you to mount the Agustina supercomputer's file system on your local machine, making it easier to manage files. Here's how to use it:

  1. Install the SSHFS client on your local machine. On Windows, you can use WinFsp; on macOS/Linux, you can use the built-in SSHFS client.
  2. Open your SSH client and connect to the Agustina supercomputer using your username and password.
  3. Once connected, use the sshfs command to mount the Agustina supercomputer's file system on your local machine. For example, on macOS/Linux, you can type sshfs username@agustina.bifi.unizar.es:/path/to/remote/directory /path/to/local/mount/point.
  4. You can now manage files on the Agustina supercomputer as if they were on your local machine.

Example: Using SSHFS to Mount Agustina's Filesystem

On macOS or Linux, you can mount a remote directory from Agustina to a local mount point using SSHFS.
First, ensure SSHFS is installed. On macOS, you can use Homebrew:

brew install sshfs fuse3

On Linux, install via your package manager:

sudo apt-get install sshfs fuse3

Then, mount the remote directory:

sshfs -o allow_other,default_permissions username@agustina.bifi.unizar.es:/path/to/remote/directory /path/to/local/mount/point
  • Replace username with your Agustina username.
  • Replace /path/to/remote/directory with the directory you want to access on Agustina.
  • Replace /path/to/local/mount/point with an existing directory on your local machine.

Remember to set the full path to the private ssh file. If you get an error like this:

fusermount3: option allow_other only allowed if 'user_allow_other' is set in /etc/fuse.conf

Uncomment the option user_allow_other in your /etc/fuse.conf.

To unmount when finished:

fusermount -u /path/to/local/mount/point

SCP (Secure Copy)

SCP is a command-line tool for securely copying files between your local machine and the Agustina supercomputer. Here's how to use it:

  1. Open your SSH client and connect to the Agustina supercomputer using your username and password.
  2. To upload a file from your local machine to the Agustina supercomputer, use the scp command. For example, scp /path/to/local/file username@agustina.bifi.unizar.es:/path/to/remote/directory.
  3. To download a file from the Agustina supercomputer to your local machine, use the scp command in the opposite direction. For example, scp username@agustina.bifi.unizar.es:/path/to/remote/file /path/to/local/directory.

To upload a file from your local machine to Agustina:

scp /path/to/local/file username@agustina.bifi.unizar.es:/path/to/remote/directory

To download a file from Agustina to your local machine:

scp username@agustina.bifi.unizar.es:/path/to/remote/file /path/to/local/directory

Rclone

Rclone is a command-line tool that can be used to transfer files between your local machine and cloud storage services, including the Agustina supercomputer. Here's how to use it:

  1. Install Rclone on your local machine with sudo apt-get install rclone.
  2. Configure Rclone to connect to the Agustina supercomputer by running rclone config and following the prompts.
  3. Use the rclone copy command to upload files from your local machine to the Agustina supercomputer, and the rclone sync command to download files from the Agustina supercomputer to your local machine. For example, rclone copy /path/to/local/file agustina:/path/to/remote/directory.

Example: Using Rclone with Password Obscuration

  1. Obscure your plain text password
rclone obscure <your-plain-text-password>

The output string will set on the rclone.conf file on step 2.

  1. Create or Edit your rclone.conf file

Create a file called rclone.conf (usually located at ~/.config/rclone/rclone.conf) with the following content.
Replace username with your Agustina username and use rclone config password to generate the obscured password string.

[agustina]
type = sftp
host = agustina.bifi.unizar.es
user = username
pass = <obscured-password>
port = 22
  1. Copy files from Agustina to your local machine

To copy a remote file or directory from Agustina to your local machine, use:

rclone --config=/full/path/to/rclone.conf copy agustina:/path/to/remote/file /path/to/local/directory

To copy an entire directory:

rclone --config=/full/path/to/rclone.conf  copy agustina:/path/to/remote/directory /path/to/local/directory --progress

Agustina also has a rclone module installed to copy files and folders from remote to local, to load the module the commmand is:

module load rclone/1.69.3

For more commands to use in rclone visit: https://rclone.org/commands/.

Remember to replace the placeholders (e.g., username, agustina.bifi.unizar.es, /path/to/remote/directory) with the appropriate values.

WinSCP (Windows)

WinSCP is a free, open-source SFTP and SCP client for Windows that provides a graphical interface for file transfers. Since you need to connect through the Bridge node, WinSCP supports SSH tunneling (jump host) connections.

Installation

  1. Download WinSCP from https://winscp.net/
  2. Install it on your Windows machine
  3. Launch WinSCP

Step-by-step configuration:

  1. Open WinSCP and click "New Session"

  2. Configure the Direct Connection to Agustina:

    • Host name: agustina.bifi.unizar.es
    • Port: 22
    • Username: your Agustina username
    • Password: your Agustina password

  3. Set Up the SSH Tunnel (Jump Host):

    • In the left panel, click "Advanced"
    • Go to "Connection" -> "Tunnel"
    • Check the box: "Connect through SSH tunnel"
    • Tunnel host: bridge.bifi.unizar.es
    • Tunnel port: 22
    • Tunnel username: your Bridge username
    • Tunnel password: your Bridge password
    • Click "OK"

  4. Save the Session (Optional):

    • Enter a session name (e.g., "Agustina via Bridge")
    • Click "Save"
    • The session will be saved for future use

  5. Connect:

    • Click "Login" to establish the connection
    • WinSCP will first connect to Bridge, then tunnel to Agustina
    • You'll see the remote file system in the right pane

File Transfer with WinSCP

Once connected:

  • Upload files: Drag and drop from the left pane (local) to the right pane (remote), or use the upload button
  • Download files: Drag and drop from the right pane (remote) to the left pane (local), or use the download button
  • Navigate directories: Double-click folders to enter them
  • Create folders: Right-click in the remote pane and select "New Folder"
  • Delete files: Right-click and select "Delete"

Rsync through Bridge

Rsync is an efficient tool for transferring files between your local machine and Agustina through the Bridge jump host. It supports resuming interrupted transfers and shows progress.

Download a file from Agustina to your local machine:

rsync -avzP -e "ssh -J username@bridge.bifi.unizar.es" username@agustina.bifi.unizar.es:/fs/agustina/username/filename /path/to/local/directory/

Upload a file from your local machine to Agustina:

rsync -avzP -e "ssh -J username@bridge.bifi.unizar.es" /path/to/local/file username@agustina.bifi.unizar.es:/fs/agustina/username/

Copy an entire directory:

rsync -avzP -e "ssh -J username@bridge.bifi.unizar.es" username@agustina.bifi.unizar.es:/fs/agustina/username/directory /path/to/local/destination/

Options explained:

  • -a: Archive mode (preserves permissions, timestamps, etc.)
  • -v: Verbose (shows progress)
  • -z: Compress during transfer
  • -e: Specify SSH command with jump host via -J
  • -P or --progress: Show detailed progress for each file