File Transfer via Globus: A Step-by-Step Tutorial

Body

Prelude

In the digital era, the ability to transfer files swiftly and securely stands as a cornerstone of academic collaboration and research management. At our university, we recognize the critical role of efficient data management and sharing in propelling academic and scientific pursuits. That's why we're excited to introduce Globus, a state-of-the-art file transfer system designed to streamline the way our community handles data. This tutorial is tailored to guide you, our esteemed faculty, researchers, and students, through the seamless process of utilizing Globus for your diverse data transfer needs. Whether you're dealing with large datasets, collaborating with external institutions, or managing sensitive information, Globus offers a robust and user-friendly platform that caters to all.

In this tutorial we will discuss:

  • Key Definitions 
  • Globus Account Setup/Log in 
  • Transfer Data Using Globus Connect Personal (GCP) 
  • Joining a Globus Group 
  • Transferring Files Between Your GCPs 
  • Sharing Files With External Users via Guest Collections

Definitions

Endpoint

An endpoint in Globus is a network location (like a server, a cluster, or a personal computer) that has been configured to use Globus services. It is an access point for data transfer, allowing users to move or share data between their endpoint and others.

Data Transfer Node (DTN)

A DTN is a high-performance server optimized for transferring large datasets. It can serve as an Endpoint, facilitating efficient data movement within the Globus network.

Globus Connect Personal (GCP)

GCP is a software application that allows a personal computer to be used as a Globus Endpoint, enabling secure and efficient data transfer to and from the computer using the Globus service. It's designed for use on non-server systems, such as desktops or laptops, for convenient access to the Globus network.

Collection

A collection in Globus is a specific set of data within an endpoint. It's akin to a folder or directory on a storage system that has been shared or made accessible through Globus.​Collections allow users to organize their data and control access to it. Users can define permissions for collections, determining who can view or transfer the data.

Guest Collection

A Guest Collection is a shared data access point configured by an endpoint owner, allowing external users (guests) to access and transfer specific data or directories without needing full access to the endpoint. This feature enables controlled and secure data sharing with collaborators who do not have accounts on the host system.

Globus Account Setup/Log in

To take advantage of Globus services, you need a Globus account.

SDSU Users

If you are affiliated with SDSU, your account is already set up and ready to use. To log in to your account follow the steps below:

  1. Navigate to http://globus.org and click “Log in”
  2. Search for South Dakota State University.Uploaded Image (Thumbnail)
  3. Log in using your MyState credentials.

Non-SDSU Users (i.e., SDSMT, BHSU)

If your organization is not found in the drop down menu, in the bottom of the page click “Globus ID to sign in” and click “Sign Up”

Uploaded Image (Thumbnail)

Fill out the form and create a Globus ID using a personal email.

Transfer Data Using Globus Connect Personal (GCP)

Create a GCP Endpoint on Local Device (PC, or Laptop)

To enable data transfer from your personal device using Globus, you need to add your device to the Globus network as an endpoint. This can be done by installing GCP application on your computer following these steps:

  • Download and install GCP.
  • Note: If you do not have write permission on a device (such as a VDI), you need to specify the directory of your account when installing the application. e.g., C:\Users\mohsen.ahmadkhani\Desktop\Globus_Install\Globus Connect Personal
Uploaded Image (Thumbnail)

  • Log in to your Globus account in the pop-up window.

  • Give your GCP a name and a description and click "Save"
Uploaded Image (Thumbnail)Uploaded Image (Thumbnail)
  • Your GCP Endpoint is now up and running.

Create a GCP Endpoint on Your Innovator Account

To create a GCP endpoint on your Innovator account, follow the steps below: 

  1. Log in to Innovator using your jacks.local account. ​
  2. Load the application using module load globus
  3. Run the app using globusconnectpersonal -setup and log in to your Globus account. Similarly, give your collection a name and a description.  Uploaded Image (Thumbnail)
  4. Run the app using globusconnectpersonal -gui
  5. In the pop-up window click “connect”.

Join a Globus Group

To be able to transfer files between two data collections, you need to be a member of a Group. 

  • Click “Groups” and search for “South Dakota State University Research Computing” – Make sure “MY GROUPS” is unchecked. 
  • Click ”Join Group” and wait for the admin approval.
Uploaded Image (Thumbnail)

Transfer Files Between Your GCPs

Now that you have a GCP collection successfully created on each workstation, you can easily transfer files and folders between the two, following these steps:

  1. Log in to your Globus Webapp at Globus.org
  2. ​ On the left vertical toolbar click file manager and from top-right corner select two-panel layout. 
  3. Click “Search” on each panel and select the collections you created from “Your Collection.”
  4. Transfer data by drag and drop.

Add a Scratch Folder to Your Innovator GCP

If you want to have access to a specific directory that you own in the HPC cluster such as your scratch directory, follow the steps below:

  1. Log in to your Innovator account.
  2. Run the following code:

  3. [john.doe@jacks.local@cllogin002 ~]$ nano ~/.Globusonline/lta/config-paths

  4. Add the following line and save the file:​ /mmfs1/scratch/jacks.local/john.doe/,1,1
  • Please note that you need to replace “john.doe” with your Innovator user name.
  • Note that the 1s after the path are read and write permissions you set to true. You can revoke these permissions by changing them to 0.

Sharing Files With External Users via Guest Collections

  • In the Globus Webapp, choose the directory you intend to share between institutions. For this tutorial, we will create a Guest Collection for the "GlobusTutorial" folder. This folder includes sub-folders and sample csv and txt data sets. Select the folder and click "Share:"

Uploaded Image (Thumbnail)

  • Click "Add Guest Collection"

Uploaded Image (Thumbnail) 

  • Give your collection a name and description appropriately and click “Create Collection.” 

Uploaded Image (Thumbnail)

  • Set permissions. At this stage, you have the option to provide access to an individual user, a group, all Globus users, or anyone on the internet. In this tutorial, we're sharing the collection with a specific user from the University of Minnesota. You can also add a message and assign read and write permissions as needed. After configuring these settings, click “Add Permission” to finalize.

Uploaded Image (Thumbnail)

After completing the process, an email will be sent to the recipient user. This email will include a URL that provides access to the shared data.

You can grant or revoke access rights to other users to the created Guest Collections from "Console" tab of the Webapp.

 

Questions or Problems

You can reach us anytime by filling out this form or by email at SDSU.HPC@sdstate.edu

SDSU RCi

 

Details

Details

Article ID: 159631
Created
Thu 12/21/23 4:51 PM
Modified
Wed 3/27/24 3:16 PM