Data volume is growing by leaps and bound; more and more data is being generated every day that needs to be computed, analyzed, and stored. While traditional data transfer tools like FTP, scp, sftp and rsync work well for local or small transfers, transferring big data long distances is still slow and prone to interruption.
Fortunately, it’s now easier than ever to get your data onto WestGrid by using Globus. This service is provided by the University of Chicago, and allows quick and easy file transfers to and from any Globus-enabled system. By using the free Globus Connect program, this includes your desktop.
Globus has many advantages over traditional file transfer tools:
- Better usage of bandwidth, resulting in a much faster transfer
- Automatic restart of interrupted transfers
- Enhanced ease of use with a web based user interface
- E-mail transfer completion and error notifications
- Ability to automate transfers, through a scriptable interface
How it Works
The Globus website uses the Globus Toolkit to make file transfers easy. The Globus Toolkit is used by most high performance computing sites to simplify the process of logging into large scientific computing systems. It does this by using cryptographic certificates (x.509, for those interested in the details) to manage user identity and to keep track of who is allowed to connect where.
Globus integrates into the Globus Toolkit cryptographic trust system, allowing you to delegate authority to connect to resources, such as WestGrid, to Globus, which manages the details on your behalf. Globus keeps track of resources using an endpoint, which points to a specific resource. These are named in such a way as to make it clear which resources are connecting to each other. For instance, the endpoint for the Silo storage facility is “westgrid#silo”.
How to use Globus to transfer from your PC to WestGrid
Step 1: Register an account on Globus. To do this, go to https://www.globus.org/SignUp and follow the instructions. Once you’ve gotten an account set up, log into the site.
Step 2: Install Globus Connect on your PC. Follow the directions on the Globus website. Be sure to choose the correct instructions for your OS. When it asks what the endpoint name should be for your PC, use something memorable that identifies your computer. This is especially important if you will be using multiple computers to connect to Globus.
Step 3: Initialize your credentials on WestGrid. Before you can connect to WestGrid from Globus, you need to have stored a special type of credential on WestGrid’s proxy server. To do this, login to bugaboo.westgrid.ca using your WestGrid account. Once you’ve logged in, run the command “myproxy-init –c 0”. This will ask for your WestGrid password, then for a proxy passphrase. The passphrase can be anything, but many people choose the same password as their usual WestGrid login. Once the process is complete, you’ll see a message stating that a credential is stored on the MyProxy server. You can now log out of bugaboo. This process will likely be automated in the future.
Step 4: Connect the endpoints on Globus. This is done by going to the “Manage Data” tab on the Globus site. You will see two boxes, one for each end of the transfer. Enter the endpoints in the boxes labeled “Endpoint” and click the “go” button. The Globus Connect endpoint for your PC will activate automatically. The WestGrid side will open a window asking for some information. Enter your username for WestGrid, and enter the passphrase you gave when creating the proxy credential, and press continue. You should now see directory listings for your PC and the WestGrid server you selected. To transfer files, simply click the file or directory in the list, and press the appropriate transfer arrow. Your transfer will begin, and you will receive an email when it is complete. That’s all there is to it!
For complete documentation on Globus, you can visit the Westgrid support page at http://westgrid.ca/support/globus_online_file_transfer_user_guide
You can also get help on the Westgrid support mailing list, email@example.com
I’ve recently returned from a conference where I was able to attend an in-depth training session on Globus, including their new features that allow you to share files with other Globus users. I’ll be looking into getting this functionality working on WestGrid, at least Silo, in the near future. Stay tuned!