Automating Transfers Between Campus and the Cloud With Globus

Tags: ,

After migrating some digital object storage and related web applications to Amazon Web Services (AWS) last year, we have created custom workflows that involve the transfer of files between campus storage and cloud storage: dataset files of up to 2TB into and out of Illinois Data Bank by campus researchers, and millions of files into and out of our Medusa digital preservation system by University Library staff. Our initial solutions included using a combination of the AWS S3 API and rclone. Hoping to make transfers more robust, transparent, and faster, we set up Globus endpoints using features supported by our campus Globus subscription. Since Globus is useful to the campus and research community in proportion to how many of our sharable files are accessible on endpoints, we would like offer pragmatic details about integrating Globus into our systems to help anyone else considering adding Globus endpoints for storage systems.