Data Archiving
A Government agency’s outdated Research and Development (R&D) data archive was reaching capacity and needed an upgrade. RedLine designed and implemented a new 50-petabyte data archive to increase capacity by 400% and provide more efficient, consolidated data access across multiple sites.
Challenge
A Government agency must archive the information it generates daily from three separate R&D sites and provide efficient access to analysts who improve and troubleshoot weather and climate prediction models. The agency’s 12 petabyte R&D infrastructure system had become outdated, unstable, and was reaching capacity. The agency had been maintaining separate R&D support services at three differently configured sites. The agency needed assistance planning for, designing, implementing, and supporting an updated and consolidated data archive solution.
Solution
To expand capacity and optimize the agency’s R&D data archives, RedLine played a key role in designing and installing a new, more efficient R&D infrastructure platform and data archive with over four times the capacity (50 petabytes) of the old system. The team developed the new data archive to support multiple sites with one Hierarchal Storage Management System (HSMS).
RedLine needed to keep the legacy archive up and running while it installed the new R&D archive. To accomplish this, the team used a multi-stage process. We installed and configured the new system with hardware to read the legacy tapes. RedLine then tuned the performance of the server kernels, network interfaces, and wide area network (WAN), transferring data at faster rates to meet the customer requirements. All data writes were transitioned to the new HSMS by changing the archive tool scripts to point to the new system. The legacy archive tape libraries were then integrated into the new HSMS environment. Finally, we copied all legacy data to the new HSMS. RedLine also created scripts to first check the new system tapes for files before retrieving the data from the legacy tapes. This process provided a smooth transition for the user community to the new HSMS.
Results
Implementing the new archive increased the agency’s R&D capacity 400% to 50 petabytes, made all data accessible to more users at multiple sites, and created a more reliable, stable system. The elimination of separate archives and staff at each site significantly reduced costs for the customer. This implementation also helped the user community streamline their workflows by using a single command set when archiving data, regardless of their location.
Following the successful design and installation of its new R&D data archive and optimizing the network infrastructure performance, the customer engaged RedLine to additionally provide a systems support administrator at one site and an analyst at another to help users port and optimize their research applications for the R&D supercomputer, improving future weather and climate prediction models.
