IU Cyberinfrastructure Enhancements
On April 5, 2006, Indiana University officials announced the acquisition of the nation's fastest university-owned supercomputer and largest disk-based storage facility. These additions to IU's cyberinfrastructure, described in detail below, will enhance researchers' ability to generate, store and analyze the increasingly large datasets involved in a variety of scientific fields. The new systems, designed in collaboration with IU's Cyberinfrastructure Task Force, will come online throughout the Summer and Fall of 2006, and will complement existing research storage and high-performance computing systems at IU.
The two computational components of the additional systems include the 20.48 Teraflop Big Red cluster and the addition of six IBM p575 Power5 systems to the Libra cluster.
Big Red is a distributed shared-memory cluster, consisting of 512 IBM BladeCenter JS21s, each with two dual-core PowerPC 970 MP processors (2.5GHz), 8GB of ECC PC3200 SDRAM, 72GB local SATA disk for scratch space, and a PCI-X Myrinet 2000 adapter for high-bandwidth, low-latency MPI applications. In addition to local scratch disk, the Big Red compute nodes will be connected via gigabit ethernet to a 200 TB GPFS filesystem, hosted on 16 IBM p505 Power5 systems attached to eight DataDirect Networks S2A9500 disk controllers.
The Big Red cluster will run SuSE Linux Enterprise Server 9. Batch jobs will be managed with IBM's Loadleveler and the Moab Workload Manager. A significant portion of Big Red will be allocated to TeraGrid utilization.
The six p575 systems being added to the Libra cluster are each equipped with eight 1.9 GHz Power5 processors, 16GB RAM and two 146GB SCSI U320 drives (RAID1). These systems will facilitate large-memory jobs, and will have access to an additional 50TB GPFS volume. Libra nodes run IBM's AIX 5.3, with Loadleveler and Moab managing batch jobs. Both Big Red and the Libra nodes will share home directory space on a 25 TB IBM N5500 NAS system.
The several storage components being added include the Data Capacitor, an NSF-funded massive digital data storage system; several terabytes of SAN-based database storage; and new disk cache and tape storage for IU's High Performance Storage System.
The Data Capacitor, built around the Lustre filesystem from CFS, consists of 535 TB of SATA disk, 12 DataDirect Networks S2A9500 controllers, and 24 Dell PowerEdge 2850 storage servers and 24 PowerEdge 1850 transfer servers. This system, intended to facilitate the rapid delivery of extremely large datasets to IU's computational resources, will be capable of transferring data at approximately 12 GB/s, aggregated across all storage servers.
An additional 50 TB of SATA disk will augment the existing Research Database Complex, providing an order of magnitude increase in the database storage available to IU researchers. The newly-established Scientific Data Services group will provide database administration and consulting services related to research databases.
IU's HPSS will benefit from 75 TBs of additional disk on the IUPUI campus and 90 TBs at IUB, as well as 18 new IBM p575 servers. Forty IBM TS1120 tape drives, divided between IUPUI and IUB, and 2000 500GB tapes will add 1 petabyte of tape storage and increase the speed with which data can be written to and read from HPSS.
These additions, and in some cases, enhancements to existing systems, will greatly increase the computational cycles and storage capacity available to IU researchers. The Big Red cluster alone has a peak theoretical performance ten times that of the existing AVIDD-B and AVIDD-I clusters, with nearly 100 times more GPFS storage. Associated network upgrades in both the Bloomington and Indianapolis data centers will allow these systems to be connected seamlessly over the I-Light network, providing for the distribution of phyiscal resources as well as support services across campuses. This integration will lead to more efficient management of Indiana University's high-performance research and storage systems, which will in turn lead to more effective science.
Please see the following links for updates regarding the status of the various systems described above:
Questions or consultations regarding the most effective utilization of these resources can be arranged; send mail to rac at indiana.edu.
