computer data center

Supercomputer Performance Using Commodity Components for GOES-R

By James Gundy, GOES-R Product Chief Software Engineer, and Bradley Brown-Bergtold, GOES-R Chief High-Performance Computing Engineer
Oct 21, 2016

The GOES-R ground system will produce 26 more weather products than the previous GOES ground system, based on the increased amount of raw data coming from the instruments in space. To be more precise, the GOES-R ground system will be receiving raw data at a rate of 100 Mbps to generate and distribute weather products and metadata at a volume of 16.1 Terabytes per day. At this rate, an average hard disk drive would be filled in just six hours.

So it’s hardly a surprise that Harris’ solution for the GOES-R ground system leverages high-performance computing technologies and techniques that support high data throughput and parallel processing. But crunching big data is only part of what the GOES-R ground system must do. It also has to meet the demand of low-latency, or near-real-time, product delivery. And it must be cost effective and sustainable for years to come.

Here’s a quick look at how we’ve designed the GOES-R ground system to do all that.

Delivering Big Data Fast

Harris has significantly reduced the time to generate GOES-R products through an innovative approach that decomposes data received from the instruments into small blocks and distributes them across multiple computer servers for parallel processing. This allows the processing of data to begin before all of the data for a full-disk scan of the earth is received from the satellite. Solar products crucial for safeguarding our nation’s infrastructure and power grids will be produced in as little as 1.8 seconds, and imagery products to support the timely forecasting of severe weather events will be generated in only 23 seconds!

In order to ensure weather products and metadata are distributed to users in a timely manner, product distribution is tightly integrated with product processing to minimize data transfer overhead. Given the small amounts of time that are allocated to individual processing and distribution jobs, optimizing data transfer operations is important to maintaining consistent throughput.

The GOES-R product generation and distribution system addresses data transfer performance using a mix of in-memory data caching and a high-performance parallel file system. Many of these technologies are commonly used in other high-performance computing systems around the world. However, Harris has further developed a system management and monitoring framework to deploy status processing and distribution, and monitor the system and its applications to quickly assess and effectively manage performance.

A key component of this framework provides data replication and service resiliency to ensure application and hardware failures do not result in the loss of product data or system time. This allows the GOES-R system to achieve system uptimes greater than 99.9 percent.

Commodity Components for System Scalability, Growth, and Sustainability

Harris designed the GOES-R ground system to be highly scalable. In fact, the system can be scaled beyond 300 percent of the current GOES-R product processing capability without the need for software modifications or enhancement!

Scalability, along with system evolution, growth, and sustainability, are all supported by the use of proven, off-the-shelf (OTS) hardware and software components.  These commodity components are grounded on open, standards-based technologies, which prevent vendor dependency and high licensing costs. As new and innovative technologies become available to improve system performance and reliability, they can be easily and cost-effectively inserted. And overall, the use of commodity components is expected to help lower the total cost of ownership.

Read more about Harris’ involvement in the GOES-R mission.