IBM Begins Work on 120 Petabyte Data Repository

As if Watson wasn’t enough for them, scientists at IBM are noe working on a brand new 120 petabyte data repository which is to be made up of 200,000 conventional hard disk drives all working together simultaneously. This new gigantic data container is expected to store somewhere around 1 trillion files and should provide the space needed to allow more powerful simulations of complex systems, similar to those used to model weather and climate.

This new system benefits from a file system known as General Parallel File System, or GPFS, which was developed at IBM Almaden in order to enable supercomputers faster data access. In addition to that, it also spreads individual files across multiple disks so that many parts of the file can be read or written to at the exact same time.

General Parallel File System leverages cluster architecture in order to provide quicker access to file data. This file data is automatically spread across multiple storage devices, providing optimal use of available storage in order to deliver high performance. In addition to that, it is also the storage engine for IBM’s Watson, the famous supercomputer that beat Ken Jennings and Brad Rutter on Jeopardy!.

But that isn’t even the most interesting part. 120 petabytes equals nearly 24 billion 5 megabyte MP3 files, which definitely seems like a lot. However, if you contrast that against the immense volume of data being amassed from sites like Facebook which, in 2009, was steadily storing 25 terabytes of logs per day, you will see that only 4,915 days of Facebook log data can be stored here.

With the amount of data online and offline growing like The Blob, it seems likely that 120 petabytes won’t seem so impressive in a couple of years. However, it does go to show you that there is a lot of room for innovation around storage and file systems despite the maturity of the market.

Source: CNET – IBM goes for really, really, really big data