MIT researchers develop flash memory solution for big data

By Nicole Miskelly | 14 July 2015

Researchers at the Massachusetts Institute of Technology (MIT) have revealed a new network design called BlueDBM which they say could make flash memory servers more efficient for big data applications.

Big data presents a variety of challenges, including large data sets which are unable to fit into traditional servers using RAM or machines full of memory. An MIT statement said: “A processor can retrieve data from RAM tens of thousands of times more rapidly than it can from the computer’s disk drive. But in the age of big data, data sets are often too large to fit in a single computer’s Ram. The data describing a single human genome would take up the RAM of somewhere between 40 and 100 typical computers.”

BlueDBM was first unveiled at the International Symposium on Computer Architecture in June and is apparently as efficient as servers using RAM, but is cheaper and saves more power. “It’s about a tenth as expensive, and it consumes about a tenth as much power,” said MIT.

According to MIT the solution was to remove the memory and create a cluster comprised entirely of solid-state drives. The Inquirer reports that the researchers were able to make a network of flash-based servers competitive with a network of RAM-based servers by moving computational power from the servers and onto the chips that control the flash drive, and according to MIT, by pre-processing some of the data on the flash drives before passing it back to the server, the chips could make distributed computation much more efficient.

The researchers built a prototype network of twenty servers each connected to a chip that can be reprogrammed to mimic different types of electrical circuits (FPGA) and each FPGA was connected to 500GB of flash chips and to the two nearest FPGA’s in the server rack. It was this connection of FPGA’s that created a very fast network that allowed any server to retrieve data from any of the flash drives.

According to the Inquirer, in order to show that the FPGA’s also executed the algorithms from the pre-processed data stored on the flash drives with application-specific algorithms, the researchers tested some of the algorithms against some popular big data applications. These included Google’s PageRank algorithm and an image search which tried to find matches for a sample image within a large database. 

MIT said that because FPGAs are reprogrammable, depending on the application, they could be loaded with different accelerators like the ones above and many applications could benefit from this functionality.

Become a bobsguide member to access the following

1. Unrestricted access to bobsguide
2. Send a proposal request
3. Insights delivered daily to your inbox
4. Career development