Background I run a chess site, https://chessbook.com. We ingest a ton of games, nearly 6 billion, to power our various statistics. Our current database has about 900 million lines. We’re working on our next evolution of this database, and that entails re-ingesting all these games, but going much deeper on each game. Our existing solution was already pretty optimized, you can’t get through 1.6 terabytes of data in a reasonable timeframe without some optimization, but we needed to really sq...| mbuffett.com