Just finished a tough project. It had to download 200GB data and ingest 100GB into database. It turned out that SQLite is too slow, and I had to use RocksDB instead.