Originally Posted: Aug 24, 2022 Dealing with huge web3 data, I see myself doing this a lot: moving data directories across machines. In this case, we have a 250 GB directory with 10K files. The network connection between machines is ~5 MBps. But using these techniques, we can achieve a 6x faster transfer (~30 MBps). There’re two ways to do this: Use compressed tar (Newer and Older) Use rsync Update (Dec 6, 2022) I wrote another post comparing the performance of various compression algorithm...
