Source: https://github.com/jbd/msrsync
Why write msrsync if tools like fpart, parsync or pftool exist ? While reasonable, their dependencies can be a point of friction given the constraints we can have on a given system. When you’re lucky, you can use your package manager (fpart seems to be well supported among various GNU/Linux and FreeBSD distribution: FreeBSD, Debian, Ubuntu, Archlinux, OBS) to deal with the requirements but more often than not, I found myself struggling with the sad state of the machine I’m working with.
That’s why the only dependencies of msrsync are python >=2.6 and rsync. What python 2.6 ? I’m aiming RHEL6 like distribution as a minimum requirement here, so I’m stuck with python 2.6. I miss some cool features, but that’s part of the project.
The devil is in the details. If you need a starting point to think about data migration, this overview by Jeff Layton is very informative: Moving Your Data – It’s Not Always Pleasant.
The “How to transfer large amounts of data via network” article by parsync author is updated regularly and its worth a read also.
If you can read french, I co-wrote an article with Ganaël Laplanche about fpart : Parallélisez vos transferts de fichiers.
You might be also interested by this Intel whitepaper on data migration : Data Migration with Intel® Enterprise Edition for Lustre* Software which mentions all of those tools (but not msrsync).