Copying multiple files in multiple directories from server to local, with files going into directories with the same name as server

Issue

Sorry if the title is confusing

Basically, on my server, I have multiple files for multiple runs of an analysis for multiple traits, e.g.:

/Weight/run1/file1.txt
/Weight/run1/file2.txt
/Weight/run2/file1.txt
/Weight/run2/file2.txt
/LegLength/run1/file1.txt
/LegLength/run1/file2.txt
/LegLength/run2/file1.txt
/LegLength/run2/file2.txt

In total I have 11 traits with two runs each, with each run generating 26 files.

I want to copy them across onto my local machine, but keeping the file hierarchy the same. I can only find code that copies from multiple directories into one directory, which is not what I want.

So far I’ve managed to hack together the following code from somewhat related stackoverflow questions, though obviously doesn’t work and just tries to find ~/Documents/path/to/local/directories on the server. I was wondering if anyone had any advice on how to tweak it?

`ssh [email protected] 'for f in /path/to/server/directories/*/*/*txt ; do newpath="${f%/*}"; newpath=${newpath:41}  ; scp [email protected]:${f} ~/Documents/path/to/local/directories/${newpath};  done'`

I also have multiple other files in each end directory on the server with different extensions, but I am not interested in copying them across.

(Edit to add: I’m using newpath=${newpath:41} because it removes /path/to/server/directories/ from the variable)

Solution

It’s a little complex to get it right but a combination of --include and --exclude in the rsync command should be able to do what you want:

rsync -av \
      --include '/Weight/' \
      --include '/Weight/run[12]/' \
      --include '/Weight/run[12]/file[12].txt' \
      --include '/LegLength/' \
      --include '/LegLength/run[12]/' \
      --include '/LegLength/run[12]/file[12].txt' \
      --exclude '/*' \
      --exclude '/*/*' \
      --exclude '/*/*/*' \
      [email protected]:'/path/to/server/directories/' ./here/

note: in this case, the first / in the includes & excludes means /path/to/server/directories/

Answered By – Fravadona

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply

(*) Required, Your email will not be published