headers
Michael Burek | 6 Sep 2008 10:38

Re: Command line multi-part copy program

 <at> Brian Lasch:

I'm pretty much under the impression that copying is limited to the disk
speed, or bus bandwidth.  I really don't think the processor is involved
enough to make a difference.  Especially at Gigahertz speeds.  Using
firewire drives will take nearly all the processing off of the cpu and put
in onto the firewire interface card.

Also, a hard drive has a single head, so a hard drive itself is most
efficient in a single thread sequential read.  Having 2 threads read from 2
areas of the drive will cause seek time to enter the time equation and slow
things down.  Having a big file, all in a row, will produce the fastest read
result.  Even if drives had 2 separate 1 gigabyte buffers, those buffers
would have to take turns reading with only 1 drive head.  And even with a
solid state drive or a flash drive, those still work faster with a
sequential read than a random read.  But their random read is much faster
than a platter hard drive random read.

PS, if you read the question, you'll get your answers to what the question
is about.

-----
 <at> Parag P. Doke

What is the bandwidth available?  If you max it out during the copying, then
no software can increase the speed.  The only hope is to zip the files on
the source computer, and transfer the zip files, when your bandwidth is the
bottleneck.  You'd have to zip each file separately and update the zip file
on the source computer when the original changes, so that you'd only copy
the updated files.  Zipping into 1 large file would always bring over every
(Continue reading)

Permalink | Reply |
headers
Parag P. Doke | 6 Sep 2008 11:57
Picon

Re: Command line multi-part copy program

Hello Andrew.
Thanks for the explanation about how robocopy helps to create mirrors.
However, that is not really what I was looking for. In my case, the
remote shared folder gets a new build created (in a folder) and my
intent is to copy over only the newly created folder to my local
machine (regardless of what happens to folders / files created in
other locations inside the source).

When I ran the robocopy command as suggested by you (including /l), it
did list the contents it was about to copy over. Without the /l, it
started to copy data.....but one file at a time. Also there was no
indication if the file copy operation of the current file was split
into chunks and then data re-joined on the destination. In short, I
found axel downloading a lot faster than robocopy (over http).

Thanks again.

Hi Brian.
I believe I've answered most of the questions in an earlier mail on
this thread. Pasting it below:
-----
1) Total size of data is somewhere around 350 MB.
2) Total # of files: roughly 50 (contained in 3 folders). Size of one
file is to the tune of 330 MB. Rest total up to around 20 MB.
3) Pardon my lack of knowledge...but I'm copying over
\\MachineName\SharedFolder. The system "MachineName" is located in the
US while I'm located in India (Pune). I'm not sure whether to call it
LAN or WAN.
4) Yes, I'm trying to keep 2 locations in sync. The copy process is
triggered each morning.
(Continue reading)

Permalink | Reply |
headers
Andrew Palm | 6 Sep 2008 13:58
Picon

Re: Command line multi-part copy program

Remove the /MIR and it will do a oneway copy...

Copying is restartable from the point it stopped, but it doesn't do delta
copies, which is what your after.

Enjoy your solution.

Andrew

On Sat, Sep 6, 2008 at 7:57 PM, Parag P. Doke <paragpdoke@...> wrote:

>   Hello Andrew.
> Thanks for the explanation about how robocopy helps to create mirrors.
> However, that is not really what I was looking for. In my case, the
> remote shared folder gets a new build created (in a folder) and my
> intent is to copy over only the newly created folder to my local
> machine (regardless of what happens to folders / files created in
> other locations inside the source).
>
> When I ran the robocopy command as suggested by you (including /l), it
> did list the contents it was about to copy over. Without the /l, it
> started to copy data.....but one file at a time. Also there was no
> indication if the file copy operation of the current file was split
> into chunks and then data re-joined on the destination. In short, I
> found axel downloading a lot faster than robocopy (over http).
>
> Thanks again.
>
> Hi Brian.
> I believe I've answered most of the questions in an earlier mail on
(Continue reading)

Permalink | Reply |

Gmane