• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Extremely slow file copy

S

Stuh505

Limp Gawd
Joined
Feb 15, 2004
Messages
488
I am trying to copy 141 GB of assorted media files from one internal WD Caviar Black 500GB drive to another. One drive has about 50 GB free and the other one is completely empty, which is where I am trying to copy to. The drives are connected using SATA cables and claim performance of 3 Gb/s transfer rate, or 384 MB/sec. However, my actual transfer rate varies from 7 to 19 MB/sec. I am using Vista Business x64 SP2. Any ideas why this is going so ridiculously slowly?
 
I'm guessing you're copying smaller files like images?

SATA2's 384 MB/sec is the maximum transfer speed, your mechanical WD hard drives can probably read at 100MB/s maximum sequentially. Sequential means the file is in a nice continuous line on the disk platter, and the laser doesn't have to move back and forth to look for data. If your data is heavily fragmented or consists of many small files, then see the link below for worst case scenarios, and why we talk about SSDs nowadays.

http://www.anandtech.com/show/2738/25

Note the performance of the WD Velociraptor and Caviar Black.
 
Oh yeah, I'm not saying you need an SSD for media files, that's a waste. I'm just saying this is normal, and just set it and leave to do something else :)
 
Thanks for showing that graph. If it is normal then that makes it better...I had called WD technical support and the person I spoke to said it wasn't normal (probably just naive).

I tried scanning my windows file copy shell extension list and there was nothing suspicious there. I also downloaded HDTune and checked the info on my drives and they didn't report any problems. When the file copying is done I will try running some benchmarks to see if they look normal.
 
Not sure if Vista SP2 fully fixed the aggressive indexing that Vista was known for (haven't run Vista in a long time), but that was always a culprit of bad disk speed. If it's trying to index your target drive as files are being written to it, that could slow things down a bit.
 
One of the drives might be in high gear trying to fix errors - and without TLER that can take a long time for each error. Check SMART info on the drives to make sure this isn't the case.
 
sniggle said:
Not sure if Vista SP2 fully fixed the aggressive indexing that Vista was known for (haven't run Vista in a long time), but that was always a culprit of bad disk speed. If it's trying to index your target drive as files are being written to it, that could slow things down a bit.
Click to expand...

When I used vista and needed to perform a big file transfer like that, I stopped the windows search service, let windows finish the operation, then restarted the service, worked great that way.
 
What type of files and how many are there? Traditionally small files will have low transfer rates.
 
That is the most important part. If there are many thousands of small files this speed is not unusual for any operating system and mechanical hard drives.
 
I'm running in to the same issue trying to copy over about 1 million 1KB files. It's transferring over a Gb network to a raided storage device from a raided server. The transfer rate is at .20Mb a second. I'm currently using RichCopy, but have tried FastCopy as well and can't seem to get these files transfered any faster.

Anyone have any types of suggestions about copying over a million small files? .20Mb/s seems ridiculously slow for a Gb connection.
 
ChristmasGT said:
I'm running in to the same issue trying to copy over about 1 million 1KB files. It's transferring over a Gb network to a raided storage device from a raided server. The transfer rate is at .20Mb a second. I'm currently using RichCopy, but have tried FastCopy as well and can't seem to get these files transfered any faster.

Anyone have any types of suggestions about copying over a million small files? .20Mb/s seems ridiculously slow for a Gb connection.
Click to expand...

It's your hard drive, see the link I provided above comparing ssd and mechanical hard drives on small file transfer speed.
 
So, I'm pretty much stuck on less than 1Mb of speed due to the Raided HDD? makes sense, any suggestions at all being able to transfer the data over in some way faster?
 
You could zip those files into one large file (or several large files). You'd be copying fewer & larger files which should substantially speed up the transfer.
 
ChristmasGT said:
So, I'm pretty much stuck on less than 1Mb of speed due to the Raided HDD? makes sense, any suggestions at all being able to transfer the data over in some way faster?
Click to expand...

Just wait it out, set it up before you sleep or go out :) an SSD would do the trick but I don't think they're worth storing media on, just OS and frequently used programs.

It's up to you to decide though, SSDs run 1.5$/gig compared with like 0.05$/gig for hard drives.

imyourzero said:
You could zip those files into one large file (or several large files). You'd be copying fewer & larger files which should substantially speed up the transfer.
Click to expand...

Wouldn't the time it takes to zip up be as long as it would take to transfer them? Works if you move files around a lot though!
 
going to give a couple of things a try this evening, I've had FastCopy running since Friday and maybe got 20% into the whole process done. Thanks a ton for your help guys, love this site.
 
Nightbird said:
Wouldn't the time it takes to zip up be as long as it would take to transfer them?
Click to expand...

Honestly, I have no idea as it would depend on the hardware of the machine where the files are stored. But .20Mb/s is really slow for a Gb network, so I would venture to guess that it would be worth it if he really has that many (1 million??) 1KB files. Wouldn't hurt to try it on a subset of the files as a test. I know when I FTP to/from my file server, the directories with lots of small files take waaay longer to transfer than a single large file of the same size (like an .iso).
 
Stuh505 said:
I am trying to copy 141 GB of assorted media files from one internal WD Caviar Black 500GB drive to another. One drive has about 50 GB free and the other one is completely empty, which is where I am trying to copy to. The drives are connected using SATA cables and claim performance of 3 Gb/s transfer rate, or 384 MB/sec. However, my actual transfer rate varies from 7 to 19 MB/sec. I am using Vista Business x64 SP2. Any ideas why this is going so ridiculously slowly?
Click to expand...

No, the max transfer is 300MB/s, not 384MB/s (divide by 10 for 10-bit on sata, not 8-bit).

As for your transfers, make sure your drivers are good first and that nothing else is running or requiring anything else from the drives.

Next, are you copying a bunch of small files such as pictures, mp3s, etc?
Or are you copying large 100MB+ files such as video or game install files?

If you are copying a ton of little files, that can kill your transfer rate due to the lower seek times of 7200RPM drives, even the Blacks have this problem. If you are copying the larger files though, it would seem that something in the hardware configuration isn't right.

Even on my server RAID1 setup I can get 30-55MB+ over gigabit when copying from one system to another, let alone from drive to drive, and that's on two PATA RAID1 drives.
 
ChristmasGT said:
I'm running in to the same issue trying to copy over about 1 million 1KB files. It's transferring over a Gb network to a raided storage device from a raided server. The transfer rate is at .20Mb a second. I'm currently using RichCopy, but have tried FastCopy as well and can't seem to get these files transfered any faster.

Anyone have any types of suggestions about copying over a million small files? .20Mb/s seems ridiculously slow for a Gb connection.
Click to expand...

yes, put them into a single ZIP file, then do the transfer. Gigabit connections do not handle small file transfers well, especially if jumbo frames are enabled.
 
eh, copying two steam folders to another computer. 9 hours. what is stupid about it is that I can download all the files in about 2 or 3 hours. So I really think there is something messed up with windows file sharing.
 
Stuh505 said:
However, my actual transfer rate varies from 7 to 19 MB/sec.
Click to expand...

Since the OP said media files, I assume he is taking about music and video, none of which should be small.

Not sure how this thread got so far without this being mentioned, but it seems likely that one (or both) of your hard drives were reverted to PIO mode by Windows because of errors. Put them back in DMA mode within Windows, and your speeds should go back to normal. After DMA mode is restored, you should also re-check your sata & power cables as well as run the WD Data Lifeguard Diagnostics to make sure nothing else is wrong.
 
Red Falcon said:
yes, put them into a single ZIP file, then do the transfer. Gigabit connections do not handle small file transfers well, especially if jumbo frames are enabled.
Click to expand...

Thanks man, I'm going to try to tell them to give that a shot, I'd imagine that would speed things up quite a bit.
 
ChristmasGT said:
Thanks man, I'm going to try to tell them to give that a shot, I'd imagine that would speed things up quite a bit.
Click to expand...

If you have the drive space, then definitely. Gigabit was designed for very large files.

When I copy movie files that are 600MB-5GB in size over my gigabit network (this is without jumbo frames) I get 34-45MB/s on average. With jumbo frames enabled, normally I get 56MB/s+. This is also with crap Cat5 UTP, not Cat5e or Cat6.

However, when I copy MP3 files which are between 1-6MB each, I only get transfer speeds of 15-20MB/s if I'm lucky, and with jumbo frames on, about 14MB/s.


So I hope when you put everything in a ZIP file that it will transfer much faster for you.
 
Just wanted to give a quick update for future archives. I was actually wrong on my file count, it was much closer to 2 million than 1 million. After zipping up the folders my download went from .02MB/s to 5MB/s which was definitely better and since the file size total was only around 20 Gig's it took only roughly an hour complete.

Anyway, thanks again!
 
You must log in or register to reply here.
Back
Top