trickery.net  

Go Back   trickery.net > Technical > Software & Security

Reply
 
Thread Tools
Old 03-Mar-2011, 11:26   #1
GLoBaL ChiLLaGe
Bosh!
GLoBaL ChiLLaGe's Avatar
Join Date: Oct 2003
Location: Guernsey
Posts: 3,247
GLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond repute
Large file copy to external disk consumes all physical mem.

I'm going insane trying to resolve an issue I am currently experiencing. Basically, I am copying large (10GB-200GB) files (1.2TB total) from my raid arrays to one of five external USB 3.0 disks- (one for each days offsite backup) using RoboCopy- achieving 80MB/sec avg. However, when I try again the next day, the speed will drop to under 30MB/sec and the system will consume all available physical memory. I can reboot the server, and the problem still exists. There doesn't seem to be any logical pattern to it. One mondays backup will run perfectly, whilst the following weeks will be slow etc. causing system to grind to a halt.

The card and external disks work fine in another server/workstation. I've tried M$ Dyncache fix. I've tried enabling caching on 'legacy' expansion cards (seems an odd suggestion by HP considering its new technology) I've tried a different USB card (same NEC controller). D2D seems fine. I'm completely lost.

I can watch the backup and task manager/perfmon, and see the physical memory shoot up to 85%. As soon as it reaches this, the backup crawls along.

I have also tried using Richcopy and explorer to copy files, but still the issue persists.


System is:
Proliant ML370 G5
Server 2008 Standard
8GB RAM
P400 RAID controller w/ 512MB BBWC
-2x 74GB 10krpm raid 1
-6x 500GB 7200rpm (3x1TB~ raid 0 arrays)
Startech USB 3.0 /w NEC controller
Seagate FreeAgent GoFlex 1.5TB USB 3.0 disks

Apps/Services running other than default install.
-Acronis Backup and Recovery 10
--SQL
-AD/GPO Tools
-IIS
GLoBaL ChiLLaGe is offline  
 stant0s 
Reply With Quote
Old 03-Mar-2011, 11:33   #2
Bilb
Amazeballs!
Bilb's Avatar
Join Date: Oct 2003
Location: Elite Elevated Equine Division
Posts: 11,283
Bilb has a reputation beyond reputeBilb has a reputation beyond reputeBilb has a reputation beyond reputeBilb has a reputation beyond reputeBilb has a reputation beyond reputeBilb has a reputation beyond reputeBilb has a reputation beyond reputeBilb has a reputation beyond reputeBilb has a reputation beyond reputeBilb has a reputation beyond reputeBilb has a reputation beyond repute
As a quick test, will copying the file across a network work fine? I'm just wondering if it's something about USB, that it holds it all in memory until it's completed.

Thus sending it to a network destination might work!?
__________________
@TomJJarvis
Bilb is offline  
Send a message via Skype™ to Bilb  jarvthelegend 
Reply With Quote
Old 03-Mar-2011, 11:54   #3
GLoBaL ChiLLaGe
Bosh!
GLoBaL ChiLLaGe's Avatar
Join Date: Oct 2003
Location: Guernsey
Posts: 3,247
GLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond repute
I did try this on my 2k3 server, and didn't seem to get above 40% network usage (1Gbit LAN) - I'm gonna try it again now and copy another file at the same time and see if I can max out the network/disk any more.

Last edited by GLoBaL ChiLLaGe; 03-Mar-2011 at 12:01.
GLoBaL ChiLLaGe is offline  
 stant0s 
Reply With Quote
Old 03-Mar-2011, 12:09   #4
burundi
WARNING: May contain nuts
burundi's Avatar
Join Date: Oct 2003
Location: Prestonia
Posts: 11,308
burundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond repute
Write caching enabled on the USB disk? If so try turning it off.

Can the USB 3 disk use a USB 2 connector, if so does the same thing happen when using USB 2?
__________________
I arose gigantic and black, I howled at the night and the night howled back.
burundi is offline  
 Geldrey 
Reply With Quote
Old 03-Mar-2011, 12:15   #5
GLoBaL ChiLLaGe
Bosh!
GLoBaL ChiLLaGe's Avatar
Join Date: Oct 2003
Location: Guernsey
Posts: 3,247
GLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond repute
by default, fast-removal was enabled. I've tried both settings.

Good idea- I will try usb2 after i've tested this.

I've got one disk using USB3 now and just checked the speed..it's dropped to 7MB/sec
GLoBaL ChiLLaGe is offline  
 stant0s 
Reply With Quote
Old 03-Mar-2011, 12:53   #6
Nomad
i fear no midget
Nomad's Avatar
Join Date: Feb 2007
Posts: 10,402
Nomad has a reputation beyond reputeNomad has a reputation beyond reputeNomad has a reputation beyond reputeNomad has a reputation beyond reputeNomad has a reputation beyond reputeNomad has a reputation beyond reputeNomad has a reputation beyond reputeNomad has a reputation beyond reputeNomad has a reputation beyond reputeNomad has a reputation beyond reputeNomad has a reputation beyond repute
had this at home with a usb 3 drive, although not sure if was that that was the issue, write speeds dropped, mem used u

turning off the indexing on that drive seemed to help
Nomad is offline  
Reply With Quote
Old 03-Mar-2011, 12:59   #7
GLoBaL ChiLLaGe
Bosh!
GLoBaL ChiLLaGe's Avatar
Join Date: Oct 2003
Location: Guernsey
Posts: 3,247
GLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond repute
Yea, I've turned it off on all drives, but didnt help. Annoyingly after a format, by default it turns it back on.
GLoBaL ChiLLaGe is offline  
 stant0s 
Reply With Quote
Old 03-Mar-2011, 13:51   #8
GLoBaL ChiLLaGe
Bosh!
GLoBaL ChiLLaGe's Avatar
Join Date: Oct 2003
Location: Guernsey
Posts: 3,247
GLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond repute
Well, copying from network share to usb3 directly - was OK. No slow down as such (still only reached 40-45% network usage on 1gbit, copying one file. Copying two files it jumped to 57% network usage and perfmon was reporting a steady 67MB/sec write to USB3 disk)

Copying from local disk to usb3 killed the box.

Copying from USB3 port1 to USB port2 was fine.

I'm wondering if it's the NEC chipset. I can't see it being an issue with raid as all my cards have their own cache?

Last edited by GLoBaL ChiLLaGe; 03-Mar-2011 at 13:53.
GLoBaL ChiLLaGe is offline  
 stant0s 
Reply With Quote
Old 03-Mar-2011, 13:59   #9
burundi
WARNING: May contain nuts
burundi's Avatar
Join Date: Oct 2003
Location: Prestonia
Posts: 11,308
burundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond reputeburundi has a reputation beyond repute
RAID card and USB card not sharing an IRQ are they?
__________________
I arose gigantic and black, I howled at the night and the night howled back.
burundi is offline  
 Geldrey 
Reply With Quote
Old 03-Mar-2011, 21:26   #10
Homer-Simpson
if thread.newpost then postreply();
Homer-Simpson's Avatar
Join Date: Oct 2003
Posts: 4,865
Homer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond repute
http://blogs.msdn.com/b/granth/archi...e-network.aspx

Quote:
The first problem that you’ll encounter when using a normal Windows file copy, XCopy, RoboCopy or TeraCopy to copy these large files is that your available memory on the source server will start to drop and eventually run out.
Might be worth a go?
Homer-Simpson is offline  
Reply With Quote
Old 14-Mar-2011, 01:01   #11
Homer-Simpson
if thread.newpost then postreply();
Homer-Simpson's Avatar
Join Date: Oct 2003
Posts: 4,865
Homer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond reputeHomer-Simpson has a reputation beyond repute
How did you get on?
Homer-Simpson is offline  
Reply With Quote
Old 14-Mar-2011, 12:37   #12
DizMatt
PostPrandial
DizMatt's Avatar
Join Date: Oct 2003
Location: just ootside Gatesheeed
Posts: 8,690
DizMatt has a reputation beyond reputeDizMatt has a reputation beyond reputeDizMatt has a reputation beyond reputeDizMatt has a reputation beyond reputeDizMatt has a reputation beyond reputeDizMatt has a reputation beyond reputeDizMatt has a reputation beyond reputeDizMatt has a reputation beyond reputeDizMatt has a reputation beyond reputeDizMatt has a reputation beyond reputeDizMatt has a reputation beyond repute
I used handybackup for this back in the day
if it fails a schedule then it'll try again
what format is on these external drives?
__________________
Photo Portfolio Bits 'n Pieces : General Random Gallery : All new photoblog : My Flickr
........................I read it on Wikipedia...so it must be true!........................
DizMatt is offline  
Send a message via MSN to DizMatt
Reply With Quote
Old 15-Mar-2011, 10:30   #13
GLoBaL ChiLLaGe
Bosh!
GLoBaL ChiLLaGe's Avatar
Join Date: Oct 2003
Location: Guernsey
Posts: 3,247
GLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond reputeGLoBaL ChiLLaGe has a reputation beyond repute
Have been reading about ESEUTIL - and did give it a go. However, one major problem is that it doesn't accept wildcards and needs a full filename. I have about 60 or so files which are named in accordance with Acronis naming convention and can't be changed. SERVERNAME.DOMAIN.DATE.TIME.TIB annoyingly!

Thanks for the info though. Think I'm gonna have to investigate some offsite backup again, hopefully I'll get the funds this time.
GLoBaL ChiLLaGe is offline  
 stant0s 
Reply With Quote
Reply

Go Back   trickery.net > Technical > Software & Security

Tags
open it in notepad, turn it off and on again

Similar Threads
Thread Thread Starter Forum Replies Last Post
Vista SP1's File Copy Improvements - Russinovich GroovYF Software & Security 3 05-Feb-2008 13:22
The fun you can have with blograffiti blurr Discussion 43 24-Aug-2006 11:26
My backup DVD's keep failing fab Hardware 4 05-May-2005 18:02
Computer Help Goody Half-Life: Counter-Strike 35 13-Jun-2004 14:00
crashes. recky Massively Multiplayer Games 7 22-Nov-2003 17:56

Users Viewing Thread: 1 (0 members and 1 guests)
 

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT +1. The time now is 14:05.


Powered by vBulletin® Version 3.7.0 Release Candidate 3
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright 2003 - 2013, trickery.net