Jump to content


 


Register a free account to unlock additional features at BleepingComputer.com
Welcome to BleepingComputer, a free community where people like yourself come together to discuss and learn how to use their computers. Using the site is easy and fun. As a guest, you can browse and view the various discussions in the forums, but can not create a new topic or reply to an existing one unless you are logged in. Other benefits of registering an account are subscribing to topics and forums, creating a blog, and having no ads shown anywhere on the site.


Click here to Register a free account now! or read our Welcome Guide to learn how to use this site.

Photo

crashing copy operations using Linux live


  • Please log in to reply
27 replies to this topic

#1 Captain Dunsel

Captain Dunsel

  • Members
  • 46 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Maryland, USA
  • Local time:07:18 AM

Posted 15 January 2016 - 09:09 AM

Hi

 

I've been copying my personal files from my internal hard drive to an external one, working in a live Linux environment.  (I've used several distros, including Zorin, Linux Mint, Lubuntu and Ubuntu -- all of them were on a flash stick.)  I have 12 GB of RAM.  When copying large files or large groups of files, the whole operation at some point inevitably crashes hard, leaving my computer unresponsive.  No input from the keyboard or mouse has any effect, and I have to press and hold the power button to shutdown and reboot.  That of course leaves the last file screwed up, and in one case made the entire external drive inaccessible.  I got nothing but an error message when I tried to mount it.  I had to move the drive to a Windows machine, where many errors were detected on bootup and an automatic scan corrected them.

 

This first happened when I was copying a folder of hundreds or possibly thousands of smaller files -- on the order of kilobytes or tens of megabytes.  It occurred to me that maybe I can't exceed system RAM, so anything close to 12 GB should be broken into multiple smaller copies.  I did that, copying much smaller groups of files at a time, and still got some crashes.  Lately I've been copying large files (recorded TV shows -- typically 1 to 4 GB), and because of my earlier experience I'm doing these one at a time, by dragging from folder to folder.  It's easy but tedious and very slow, and still sometimes that causes a crash. 

 

I'd much rather copy the whole folder, for time and effort.  So my question is:

 

Why can't I do that??  Is there something I'm overlooking... something inherent in using a live Linux distro which is causing this?

 

Thank you.

 

Oh, I forgot I have a picture or two of error messages, but now I have no time to attach.  If anyone wants to see it I'll post later...



BC AdBot (Login to Remove)

 


#2 stratosgr

stratosgr

  • Members
  • 88 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Greece
  • Local time:01:18 PM

Posted 15 January 2016 - 09:35 AM

Copying files does not require much resources from your system so I don't think this is the problem. 

Your drive was inaccessible because it was not unmounted correctly so dirty removal flag was on for the stick. You could force mount it or fsck it. 

 

My thoughts goes to usb power and or usb port failure. 

First of all remove all unnecessary usb devices and try copying your files. Also try using another port to see what happens. 

 

On your live usb stick is persistent storage on? Can you keep logs after the crash or they're deleted on reboot?

 

Checking your dmesg log should hint you the error. 



#3 DeimosChaos

DeimosChaos

  • BC Advisor
  • 1,420 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:United States, Delaware
  • Local time:07:18 AM

Posted 15 January 2016 - 10:35 AM

The cause could be that you are using them on a live USB drive. It may not like being booted from a USB drive then copying tons of files to another USB drive. I would try Puppy linux and boot off a Live CD instead. Puppy runs entirely from RAM so this would eliminate the multiple USB drives and trying to boot off one and put lots of files onto another. I could be entirely wrong... but its worth a shot.


OS - Ubuntu 14.04/16.04 & Windows 10
Custom Desktop PC / Lenovo Y580 / Sager NP8258 / Dell XPS 13 (9350)
_____________________________________________________
Bachelor of Science in Computing Security from Drexel University
Security +


#4 Al1000

Al1000

  • Global Moderator
  • 7,721 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Scotland
  • Local time:12:18 PM

Posted 15 January 2016 - 02:54 PM

It's the same using the cp command in the terminal as it is dragging and dropping. I just booted up with a Puppy CD and copied a 6.7GiB directory from one partition to another, and saw memory usage more than quadruple. It didn't go back down after the directory had finished copying either. Not even after closing the terminal windows and unmounting the partitions.

I then deleted the copy of the directory, and booted up with Kubuntu to copy it again. This time the computer only used around 40MiB more RAM to copy the 6.7GiB directory.

So I'm not sure why this is, but it does seem that live Linux uses considerably more RAM to copy files than Linux installed does.

It's worth noting though that this computer only has 4GiB of RAM, and there was still plenty to spare while the directory was copying with Puppy on a live CD too. So I second DC's suggestion re trying Puppy.

Installing Linux to a USB stick rather than using it "live" would be another option.

Edited by Al1000, 15 January 2016 - 02:56 PM.


#5 raw

raw

    Bleeping Hacker


  • Members
  • 2,577 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Texas
  • Local time:06:18 AM

Posted 15 January 2016 - 06:45 PM

 

I've been copying my personal files from my internal hard drive to an external one

 

How is the external drive formatted?


rawsig.png

 rawcreations.net          @raw_creations


Current systems: WHAT OS, BackTrack-raw, PCLinuxOS, Peppermint OS 6, Kali Linux

and a custom Linux From Scratch server hosting a bunch of top secret stuff.


#6 Captain Dunsel

Captain Dunsel
  • Topic Starter

  • Members
  • 46 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Maryland, USA
  • Local time:07:18 AM

Posted 15 January 2016 - 10:59 PM

Copying files does not require much resources from your system so I don't think this is the problem. 

Your drive was inaccessible because it was not unmounted correctly so dirty removal flag was on for the stick. You could force mount it or fsck it. 

 

My thoughts goes to usb power and or usb port failure. 

First of all remove all unnecessary usb devices and try copying your files. Also try using another port to see what happens. 

 

On your live usb stick is persistent storage on? Can you keep logs after the crash or they're deleted on reboot?

 

Checking your dmesg log should hint you the error. 

 

I'm sorry, you don't think what is the problem?  My concern about the system RAM? 

 

You say "dirty removal flag was on for the stick".  You mean USB stick?  Well, it's a hard drive that I'm copying to.  But that's interesting anyway, and I guess this "dirty removal flag" can happen on a hard drive as well.  Your explanation makes sense, and I suspected it was something like that.

 

"You could force mount it or fsck it."  How do I do that?  I've heard the term fsck, but don't know anything about it.  What does that do?  Anyway, it's just for info at this point, because Windows cleaned the disk up so it's usable again.

 

As for USB power or port failure, I just can't see that as the explanation.  If either of those were the case, it seems to me I'd have problems not just on large files or large groups of files, but any files, at least intermittently.  For what it's worth, I just installed the USB 3.0 card that I am using for the job, a couple weeks ago.  I did connect a power cable to the card.  But I will take your advice and remove a few unnecessary drives while doing the copying.

 

No, I don't have persistent storage on my live USB stick, at least not with the distro I've been using for most of the copies -- Zorin.  I had Linux Mint 17 and L.M. 17.3 on two other sticks, but something went wrong with both of those distros and neither one will boot now.   :-(   (I have three different sticks, each with more than one distro on them, created with YUMI, a multiboot creation tool.)  I sometimes wonder if I'd be better off with a single distro on each stick, because I also see lots of browser crashes in every live Linux I've tried, and they're all on these multiboot sticks.  Hmm.  So anyway, I suspect all logs are deleted on reboot.



#7 Captain Dunsel

Captain Dunsel
  • Topic Starter

  • Members
  • 46 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Maryland, USA
  • Local time:07:18 AM

Posted 15 January 2016 - 11:02 PM

The cause could be that you are using them on a live USB drive. It may not like being booted from a USB drive then copying tons of files to another USB drive. I would try Puppy linux and boot off a Live CD instead. Puppy runs entirely from RAM so this would eliminate the multiple USB drives and trying to boot off one and put lots of files onto another. I could be entirely wrong... but its worth a shot.

 

I like Puppy Linux and I tried that some time ago.  The problem is that in Puppy, the source drive I'm copying from (which is a RAID array) does not show up.  It seems that Puppy cannot mount a RAID array.  However, I have considered trying a live CD of something, just to see if it makes a difference.  All I've ever used (in the Linux world) is a multiboot stick created with YUMI, and one early Linux Mint stick created with Unetbootin.



#8 Captain Dunsel

Captain Dunsel
  • Topic Starter

  • Members
  • 46 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Maryland, USA
  • Local time:07:18 AM

Posted 15 January 2016 - 11:11 PM

It's the same using the cp command in the terminal as it is dragging and dropping. I just booted up with a Puppy CD and copied a 6.7GiB directory from one partition to another, and saw memory usage more than quadruple. It didn't go back down after the directory had finished copying either. Not even after closing the terminal windows and unmounting the partitions.

I then deleted the copy of the directory, and booted up with Kubuntu to copy it again. This time the computer only used around 40MiB more RAM to copy the 6.7GiB directory.

So I'm not sure why this is, but it does seem that live Linux uses considerably more RAM to copy files than Linux installed does.

It's worth noting though that this computer only has 4GiB of RAM, and there was still plenty to spare while the directory was copying with Puppy on a live CD too. So I second DC's suggestion re trying Puppy.

Installing Linux to a USB stick rather than using it "live" would be another option.

 

Interesting about the quadruple+ memory usage in Puppy.  I also have Lubuntu, which I suppose is similar to Kubuntu; I guess I should try my copies in that one (can't recall if I already have).  Oh, but your point is that installed Linux seems to use less RAM.  [sigh]  I've also considered installing, at least long enough to make all these personal file copies, and then remove.  But I guess I'm afraid of that Linux partition causing troubles when I go to reinstall Windows. 

 

So are you thinking it's a RAM issue when my computer crashes?  Although I have 12 GB of RAM, it does seem to work much more reliably when I copy smaller chunks of data at a time.  But it's so tedious and  s l o w. 

 

But anyway, as I mentioned above, Puppy is useless to me unless I can somehow make it see my RAID array, since that's the drive I'm copying from.

 

As for installing to a USB stick rather than using it live, what would that accomplish?  Or putting it another way, I'm curious to understand why that might work better.



#9 Captain Dunsel

Captain Dunsel
  • Topic Starter

  • Members
  • 46 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Maryland, USA
  • Local time:07:18 AM

Posted 15 January 2016 - 11:19 PM

 

 

I've been copying my personal files from my internal hard drive to an external one

 

How is the external drive formatted?

 

 

I'm virtually certain the external drive uses the NTFS file system, if that's what you mean.  But I just looked at its properties and that file system field is blank, so I'm not sure.  I've seen this somewhere, but can't recall where -- maybe on bootup.  If you tell me how to find out I'll do that.



#10 Captain Dunsel

Captain Dunsel
  • Topic Starter

  • Members
  • 46 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Maryland, USA
  • Local time:07:18 AM

Posted 15 January 2016 - 11:32 PM

I was going to post an image of the error screen I got recently, but I just realized I don't know how to attach an image!  :scratchhead:  I thought sure that was possible.  I see an image icon in the compose-window toolbar but it seems only a URL, not a file path, works there.

 

What am I missing?

 

 



#11 Al1000

Al1000

  • Global Moderator
  • 7,721 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Scotland
  • Local time:12:18 PM

Posted 16 January 2016 - 01:59 AM

As for installing to a USB stick rather than using it live, what would that accomplish? Or putting it another way, I'm curious to understand why that might work better.


Because my point is that:

... installed Linux seems to use less RAM.


But I guess I'm afraid of that Linux partition causing troubles when I go to reinstall Windows.


I'm suggesting installing it to a USB stick, not to an internal drive.

#12 Al1000

Al1000

  • Global Moderator
  • 7,721 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Scotland
  • Local time:12:18 PM

Posted 16 January 2016 - 02:01 AM

I was going to post an image of the error screen I got recently, but I just realized I don't know how to attach an image!


There is no facility for attaching images in this forum; you would have to use an image hosting site.

#13 Captain Dunsel

Captain Dunsel
  • Topic Starter

  • Members
  • 46 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Maryland, USA
  • Local time:07:18 AM

Posted 16 January 2016 - 10:36 PM

Because my point is that:



... installed Linux seems to use less RAM.


OK, point taken.  So you're thinking it's a RAM issue causing the crashes?  Me too, sort of, but see below...

 

 


I'm suggesting installing it to a USB stick, not to an internal drive.

 

 

 

Oh, yeah.  I knew that when I first read your post, but I somehow forgot it when I got around to replying.  :blush:

 

You know, I'm mentally stuck on why this should be necessary, but I'm willing to consider it.  It just seems to me like the live Linux would be "smart" enough to continuously flush the memory as the copies are proceeding, rather than let it build up to the point of bringing down the computer!!  :angry:  We're talking about "copy", a very basic operation; it's not like I'm editing videos or running a server.  But I'm not a programmer; perhaps it's harder than it seems.

 

I'll look for instructions for installing to a flash drive.  (And if you know of any you could point me to, I'd appreciate it.)  I'll see if that looks like something I want to try.

 

Thanks



#14 Captain Dunsel

Captain Dunsel
  • Topic Starter

  • Members
  • 46 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Maryland, USA
  • Local time:07:18 AM

Posted 16 January 2016 - 10:43 PM

There is no facility for attaching images in this forum; you would have to use an image hosting site.

 

Hmm, ok.  That's surprising, as it seems like that would be a very useful feature.  But of course I understand lots of images would take up a lot of server space.  At least I know I'm not going blind. Thanks



#15 pcpunk

pcpunk

  • Members
  • 5,865 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Florida
  • Local time:07:18 AM

Posted 17 January 2016 - 06:22 PM

I've been copying my personal files from my internal hard drive to an external one

Captain, am I missing something here?  Why would you need to do this with LinuxLiveEnviroment?  Can't you just Copy it to the other drive directly?


Edited by pcpunk, 17 January 2016 - 09:56 PM.

sBCcBvM.png

Created by Mike_Walsh

 

KDE, Ruler of all Distro's

eps2.4_m4ster-s1ave.aes_pcpunk_leavemehere

 





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users