Jump to content


 


Register a free account to unlock additional features at BleepingComputer.com
Welcome to BleepingComputer, a free community where people like yourself come together to discuss and learn how to use their computers. Using the site is easy and fun. As a guest, you can browse and view the various discussions in the forums, but can not create a new topic or reply to an existing one unless you are logged in. Other benefits of registering an account are subscribing to topics and forums, creating a blog, and having no ads shown anywhere on the site.


Click here to Register a free account now! or read our Welcome Guide to learn how to use this site.

Photo

is there a program that does this? (backup and cross checking)


  • Please log in to reply
6 replies to this topic

#1 IPT

IPT

  • Members
  • 99 posts
  • OFFLINE
  •  
  • Local time:07:39 PM

Posted 11 April 2015 - 02:24 PM

Oh boy. i have roughly 75000 photo images saved on external hardrives. Due to my lack of diligence I have managed to edit some on one drive but not the other, added some files to some folders, deleted others, etc.  The two external drives have similar file structures but as mentioned some contain or don't contain all the same files or even the same folders.

 

My current wish is too get two 5 TB drives and populate these two drives with mirror images of the photos and data. I want the actual files copied. So in essence i could just grab either drive and have all the images open on any system. 

 

So is there software (free would be ideal but it is what it is) that would:

 

1 -  cross check current drives and add all files to a master drive without redundancy (ie - have master drive A (new clean drive) and copy drive B to A. Then have it copy drive C to A also but only add files that are different or not there?). 

 

2 - have it manually or automatically cross check the two new drives and update any new additions or changes to current files/folders. So in essence i work off only one drive and it maintains a mirror image on another drive that i can just grab with actual true files on it to run off another computer?

 

I realize it'll take hours if not days for a program to sort this out but once I got it all organized it should be maintained easily. I never should have let it get this messed up but alas I did :).


Edited by IPT, 11 April 2015 - 02:26 PM.


BC AdBot (Login to Remove)

 


#2 SpywareDoc

SpywareDoc

  • Members
  • 676 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Maryland, USA
  • Local time:11:39 PM

Posted 11 April 2015 - 02:50 PM

Microsoft's SyncToy 2.1.



#3 IPT

IPT
  • Topic Starter

  • Members
  • 99 posts
  • OFFLINE
  •  
  • Local time:07:39 PM

Posted 12 April 2015 - 12:46 AM

Thanks. that looks perfect. 

 

So it says this:

 

  • Synchronize: New and updated files are copied both ways. Renames and deletes in one folder is repeated on the other.

  • Echo: New and updated files are copied left to right. Renames and deletes on the left are repeated on the right.

So I had two drives and each might have either a folder the other doesn't have, or some files were deleted or modified vs the same folder on the other drive. On my very first attempt I would use synchronize? Would that then provide me with a compete "master" drive? Then I would make that the Left folder and do all my editing and work there using "echo" to the back up drive? Am I understanding that correctly or do I have to manually synch the very first time?


Edited by IPT, 12 April 2015 - 12:46 AM.


#4 Guest_hollowface_*

Guest_hollowface_*

  • Guests
  • OFFLINE
  •  

Posted 12 April 2015 - 02:08 AM

My understanding is you have photos stored on 2 drives, and you wish to duplicate all of it onto another drive. If you can't find a program you love, consider creating a batch script that runs robocopy commands. Robocopy comes installed on Windows. For example I use batch script to backup data from my hard drives to my backup drive, but each time I run the script only new files are copied to the backup drive, rather than re-copying everything. You can even make the script output it's results to a file so that you can see if there were errors. I don't have as many pictures as you, but I've got more than 10,000 (low resolution ones from the internet) on one of my drives. To learn more about robocopy open your terminal and type "robocopy /?". This will display information about the various options it has.

 

You may also be interested in a file/folder syncronization tool called Synkron (there is also a portable version available from PortableApps.com).



#5 IPT

IPT
  • Topic Starter

  • Members
  • 99 posts
  • OFFLINE
  •  

Posted 12 April 2015 - 11:29 AM

thanks. Yes, ultimately I'll be adding and modifying photos on one drive. That could be adding (most often) or removing unwanted images. I will also be modifying some existing files. So the program would ideally recognize files that were modified,  deleted or added.

 

The bigger challenge is going to be merging the 2-3 HD's I have now. Making sure all the files are accounted for on the newly established master drive. Also making sure the files are the most recent ones (in case I edited an image on one drive but not the same image on the other).



#6 Guest_hollowface_*

Guest_hollowface_*

  • Guests
  • OFFLINE
  •  

Posted 12 April 2015 - 09:04 PM

Based on your comments, my understanding is that you have data on 2 drives, and you wish to copy all of it to a third drive (to be a master copy). Then you plan to delete the contents of the 2 original drives so that you can use 1 as a working drive. Then you plan to copy the master drive contents to the working drive. From then on the working drive is where new files will be saved and old files will be edited. The working drive will have its contents copied to the master drive (which has not been changed) on a regular basis for backup purposes, but rather than re-copy everything you just want new and modified files to be copied since the other stuff should already be on the master drive. Do I have that correct?

If you decide to go the batch script route, I've modified my backup batch script to make some examples that you can adapt. Make sure not to confuse the drive letters or you could mess all your data up :P. I've only tested this on Windows 8. Not sure what version of Windows you have, but if it isn't Windows 8 DO NOT USE THESE SCRIPTS WITHOUT VERIFYING THAT ROBOCOPY HAS THE OPTIONS USED. To run a batch script right-click and choose "run as administrator". When using "/COPYALL" you need admin rights, but if you use "/COPY:" instead you won't always need to be an admin depending on which attributes you choose to copy. Do some research into robocopy, and it's options. Test it out a bit too. It's important to understand the commands you're running in case I've misunderstood what you want, or god-forbid made an error.

 

Steps:
1. To get the contents of drive1 and drive2 to drive3 (master) you can adapt this below example. Paste the code below into Notepad, edit as needed, then save as "drive1anddrive2todrive3.bat" with file type as "all files".

robocopy "A:\" "X:\" "*" /E /Z /COPYALL /DCOPY:DT /XD "A:\System Volume Information" /XF "X:\log.txt" /SL /MT:8 /R:4 /FP > "X:\log.txt"
robocopy "B:\" "X:\" "*" /E /Z /COPYALL /DCOPY:DT /XD "B:\System Volume Information" /XF "X:\log2.txt" /SL /MT:8 /R:4 /FP > "X:\log2.txt"
attrib -S -H "X:\" /L

- "A:\" is the source you are copying from. Substitute it for the letter of drive1.
- "B:\" is the other source you are copying from. Substitute it for the letter of drive2.
- "X:\" is the destination. Substitute it for the letter of drive3 (master).
- "attrib -S -H "X:\" /L" is used to remove hidden and system file attributes from the destination as robocopy sometimes wrongly adds them to the destination which would result in the folder being invisible unless you have your system setup to display files/folders with those attributes. Make sure to subsitute "X:\" for the destination drive.
- Please make sure you don't have any of the files you're copying open while when running the script, because Robocopy won't copy them.
- The "> "X:\log.txt" " part is to pipe the terminal output to a file for logging purposes. You can open the file in notepad to see if there were any errors, and what was copied.

2. To clone the contents of drive3 (master) to drive1 (working) you can adapt this example below. Any content found on drive1 (working) that isn't on drive3 (master) will be deleted to ensure they are identical. Paste the code below into Notepad, edit as needed, then save as "mastertoworking.bat" with file type as "all files". Before running the script, delete the log.txt and log2.txt files from drive3 (master).

robocopy "X:\" "A:\" "*" /E /Z /COPYALL /DCOPY:DT /PURGE /XD "X:\System Volume Information" /XF "A:\log.txt" /SL /MT:8 /R:4 /FP > "A:\log.txt"
attrib -S -H "A:\" /L

- "X:\" is the location of the data you want to copy data from.
- "A:\" is the location you are copying data to.
- "*" After the source and desitination is normally where you specify which files to copy. In this case an asterix is used because it is a wildcard, using it like this will result in all files being copied, which is typically what you want.
- "/E" means copy directories and their subdirectories including empty ones.
- "/Z" In the event that a file is only partially copied the use of /Z allows it to be restarted from it's current progress point rather than re-copying the whole file.
- "/COPYALL" tells Robocopy to copy all file attributes. You could use "/COPY:" instead if you wish to manually specify what to copy. For example, I use "/COPY:DT".
- "/DCOPY:" same thing as "/COPY" except it's for directories.
- "/PURGE" will delete things from the destination that don't exist in the source. In this case anything on drive1 (working) that isn't on drive3 (master).
- "/SL" will make sure that symbolic links are copied rather than what they point to.
- "/MT:8" is for multithreading, in this case it specifies to use 8 threads. Using more threads than your computer can handle will result in poor performance.
- "/R:4" tells robocopy how many times to retry a failed copy attempt. In this case it will be 4 times.
- "/FP" is used to log the full path.
- "/XD" is used to exclude a directory from the source. In this case "/XD "X:\System Volume Information" " which is not something you want to copy as it's a hidden system directory specific to the partition it's on.
- "/XF" is used to exclude files. In this case we're excluding a log file so it so that it won't be purged accidentally.
- The "attrib" line is used to fix permissions on the destination as sometimes robocopy flags the destination as a system folder (which makes it disappear). Make sure to substitute "A:\" for the letter of drive1 (working).
- Please make sure you don't have any of the files you're copying open while when running the script, because Robocopy won't copy them.

3. Okay, now that you have all your data stored on drive1 (working), and duplicated to drive3 (master) you can delete the contents of drive2. The fastest way to delete the contents of a drive partition is to re-format the partition. To do this just right-click on the drive in the computer area, and choose "format" from the right-click menu. The choose your format settings (Ex: NTFS, default allocation size, quick-format). You can also delete the 2 batch scripts you've made thus far.

4. To clone the contents of drive1 (working) to drive3 (master) you can adapt this example below. Any content found on drive3 (master) that isn't on drive1 (working) will be deleted to ensure they are identical. Paste the code below into Notepad, edit as needed, then save as "syncronize.bat" with file type as "all files". Delete log.txt from drive1 (working), before running the script.

robocopy "A:\" "X:\" "*" /E /Z /COPYALL /DCOPY:DT /PURGE /XD "A:\System Volume Information" /XF "X:\log.txt" /SL /MT:8 /R:4 /FP > "X:\log.txt"
attrib -S -H "X:\" /L

You can re-use the "syncronize.bat" again when-ever you want to copy changed and new files from drive1 (working) to drive3 (master). Remember, files that aren't found on drive1 (working) will be deleted from drive3 (master) so that drive3 (master) has the same files/folders that drive1 (working) does. For example if you edit a file, and re-run the script the original file will be deleted from drive3 (master), and the modified one from drive1 (working) will be copied to drive3 (master). You DO NOT need to delete the log.txt from drive3 (master), it will be overwritten each time you run the script. It is safe to delete though it you prefer not to have it lingering. It's also worth pointing out that you don't have to save the log to drive3 (master). You can save it anywhere on your computer you want. If you do change the path you may need to remove, or change the "/XF" option in your script though.

I'd advise you to copy some files to a few test folders on your computer, and try using the script on them so that you can see first hand what the script will do under various circumstances. That way if it does something unexpected it happens to your test files, rather than your important files.

Below is an example of how a log file will look (hopefully when I post it it doesn't get squished?):
-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows                              
-------------------------------------------------------------------------------

  Started : April 12, 2015 6:26:48 PM
   Source : C:\Users\me\Desktop\drive1\
     Dest : C:\Users\me\Desktop\drive3\

    Files : *
        
Exc Files : C:\Users\me\Desktop\drive3\log.txt
        
 Exc Dirs : C:\Users\me\Desktop\drive1\System Volume Information
        
  Options : /FP /S /E /DCOPY:DT /COPY:DATSOU /PURGE /Z /MT:8 /R:4 /W:30

------------------------------------------------------------------------------


------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         4         4         0         0         0         0
   Files :         9         0         9         0         0         1
   Bytes :         0         0         0         0         0         0
   Times :   0:00:00   0:00:00                       0:00:00   0:00:00
   Ended : April 12, 2015 6:26:48 PM

As you can see from the log 9 files were detected, but because they already exist in the destination (and have not been modified) they were skipped. Now I'll
modify one of the files, and post the log again (hopefully it doesn't get squished).

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows                              
-------------------------------------------------------------------------------

  Started : April 12, 2015 6:31:13 PM
   Source : C:\Users\me\Desktop\drive1\
     Dest : C:\Users\me\Desktop\drive3\

    Files : *
        
Exc Files : C:\Users\me\Desktop\drive3\log.txt
        
 Exc Dirs : C:\Users\me\Desktop\drive1\System Volume Information
        
  Options : /FP /S /E /DCOPY:DT /COPY:DATSOU /PURGE /Z /MT:8 /R:4 /W:30

------------------------------------------------------------------------------

        Newer                   26    C:\Users\me\Desktop\drive1\test2\test5.txt
100%  

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         4         4         0         0         0         0
   Files :         9         1         8         0         0         1
   Bytes :        26        26         0         0         0         0
   Times :   0:00:00   0:00:00                       0:00:00   0:00:00
   Ended : April 12, 2015 6:31:13 PM

As you can see the file I modified was copied, but the other 8 were skipped because they hadn't been changed.

 

Anyways, if you decide to go the robocopy + batchscript route, I hope this helps. :)



#7 IPT

IPT
  • Topic Starter

  • Members
  • 99 posts
  • OFFLINE
  •  
  • Local time:07:39 PM

Posted 15 April 2015 - 12:05 AM

Holy cow, thanks! Have Windows 7 right now but might need to upgrade to use this :). Yes, you have it correct what I would like to do. 






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users