Jump to content


 


Register a free account to unlock additional features at BleepingComputer.com
Welcome to BleepingComputer, a free community where people like yourself come together to discuss and learn how to use their computers. Using the site is easy and fun. As a guest, you can browse and view the various discussions in the forums, but can not create a new topic or reply to an existing one unless you are logged in. Other benefits of registering an account are subscribing to topics and forums, creating a blog, and having no ads shown anywhere on the site.


Click here to Register a free account now! or read our Welcome Guide to learn how to use this site.

Remove Duplicates Macro


  • Please log in to reply
10 replies to this topic

#1 Guest_Illegal Alien_*

Guest_Illegal Alien_*

  • Guests
  • OFFLINE
  •  

Posted 03 August 2008 - 09:41 AM

I think this is the write forum for this question. . .

I'm looking for a macro or plugin that will remove duplicate lines from a file. It has to find only duplicate lines, not words or strings. Anyone know where I can find such a macro, or how I can make one? I have MS Word '07 and Notepad++

BC AdBot (Login to Remove)

 


#2 groovicus

groovicus

  • Security Colleague
  • 9,963 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Centerville, SD

Posted 03 August 2008 - 12:26 PM

I found a couple hundred on Google.. any reason why one of those will not work?

#3 Guest_Illegal Alien_*

Guest_Illegal Alien_*

  • Guests
  • OFFLINE
  •  

Posted 03 August 2008 - 01:28 PM

All the ones I've looked at are:
1. For Excel, not Notepad++ or Word
2. For MS Word TABLES not plain text files
3. Don't understand how you use them
4. They don't work, or links are outdated

#4 Guest_Illegal Alien_*

Guest_Illegal Alien_*

  • Guests
  • OFFLINE
  •  

Posted 03 August 2008 - 02:41 PM

What is 'awk' and how do you use it?

#5 groovicus

groovicus

  • Security Colleague
  • 9,963 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Centerville, SD
  • Local time:12:19 AM

Posted 03 August 2008 - 03:07 PM

You start out with Linux..... perhaps you would be more interested in Gawk?

Would I be correct is assuming that all you want is a GUI that you input a file name, and it deletes the duplicates?

I threw together a simple java application that will do the job. If you have java installed on your system (and most people do), all you have to do is double-click on the file to run it. It will open a file browser. Select the file that you want to remove duplicates from, and select ok. Of course, make a backup copy of the file first, just in case something goes wrong.

Disclaimer and source code can be found here.

Attached Files



#6 Guest_Illegal Alien_*

Guest_Illegal Alien_*

  • Guests
  • OFFLINE
  •  

Posted 04 August 2008 - 09:03 AM

Cool, thanks! CLI doesn't bother me. I know how to make batch files and use AutoHotkey to make scripts (simple, like AutoIt3)

Script worked like a charm on a 25,000 line document. I used it on my "drivers\etc\host" file, basically to update it with a newer list since replacing it would delete some of my own entries. Also used it to weed out all the dupes in a 50,000 line word list. So now you know it works =]

Edited by Illegal Alien, 04 August 2008 - 09:16 AM.


#7 groovicus

groovicus

  • Security Colleague
  • 9,963 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Centerville, SD

Posted 04 August 2008 - 10:49 AM

Cool! It's always fun to write a program that someone actually finds useful. :thumbsup:

#8 dadaku392

dadaku392

  • Members
  • 1 posts
  • OFFLINE
  •  

Posted 06 November 2009 - 12:52 PM

groovicus: thank you, thank you, thank you!!!

#9 vinstah

vinstah

  • Members
  • 1 posts
  • OFFLINE
  •  
  • Local time:12:19 AM

Posted 17 January 2010 - 09:25 PM

Thanks! :thumbsup:

#10 groovicus

groovicus

  • Security Colleague
  • 9,963 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Centerville, SD

Posted 17 January 2010 - 09:43 PM

Updated link so that it actually worked.

#11 Snookrc

Snookrc

  • Members
  • 1 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Everett, WA
  • Local time:12:19 AM

Posted 26 August 2010 - 03:36 AM

Thank you :thumbsup:




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users