BleepingComputer.com: Remove Duplicates Macro

Welcome to BleepingComputer, a free community where people like yourself come together to discuss and learn how to use their computers. Using the site is easy and fun. As a guest, you can browse and view the various discussions in the forums, but can not create a new topic or reply to an existing one unless you are logged in. Other benefits of registering an account are subscribing to topics and forums, creating a blog, and having no ads shown anywhere on the site.

Click here to Register a free account now! or read our Welcome Guide to learn how to use this site.

Page 1 of 1

You cannot start a new topic
You cannot reply to this topic

Remove Duplicates Macro

#1 Illegal Alien

Member

Group: Members
Posts: 33
Joined: 03-August 08

Posted 03 August 2008 - 09:41 AM

I think this is the write forum for this question. . .

I'm looking for a macro or plugin that will remove duplicate lines from a file. It has to find only duplicate lines, not words or strings. Anyone know where I can find such a macro, or how I can make one? I have MS Word '07 and Notepad++

#2 groovicus

Hail Groovicus!

Group: Moderator
Posts: 9,522
Joined: 05-June 04
Gender:Male
Location:Centerville, SD

Posted 03 August 2008 - 12:26 PM

I found a couple hundred on Google.. any reason why one of those will not work?

"Take the risk of thinking for yourself, much more happiness, truth, beauty, and wisdom will come to you that way" - Christopher Hitchens

#3 Illegal Alien

Member

Group: Members
Posts: 33
Joined: 03-August 08

Posted 03 August 2008 - 01:28 PM

All the ones I've looked at are:
1. For Excel, not Notepad++ or Word
2. For MS Word TABLES not plain text files
3. Don't understand how you use them
4. They don't work, or links are outdated

#4 Illegal Alien

Member

Group: Members
Posts: 33
Joined: 03-August 08

Posted 03 August 2008 - 02:41 PM

What is 'awk' and how do you use it?

#5 groovicus

Hail Groovicus!

Group: Moderator
Posts: 9,522
Joined: 05-June 04
Gender:Male
Location:Centerville, SD

Posted 03 August 2008 - 03:07 PM

You start out with Linux..... perhaps you would be more interested in Gawk?

Would I be correct is assuming that all you want is a GUI that you input a file name, and it deletes the duplicates?

I threw together a simple java application that will do the job. If you have java installed on your system (and most people do), all you have to do is double-click on the file to run it. It will open a file browser. Select the file that you want to remove duplicates from, and select ok. Of course, make a backup copy of the file first, just in case something goes wrong.

Disclaimer and source code can be found here.

Attached File(s)

rmdup.jar (2.6K)
Number of downloads: 158

"Take the risk of thinking for yourself, much more happiness, truth, beauty, and wisdom will come to you that way" - Christopher Hitchens

#6 Illegal Alien

Member

Group: Members
Posts: 33
Joined: 03-August 08

Posted 04 August 2008 - 09:03 AM

Cool, thanks! CLI doesn't bother me. I know how to make batch files and use AutoHotkey to make scripts (simple, like AutoIt3)

Script worked like a charm on a 25,000 line document. I used it on my "drivers\etc\host" file, basically to update it with a newer list since replacing it would delete some of my own entries. Also used it to weed out all the dupes in a 50,000 line word list. So now you know it works =]

This post has been edited by Illegal Alien: 04 August 2008 - 09:16 AM