Jump to content


 


Register a free account to unlock additional features at BleepingComputer.com
Welcome to BleepingComputer, a free community where people like yourself come together to discuss and learn how to use their computers. Using the site is easy and fun. As a guest, you can browse and view the various discussions in the forums, but can not create a new topic or reply to an existing one unless you are logged in. Other benefits of registering an account are subscribing to topics and forums, creating a blog, and having no ads shown anywhere on the site.


Click here to Register a free account now! or read our Welcome Guide to learn how to use this site.

Photo

Grabbing IPs from a text file?


  • Please log in to reply
26 replies to this topic

#1 master131

master131

  • Members
  • 366 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Melbourne, Australia
  • Local time:05:11 PM

Posted 02 August 2010 - 04:10 AM

I have a .log file that contains a dump from a program. I need a batch file that can grab the IP Adresses (without the port number) from the file and save it to another file named IP.txt. Also, I need it to be able to delete duplicate IPs from IP.txt.

Here is a 'sample' of the log file

2010-08-01 00:07:57 - Program: INFO: Starting server...
2010-08-01 00:07:57 - IPServer: INFO: Starting IPServer
2010-08-01 00:07:57 - LogServer: INFO: Starting LogServer
2010-08-01 00:07:57 - MatchServer: INFO: Starting MatchServer for playlist 1
2010-08-01 00:07:57 - MatchServer: INFO: Starting MatchServer for playlist 2
2010-08-01 00:07:57 - MatchServer: INFO: Starting MatchServer for playlist 3
2010-08-01 00:07:57 - MatchServer: INFO: Starting MatchServer for playlist 4
2010-08-01 00:07:57 - MatchServer: INFO: Starting MatchServer for playlist 5
2010-08-01 00:07:57 - CSHTTPServer: INFO: Starting HttpHandler
2010-08-01 00:07:59 - HttpHandler: DEBUG: HTTP request for 
2010-08-01 00:08:07 - IPServer: DEBUG: Handling IP request from 011000011E4076D9
2010-08-01 00:08:07 - UdpServer: DEBUG: Received packet at IPServer from 114.76.65.170:28960
2010-08-01 00:08:07 - UdpServer: DEBUG: Received packet at IPServer from 114.76.65.170:28960
2010-08-01 00:08:08 - UdpServer: DEBUG: Received packet at LogServer from 114.76.65.170:63865
2010-08-01 00:08:11 - UdpServer: DEBUG: Received packet at IPServer from 114.76.65.170:28960
2010-08-01 00:08:17 - UdpServer: DEBUG: Received packet at IPServer from 114.76.65.170:28960
2010-08-01 00:08:25 - UdpServer: DEBUG: Received packet at IPServer from 114.76.65.170:28960
2010-08-01 00:08:37 - UdpServer: DEBUG: Received packet at IPServer from 114.76.65.170:28960
2010-08-01 00:08:57 - UdpServer: DEBUG: Received packet at IPServer from 114.76.65.170:28960
2010-08-01 00:09:33 - UdpServer: DEBUG: Received packet at IPServer from 114.76.65.170:28960


BC AdBot (Login to Remove)

 


#2 PropagandaPanda

PropagandaPanda


  • Malware Response Team
  • 10,433 posts
  • OFFLINE
  •  
  • Gender:Male
  • Local time:02:11 AM

Posted 02 August 2010 - 09:31 AM

Hello.

Will the IP addresses only appear in these kinds of lines?

2010-08-01 00:09:33 - UdpServer: DEBUG: Received packet at IPServer from 114.76.65.170:28960

In other words, will it always begin with "????-??-?? ??:??:?? - UdpServer: DEBUG: Received packet at IPServer from"

If so, it can be done from a batch file with some simple string manipulation.

Environment variable substitution has been enhanced as follows:

%PATH:str1=str2%

would expand the PATH environment variable, substituting each occurrence
of "str1" in the expanded result with "str2". "str2" can be the empty
string to effectively delete all occurrences of "str1" from the expanded
output. "str1" can begin with an asterisk, in which case it will match
everything from the beginning of the expanded output to the first
occurrence of the remaining portion of str1.

May also specify substrings for an expansion.

%PATH:~10,5%

would expand the PATH environment variable, and then use only the 5
characters that begin at the 11th (offset 10) character of the expanded
result. If the length is not specified, then it defaults to the
remainder of the variable value. If either number (offset or length) is
negative, then the number used is the length of the environment variable
value added to the offset or length specified.

%PATH:~-10%

would extract the last 10 characters of the PATH variable.

%PATH:~0,-2%

would extract all but the last 2 characters of the PATH variable.


With Regards,
The Panda

#3 master131

master131
  • Topic Starter

  • Members
  • 366 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Melbourne, Australia
  • Local time:05:11 PM

Posted 03 August 2010 - 01:42 AM

Yes they will except I wouldn't know how to put that in a batch file. I haven't been using Command Prompt or writing batch files for a while :thumbsup:

Edited by master131, 03 August 2010 - 01:58 AM.


#4 Budapest

Budapest

    Bleepin' Cynic


  • Moderator
  • 23,579 posts
  • OFFLINE
  •  
  • Gender:Male
  • Local time:04:11 PM

Posted 03 August 2010 - 01:47 AM

You could also achieve this by copy/pasting the data into EXCEL and then doing a "Text to Columns" to get all the IP addresses in one column. Then you could delete all the other data you don't want.
The power of accurate observation is commonly called cynicism by those who haven't got it.

—George Bernard Shaw

#5 master131

master131
  • Topic Starter

  • Members
  • 366 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Melbourne, Australia
  • Local time:05:11 PM

Posted 03 August 2010 - 01:59 AM

I need it to run repetitively. :thumbsup:

#6 housec

housec

  • Members
  • 15 posts
  • OFFLINE
  •  
  • Local time:07:11 AM

Posted 03 August 2010 - 07:55 AM

Also, I need it to be able to delete duplicate IPs from IP.txt.


How big will your log file be? Not a big deal if it's a few Mg, but if it's hundred of Gb, many unique IP's..

As for matching, not had to use a batch file in some time, so at abit of a loss how you would do this. Does it have to be a batch file?
Tend to find something like a python script tends to be perform quite will at this task, regex such as from\s(?<IP>(\d+\.){3}\d+) should do the trick to extract your IP. Read a line at a time, and assuming your file isn't 'too' big, throw IP group result into a hashset. When your done write your IP collection out to file.

IF you can't install python on the servers in question, could maybe use vbscript or javascript, both alot more powerful than a batchfile. They have regex engines and hash based collections for quick lookup of stored IP's.

Edited by housec84, 03 August 2010 - 07:56 AM.


#7 groovicus

groovicus

  • Security Colleague
  • 9,963 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Centerville, SD
  • Local time:12:11 AM

Posted 03 August 2010 - 11:02 AM

So what you are really asking is for someone to write it for you? Is that the message I am getting? :thumbsup:

Edited by groovicus, 03 August 2010 - 11:02 AM.


#8 master131

master131
  • Topic Starter

  • Members
  • 366 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Melbourne, Australia
  • Local time:05:11 PM

Posted 03 August 2010 - 04:47 PM

No, I just need help writing one because I fail at programmng languages :thumbsup:

Edited by master131, 03 August 2010 - 04:52 PM.


#9 Romeo29

Romeo29

    Learning To Bleep


  • Members
  • 3,194 posts
  • OFFLINE
  •  
  • Gender:Not Telling
  • Location:127.0.0.1
  • Local time:01:11 AM

Posted 04 August 2010 - 01:20 AM

What language do you want help in? This is a basic file input task and can be done in all languages.

In batch scripting, you can use the famous for /f loop to extract information from the file. In the command prompt type help for for more information. You will need two nested for loops, first with delims=" " and another with delims = ":". This would extract IP addresses from this log file.

To extract non-repeated IP addresses, you can use findstr command on the file :thumbsup:

I think this information is good enough to get you started. Create a batch script and if you face problems then ask here. Nobody is going to post the full code :flowers:

#10 housec

housec

  • Members
  • 15 posts
  • OFFLINE
  •  
  • Local time:07:11 AM

Posted 04 August 2010 - 10:24 AM

[codebox]import re

f = open('c:\ip.txt', 'r')
s = set()
line = f.readline()
while(line != ''):
m = re.search('\s((\d+\.){3}\d+)', line)
if(m != None):
s.add(m.group(1))
line = f.readline()

f.close()
out = open('c:\out.txt', 'w')

for x in s:
out.write(x + '\n')

print 'Done'
[/codebox]

#11 master131

master131
  • Topic Starter

  • Members
  • 366 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Melbourne, Australia
  • Local time:05:11 PM

Posted 04 August 2010 - 04:55 PM

OK. I've seem to have gotten it now, but how should I go about deleting the duplicates and removing the port number?
This is what I have so far.
for /f "tokens=7 delims=from" %%G IN (NetServer.log) DO @echo %%G>>IP.txt

This is what IP.txt looks like:
ting...
ve
 playlist 1
 playlist 2
 playlist 3
 playlist 4
 playlist 5
 114.76.65.170:28960 ve
 114.76.65.170:63865
changel 
 77.166.101.88:55921
 77.166.101.88:55921
 114.76.65.170:63865
 77.166.101.88:55921
 77.166.101.88:55921
 77.166.101.88:55921
 114.76.65.170:63865
 114.76.65.170:63865
 114.76.65.170:63865
a11 
 89.39.198.36:55463
 89.39.198.36:55463
changel 
 77.166.101.88:56447
 114.76.65.170:63865
 77.166.101.88:56447
 77.18.91.20:28960 ve
 77.18.91.20:53604
ECHO is off.
 77.253.14.201:49899
 77.253.14.201:49899
 77.18.91.20:53604
 77.18.91.20:53604
 77.18.91.20:53604
 89.39.198.36:55463
 77.166.101.88:56628
changel 
 91.63.107.105:52976
 91.63.107.105:52976
 114.76.65.170:63865
 77.18.91.20:53604
 77.18.91.20:53604
 77.18.91.20:53604
 77.18.91.20:53604
 77.18.91.20:53604
 77.18.91.20:53604
 77.18.91.20:53604
 77.18.91.20:53604
 77.18.91.20:53604
 77.18.91.20:53604
 77.18.91.20:53604

Edited by master131, 04 August 2010 - 04:57 PM.


#12 PropagandaPanda

PropagandaPanda


  • Malware Response Team
  • 10,433 posts
  • OFFLINE
  •  
  • Gender:Male
  • Local time:02:11 AM

Posted 04 August 2010 - 05:30 PM

Hello.

Not a bad start. Why not modify the code to process only the lines that begin with ".................... - UdpServer: DEBUG: Received packet at IPServer from " ? You can then use string slicing to extract the IP part of the line.

With Regards,
The Panda

#13 Romeo29

Romeo29

    Learning To Bleep


  • Members
  • 3,194 posts
  • OFFLINE
  •  
  • Gender:Not Telling
  • Location:127.0.0.1
  • Local time:01:11 AM

Posted 04 August 2010 - 06:28 PM

Hint : If you split the line containing the IP address by single space, then you would get IP address as 11th part. So you can just set tokens to 11 and delims to a single space.

#14 master131

master131
  • Topic Starter

  • Members
  • 366 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Melbourne, Australia
  • Local time:05:11 PM

Posted 05 August 2010 - 01:29 AM

Yay, I got it working now.
@echo off
for /f "tokens=11 delims= " %%G IN (NetServer.log) DO @echo %%G>>IP.txt
for /f "tokens=1 delims=:" %%G IN (IP.txt) DO @echo %%G>>IP2.txt
pause

Result:
114.76.65.170
114.76.65.170
version
114.76.65.170
114.76.65.170
114.76.65.170
114.76.65.170
114.76.65.170
114.76.65.170
114.76.65.170
77.166.101.88
version
77.166.101.88
77.166.101.88
77.166.101.88

How can I delete the duplicate IP addresses and filter out the word 'version'. Also, if I wanted this to run in loops because the log is constantly written to, how would I make it continue from the last line it did instead of redoing the whole log which is time consuming?

Edited by master131, 05 August 2010 - 01:29 AM.


#15 Romeo29

Romeo29

    Learning To Bleep


  • Members
  • 3,194 posts
  • OFFLINE
  •  
  • Gender:Not Telling
  • Location:127.0.0.1
  • Local time:01:11 AM

Posted 05 August 2010 - 09:08 AM

use find command and then errorlevel.

Suppose you are storing ip address in store.txt then, check if it contains already an ip address like this

findstr "%%G" store.txt > nul
if errorlevel 1 echo %%G >> store.txt

I don't know why are you getting "version". Your sample data does not have any "version" in it. May be somebody else can shed some light on it.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users