I tried multiple ways as per my code knowledge. But not able to extract the required protocols (HTTP, MODBUS, SNMP, FTP, Bacnet).
Header: timestamp, IP, IP Country, Request
Input Raw
2019-08-27 12:11:08,078 TFTP Serving File System at /data/tftp/ in vfs. TFTP data_fs sub directory: /tftp
2019-08-27 12:11:08,081 Found and enabled tftp protocol.
2019-08-27 12:11:08,081 No proxy template found. Service will remain unconfigured/stopped.
2019-08-27 12:11:08,082 Modbus server started on: ('0.0.0.0', 5020)
2019-08-27 12:11:08,083 S7Comm server started on: ('0.0.0.0', 10201)
2019-08-27 12:11:08,083 HTTP server started on: ('0.0.0.0', 8800)
2019-08-27 12:11:08,453 SNMP server started on: ('0.0.0.0', 16100)
2019-08-27 12:11:08,455 Bacnet server started on: ('0.0.0.0', 47808)
2019-08-27 12:11:08,456 IPMI server started on: ('0.0.0.0', 6230)
2019-08-27 12:11:08,456 handle server PID [ 1] running on ('0.0.0.0', 44818)
2019-08-27 12:11:08,457 handle server PID [ 1] responding to external done/disable signal in object 140147125794088
2019-08-27 12:11:08,458 FTP server started on: ('0.0.0.0', 2121)
2019-08-27 12:11:08,458 Starting TFTP server at ('0.0.0.0', 6969)
2019-08-27 12:12:36,979 New http session from 202.3.77.166 (ca5a29c6-e107-4b18-99c1-08338aabfbc5)
2019-08-27 12:12:36,981 HTTP/1.1 GET request from ('202.3.77.166', 34142): ('/', [('Host', '67.207.87.192'), ('Connection', 'keep-alive'), ('Upgrade-Insecure-Requests', '1'), ('User-Agent', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36'), ('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3'), ('Accept-Encoding', 'gzip, deflate'), ('Accept-Language', 'en-US,en;q=0.9,bn;q=0.8')], None). ca5a29c6-e107-4b18-99c1-08338aabfbc5
2019-08-27 12:12:36,981 HTTP/1.1 response to ('202.3.77.166', 34142): 302. ca5a29c6-e107-4b18-99c1-08338aabfbc5
2019-08-27 12:12:37,553 HTTP/1.1 GET request from ('202.3.77.166', 34142): ('/index.html', [('Host', '67.207.87.192'), ('Connection', 'keep-alive'), ('Upgrade-Insecure-Requests', '1'), ('User-Agent', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36'), ('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3'), ('Accept-Encoding', 'gzip, deflate'), ('Accept-Language', 'en-US,en;q=0.9,bn;q=0.8')], None). ca5a29c6-e107-4b18-99c1-08338aabfbc5
2019-08-27 12:12:37,553 HTTP/1.1 response to ('202.3.77.166', 34142): 200. ca5a29c6-e107-4b18-99c1-08338aabfbc5
2019-08-27 12:12:38,760 HTTP/1.1 GET request from ('202.3.77.166', 34142): ('/favicon.ico', [('Host', '67.207.87.192'), ('Connection', 'keep-alive'), ('User-Agent', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36'), ('Accept', 'image/webp,image/apng,image/*,*/*;q=0.8'), ('Referer', 'http://67.207.87.192/index.html'), ('Accept-Encoding', 'gzip, deflate'), ('Accept-Language', 'en-US,en;q=0.9,bn;q=0.8'), ('Cookie', 'path=/')], None). ca5a29c6-e107-4b18-99c1-08338aabfbc5
2019-08-27 12:12:38,761 HTTP/1.1 response to ('202.3.77.166', 34142): 404. ca5a29c6-e107-4b18-99c1-08338aabfbc5
2019-08-27 12:13:08,778 Session timed out: ca5a29c6-e107-4b18-99c1-08338aabfbc5
2019-08-27 12:14:17,470 New http session from 165.16.37.187 (d39c9889-d4ab-41f2-b4c4-6d16ce34c830)
2019-08-27 12:14:17,471 HTTP/1.1 GET request from ('165.16.37.187', 56987): ('/', [('Host', '67.207.87.192:80'), ('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'), ('Content-Length', '0')], b''). d39c9889-d4ab-41f2-b4c4-6d16ce34c830
2019-08-27 12:14:17,471 HTTP/1.1 response to ('165.16.37.187', 56987): 302. d39c9889-d4ab-41f2-b4c4-6d16ce34c830
2019-08-27 12:14:47,488 Session timed out: d39c9889-d4ab-41f2-b4c4-6d16ce34c830
2019-08-27 12:15:51,814 New modbus session from 202.3.77.166 (b6f3d1ea-dd32-452f-9dab-ab996dfeca5a)
2019-08-27 12:15:51,814 New Modbus connection from 202.3.77.166:39755. (b6f3d1ea-dd32-452f-9dab-ab996dfeca5a)
2019-08-27 12:15:51,816 Exception caught: Modbus Error: Exception code = 2. (A proper response will be sent to the peer)
2019-08-27 12:15:51,818 Modbus traffic from 202.3.77.166: {'request': b'00010000004f0110000000244861626364656667685aee12a9881234edccffff56781234b85241b2b852c1b241112152ffffffffbeefdead56781234beefdea
I build the code. But the code have too much bug or I used very wrong coding technique. Please help me. It will improve my way of understanding code.
Code:
import pandas as pd
import re
import datetime
from datetime import datetime
http=[]
time=[]
def reader(file):
with open(file, 'r') as f:
for line in f:
line_split=line.split(" ")
Timestamp=line_split[0:2]
Timestamp=str(Timestamp[0])+" "+str(Timestamp[1])
Timestamp=Timestamp.split(",")[0]
main_line=line_split[2:]
#time.append(Timestamp)
print(main_line[1])
if (
main_line[0]=="HTTP/1.1" or main_line[0]=="Modbus" or main_line[0]=="Modbus" or main_line[0]=="Bacnet" and
main_line[1]=="http" or main_line[1]=="modbus" or main_line[1]=="Bacnet" or main_line[1]=="ftp"
):
http.append(Timestamp)
http.append(main_line)
print(main_line,"====>", Timestamp)
continue
database = ({'Timestamp':http[0], 'Request':http})
dataframe = pd.DataFrame(database)
print(dataframe.head())
dataframe.to_csv('task_raw1.csv', index=False)
if __name__ == "__main__":
reader("/content/drive/MyDrive/task2.log")
Output:
Timestamp,Request
2019-08-27 11:58:07,2019-08-27 11:58:07
2019-08-27 11:58:07,"['Conpot', 'modbus', 'initialized\n']"
2019-08-27 11:58:07,2019-08-27 11:58:07
2019-08-27 11:58:07,"['Conpot', 'Bacnet', 'initialized', 'using', 'the', '/home/conpot/.local/lib/python3.6/site-packages/conpot-0.6.0-py3.6.egg/conpot/templates/default/bacnet/bacnet.xml', 'template.\n']"
2019-08-27 11:58:07,2019-08-27 11:58:07
2019-08-27 11:58:07,"['Modbus', 'server', 'started', 'on:', ""('0.0.0.0',"", '5020)\n']"
2019-08-27 11:58:07,2019-08-27 12:05:46
2019-08-27 11:58:07,"['Conpot', 'modbus', 'initialized\n']"
2019-08-27 11:58:07,2019-08-27 12:05:46
2019-08-27 11:58:07,"['Conpot', 'Bacnet', 'initialized', 'using', 'the', '/home/conpot/.local/lib/python3.6/site-packages/conpot-0.6.0-py3.6.egg/conpot/templates/default/bacnet/bacnet.xml', 'template.\n']"
2019-08-27 11:58:07,2019-08-27 12:05:46
2019-08-27 11:58:07,"['Modbus', 'server', 'started', 'on:', ""('0.0.0.0',"", '5020)\n']"
2019-08-27 11:58:07,2019-08-27 12:11:08
2019-08-27 11:58:07,"['Conpot', 'modbus', 'initialized\n']"
2019-08-27 11:58:07,2019-08-27 12:11:08
2019-08-27 11:58:07,"['Conpot', 'Bacnet', 'initialized', 'using', 'the', '/home/conpot/.local/lib/python3.6/site-packages/conpot-0.6.0-py3.6.egg/conpot/templates/default/bacnet/bacnet.xml', 'template.\n']"
2019-08-27 11:58:07,2019-08-27 12:11:08
2019-08-27 11:58:07,"['Modbus', 'server', 'started', 'on:', ""('0.0.0.0',"", '5020)\n']"
2019-08-27 11:58:07,2019-08-27 12:12:36
2019-08-27 11:58:07,"['HTTP/1.1', 'GET', 'request', 'from', ""('202.3.77.166',"", '34142):', ""('/',"", ""[('Host',"", ""'67.207.87.192'),"", ""('Connection',"", ""'keep-alive'),"", ""('Upgrade-Insecure-Requests',"", ""'1'),"", ""('User-Agent',"", ""'Mozilla/5.0"", '(X11;', 'Linux', 'x86_64)', 'AppleWebKit/537.36', '(KHTML,', 'like', 'Gecko)', 'Chrome/76.0.3809.132', ""Safari/537.36'),"", ""('Accept',"", ""'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3'),"", ""('Accept-Encoding',"", ""'gzip,"", ""deflate'),"", ""('Accept-Language',"", ""'en-US,en;q=0.9,bn;q=0.8')],"", 'None).', 'ca5a29c6-e107-4b18-99c1-08338aabfbc5\n']"
2019-08-27 11:58:07,2019-08-27 12:12:36
2019-08-27 11:58:07,"['HTTP/1.1', 'response', 'to', ""('202.3.77.166',"", '34142):', '302.', 'ca5a29c6-e107-4b18-99c1-08338aabfbc5\n']"
2019-08-27 11:58:07,2019-08-27 12:12:37
2019-08-27 11:58:07,"['HTTP/1.1', 'GET', 'request', 'from', ""('202.3.77.166',"", '34142):', ""('/index.html',"", ""[('Host',"", ""'67.207.87.192'),"", ""('Connection',"", ""'keep-alive'),"", ""('Upgrade-Insecure-Requests',"", ""'1'),"", ""('User-Agent',"", ""'Mozilla/5.0"", '(X11;', 'Linux', 'x86_64)', 'AppleWebKit/537.36', '(KHTML,', 'like', 'Gecko)', 'Chrome/76.0.3809.132', ""Safari/537.36'),"", ""('Accept',"", ""'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3'),"", ""('Accept-Encoding',"", ""'gzip,"", ""deflate'),"", ""('Accept-Language',"", ""'en-US,en;q=0.9,bn;q=0.8')],"", 'None).', 'ca5a29c6-e107-4b18-99c1-08338aabfbc5\n']"
2019-08-27 11:58:07,2019-08-27 12:12:37
2019-08-27 11:58:07,"['HTTP/1.1', 'response', 'to', ""('202.3.77.166',"", '34142):', '200.', 'ca5a29c6-e107-4b18-99c1-08338aabfbc5\n']"
Required Output
Timestamp, IP, IP Location, Request
2019-08-27 11:58:07, 202.3.77.166, India, HTTP
2019-08-27 11:58:07, 172.3.77.166, US, MODBUS
2019-08-27 11:58:07, 202.3.77.168, India, HTTP
2019-08-27 11:58:07, 172.13.77.198, India, HTTP