引 入
日志文件,是我们记录用户行为的重要手段。而对于不同的用户,我们往往又会根据IP来区分,所以统计日志文件中的IP访问,对于数据分析人员和相关运营专员来说,是一件重要的事情,这里,采用python这门语言来完成这个小功能。
一、分析IP格式
这里只讨论ipv4。
分析IP格式思路有许多,这里我只分析其中一种比较容易理解的。
1) 从分析一个从1~255的数字开始
一个1~255的数细分成以下5个分组。
数字分组 | 正则表达式表示 |
1~9 | [1-9] |
10~99 | [1-9][0-9] |
100~199 | 1[0-9]{2} |
200~249 | 2[0-4][0-9] |
250~255 | 25[0-5] |
所以一个1~255的数字可以用正则表达式“[1-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]”来表示
2) 分析IP
IP地址的长度为32位,分为4段,每段8位,用十进制数字表示,每段数字范围为0~255,段与段之间用英文句点“.”隔开。例如:某台计算机IP地址为192.168.1.1。所以IP地址可以用简单表示为:
IP = (1~255.) *3 + (1~255)
相关正则表达式为:
设num(1~255) = [1-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]
得:IP=(num\.){3}num
二、具体代码实现 CountIP.py
#!/usr/bin/env python #-*- coding: utf-8 -*- import re #导入正则表达式模块 import sys #以只读方式打开文件,sys.argv[1]表示的是运行时传入的第二个参数 f = open(sys.argv[1], "r") arr = {} #用字典来存储IP跟访问次数 #num表示1-255之间的字串,\b为单词的词首或词尾锚定 num=‘\\b([1-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\\b‘ lines = f.readlines() #遍历文件的每一行 for line in lines: pattern = re.compile(r‘(‘+num+‘\.){3}‘+num) #python中用“+”来连接字符串 match = pattern.match(line) if match: ip = match.group() if(arr.has_key(ip)): arr[ip] += 1 else: arr[ip]=1 f.close() for key in arr: print key + "--->" + str(arr[key])
三、测试
某天日志文件
[root@bogon ~]# cat 20160215.log 192.168.1.102 - - [15/Feb/2016:07:18:12 -0800] "GET /index.php HTTP/1.1" 200 40 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36" 192.168.1.101 - - [15/Feb/2016:07:18:12 -0800] "GET /index.php HTTP/1.1" 200 40 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36" 192.168.1.101 - - [15/Feb/2016:07:18:12 -0800] "GET /index.php HTTP/1.1" 200 40 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36" 192.168.1.101 - - [15/Feb/2016:07:18:12 -0800] "GET /index.php HTTP/1.1" 200 40 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36" 192.168.1.101 - - [15/Feb/2016:07:18:12 -0800] "GET /index.php HTTP/1.1" 200 40 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36" 192.168.1.102 - - [15/Feb/2016:07:18:12 -0800] "GET /index.php HTTP/1.1" 200 40 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36" 192.168.1.101 - - [15/Feb/2016:07:18:57 -0800] "GET /index.php HTTP/1.1" 200 40 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36" 192.168.1.103 - - [15/Feb/2016:07:34:08 -0800] "GET /index.php HTTP/1.1" 200 7237 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36" 192.168.1.101 - - [15/Feb/2016:07:34:08 -0800] "GET /index.php?=PHPE9568F34-D428-11d2-A769-00AA001ACF42 HTTP/1.1" 200 2547 "http://192.168.1.106/index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36" 192.168.1.101 - - [15/Feb/2016:07:34:08 -0800] "GET /index.php?=PHPE9568F35-D428-11d2-A769-00AA001ACF42 HTTP/1.1" 200 2169 "http://192.168.1.106/index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36" 192.168.1.1015 - - [15/Feb/2016:07:34:08 -0800] "GET /index.php?=PHPE9568F35-D428-11d2-A769-00AA001ACF42 HTTP/1.1" 200 2169 "http://192.168.1.106/index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36" 192.168.262.101 - - [15/Feb/2016:07:34:08 -0800] "GET /index.php?=PHPE9568F35-D428-11d2-A769-00AA001ACF42 HTTP/1.1" 200 2169 "http://192.168.1.106/index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36" 111.168.1.101 - - [15/Feb/2016:07:34:08 -0800] "GET /index.php?=PHPE9568F35-D428-11d2-A769-00AA001ACF42 HTTP/1.1" 200 2169 "http://192.168.1.106/index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36"
运行结果如下
[root@bogon ~]# python CountIP.py 20160215.log 192.168.1.101--->9 192.168.1.102--->2 192.168.1.103--->1 111.168.1.101--->1
结 语
对于写多了shell跟java的我来说,python还是一门很能吸引我眼球的语言,语法比shell丰富,跟linux系统的交互比java方便,让我爱不释手啊。后面,我会用python写越来越多的小功能,小有趣,敬请期待。
附:如对上面描述有疑问,期待与朋友您共同探讨。本人QQ:1084569767
本文出自 “小蚊子的家园” 博客,请务必保留此出处http://timber.blog.51cto.com/7677013/1744458
原文地址:http://timber.blog.51cto.com/7677013/1744458