代码之家 › 专栏 › 技术社区 › Ratan Kumar

hadoop中的数据包计数(使用Mapreduce)

hping snort packet-capture mapreduce hadoop

3

Ratan Kumar · 技术社区 · 10 年前

事情已经完成:

从以下链接安装Hadoop:

http://www.cloudera.com/content/cloudera/en/documentation/cdh4/v4-2-0/CDH4-Installation-Guide/cdh4ig_topic_4_4.html

已安装Hping3以使用以下方法生成洪水请求:

sudo hping3 -c 10000 -d 120 -S -w 64 -p 8000 --flood --rand-source 192.168.1.12

已安装snort以使用以下方法记录上述请求:

sudo snort -ved -h 192.168.1.0/24 -l .

这将生成日志文件snort.Log.142702131

我可以用它来读

sudo snort -r snort.log.1427021231

其给出以下形式的输出:

=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+

03/22-16:17:14.259633 192.168.1.12:8000->117.247.194.105:46639 TCP TTL:64 TOS:0x0 ID:0 IpLen:20 DgmLen:44 DF A. S序列:0x6EE4A6B确认:0x6DF6015B获胜:0x7210 TcpLen:24 TCP选项(1)=>MSS:1460 =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+

我用过

hdfs dfs -put <localsrc> ... <dst>

将此日志文件复制到HDFS。

现在,我需要帮助:

如何统计日志文件中源IP地址、目标IP地址、端口地址、协议和时间戳的总数。

(我是否必须编写自己的地图缩减程序?或者有一个库来实现这一点。)

我还发现

https://github.com/ssallys/p3

但无法使其运行。查看了JAR文件的内容,但无法运行它。

ratan@lenovo:~/Desktop$ hadoop jar ./p3lite.jar p3.pcap.examples.PacketCount

Exception in thread "main" java.lang.ClassNotFoundException:        nflow.runner.Runner
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
at org.apache.hadoop.util.RunJar.main(RunJar.java:201)

谢谢

1 回复 | 直到 10 年前

1

Davis Broda 10 年前

快速搜索后,这似乎是您可能需要自定义MapReduce作业的内容。

该算法看起来类似于以下伪代码:

Parse the file line by line (or parse every n lines if logs are more than one line long).

in the mapper, use regex to figure out if something is a source IP, destination IP etc.

output these with key value structure of <Type, count> 
    type is the type of text that was matched (ex. source IP)
    count is the number of times it was matched in the record

have reducer sum all of the values from the mappers, and get global totals for each type of information you want

write to file in desired format.