标签：分享图片 cep hdfs cin 输入 count hdf err lsp
1. 用Python编写WordCount程序并提交任务
|
程序 |
WordCount |
|
输入 |
一个包含大量单词的文本文件 |
|
输出 |
文件中每个单词及其出现次数（频数），并按照单词字母顺序排序，每个单词和其频数占一行，单词和频数之间有间隔 |
|
1
2
|
cd /home/hadoop/wcsudo gedit mapper.py |
#!/usr/bin/env python
import sys
for i in stdin:
i = i.strip()
words = i.split()
for word in words:
print ‘%s\t%s‘ % (word,1)
chmod a+x /home/hadoop/mapper.py
cd /home/hadoop/wc sudo gedit reducer.py
#!/usr/bin/env python
from operator import itemgetter
import sys
current_word = None
current_count = 0
word = None
for i in stdin:
i = i.strip()
word, count = i.split(‘\t‘,1)
try:
count = int(count)
except ValueError:
continue
if current_word == word:
current_count += count
else:
if current_word:
print ‘%s\t%s‘ % (current_word, current_count)
current_count = count
current_word = word
if current_word == word:
print ‘%s\t%s‘ % (current_word, current_count)
chmod a+x /home/hadoop/reduce.py
echo "foo foo quux labs foo bar quux" | /home/hadoop/wc/mapper.py echo "foo foo quux labs foo bar quux" | /home/hadoop/wc/mapper.py | sort -k1,1 | /home/hadoop/wc/reducer.p
cd /home/hadoop/wc wget http://www.gutenberg.org/files/5000/5000-8.txt wget http://www.gutenberg.org/cache/epub/20417/pg20417.txt cd /usr/hadoop/wc hdfs dfs -put /home/hadoop/hadoop/gutenberg/*.txt /user/hadoop/input
2. 用mapreduce 处理气象数据集
编写程序求每日最高最低气温，区间最高最低气温
标签：分享图片 cep hdfs cin 输入 count hdf err lsp
原文地址：https://www.cnblogs.com/sunset-Panda/p/9021927.html
