Dumbo is a python module for written and running the map-reduce program in the hadoop Cluster. it is working only for the unix/linux environment. it contain the two kind of function, one is mapper and another one is reducer and finally it has one run method. which is calling the mapper function,reducer and combiner function(optional) and it was submit the map-reduce program in the hadoop Cluster.
how to install dumbo?
On the machine from which you want to run your Dumbo programs, do:
$ wget -O ez_setup.py http://bit.ly/ezsetup
$ python ez_setup.py -z dumbo
or, if you already have easy_install installed, simply:
$ easy_install -z dumbo
$ wget -O ez_setup.py http://bit.ly/ezsetup
$ python ez_setup.py -z dumbo
or, if you already have easy_install installed, simply:
$ easy_install -z dumbo
simple wordcount program in dumbo?.
'''
Created on 17-Aug-2014
@author: srinivasan
'''
def mapper(key, value):
for word in value.split():
yield word, 1
def reducer(key, values):
yield key, sum(values)
if __name__ == "__main__":
import dumbo
dumbo.run(mapper, reducer)
Created on 17-Aug-2014
@author: srinivasan
'''
def mapper(key, value):
for word in value.split():
yield word, 1
def reducer(key, values):
yield key, sum(values)
if __name__ == "__main__":
import dumbo
dumbo.run(mapper, reducer)
How to submit the program in the hadoop Cluster?
dumbo start WordCount.py -hadoop /usr/hadoop -input /user/root/capacity-scheduler.xml -output /user/hadoopresult
note: it will store the output in the sequence file format.
No comments:
Post a Comment