Sunday, 17 August 2014

Introduction Of Dumbo

        Dumbo is a python module for written and running the map-reduce program in the hadoop Cluster. it is working only for the unix/linux environment.  it contain the two kind of function, one is mapper and another one is reducer and finally it has one run method. which is calling the mapper function,reducer and combiner function(optional) and it was submit the map-reduce program in the hadoop Cluster.

how to install dumbo?
     On the machine from which you want to run your Dumbo programs, do:
        $ wget -O ez_setup.py http://bit.ly/ezsetup
        $ python ez_setup.py -z dumbo

or, if you already have easy_install installed, simply:
        $ easy_install -z dumbo
simple wordcount program in dumbo?.

'''
Created on 17-Aug-2014

@author: srinivasan
'''
def mapper(key, value):
    for word in value.split():
        yield word, 1

def reducer(key, values):
    yield key, sum(values)

if __name__ == "__main__":
    import dumbo
    dumbo.run(mapper, reducer)
  

How to submit the program in the hadoop Cluster?
dumbo start WordCount.py -hadoop /usr/hadoop -input /user/root/capacity-scheduler.xml -output /user/hadoopresult

note: it will store the output in the sequence file format.

No comments:

Post a Comment