hadoop的wordcount java舉例分析

本篇內容介紹了“hadoop的wordcount java舉例分析”的有關知識，在實際案例的操作過程中，不少人都會遇到這樣的困境，接下來就讓小編帶領大家學習一下如何處理這些情況吧！希望大家仔細閱讀，能夠學有所成！

1.導入hadoop需要用到的包

hadoop-2.4.2/share/hadoop/mapreduce/*.jar
hadoop-2.4.2/share/hadoop/mapreduce/lib/*.jar

hadoop-2.4.2/share/hadoop/common/*.jar
hadoop-2.4.2/share/hadoop/common/lib/*.jar

2.編寫java程序

package demo;

import java.io.IOException;
import java.util.*;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;

public class WordCount {

 public static class Map extends MapReduceBase implements Mapper{
  private final static IntWritable one=new IntWritable(1);
  private Text word=new Text();
  @Override
  public void map(LongWritable key,  Text value,
    OutputCollector output, Reporter reporter)
    throws IOException {
   // TODO Auto-generated method stub
   String line=value.toString();
   StringTokenizer tokenizer=new StringTokenizer(line);
   while (tokenizer.hasMoreTokens()){
    word.set(tokenizer.nextToken());
    output.collect(word,one);
    }
   }
   
  }
 //舊版本
 public static class Reduce extends MapReduceBase implements Reducer{

  @Override
  public void reduce(Text key, Iterator values,
    OutputCollector output, Reporter reporter)
    throws IOException {
   // TODO Auto-generated method stub
    int sum=0;
    while(values.hasNext()){
     sum+=values.next().get();
    }
    output.collect(key, new IntWritable(sum)); 
  }
 }

 
 public static void main(String[] args) throws Exception{
  // TODO Auto-generated method stub
  //System.setProperty("HADOOP_USER_NAME","root");  
  JobConf conf=new JobConf(WordCount.class);
  //conf.set("fs.defaultFS","hdfs://192.168.1.120:9000");
  conf.setJobName("wordcount");
  conf.setOutputKeyClass(Text.class);
  conf.setOutputValueClass(IntWritable.class);

  conf.setMapperClass(Map.class);
  conf.setReducerClass(Reduce.class);
  
  conf.setInputFormat(TextInputFormat.class);
  conf.setOutputFormat(TextOutputFormat.class);
  
  FileInputFormat.setInputPaths(conf,new Path(args[0]));
  FileOutputFormat.setOutputPath(conf,new Path(args[1]));

  JobClient.runJob(conf);
 }

}

3.導出為jar文件

4.上傳到linux系統中。

5.新建input目錄，如果有output目錄，先刪除

6.上傳jar包后，到jar包的目錄下，執行

    hadoop jar WordCount.jar demo.WordCount /input/* /output/

7.如果執行時不帶“/”，會在hadoop目錄中新建/user/root下新建兩個文件夾，會提示

      Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:  /input  Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://master:9000/user/root/input

  只需要在執行的時候帶上“/”就行。

8.獲取分離后的文件

hadoop fs -get /output/* output/

“hadoop的wordcount java舉例分析”的內容就介紹到這里了，感謝大家的閱讀。如果想了解更多行業相關的知識可以關注億速云網站，小編將為大家輸出更多高質量的實用文章！