數據采集工具的安裝和使用

發布時間：2020-06-28 11:31:20 來源：網絡閱讀：428 作者：sky9890 欄目：大數據

? ?數據采集是大數據分析全流程的重要環節，典型的數據采集工具包括ETL工具、日志采集工具、數據遷移工具等。

? ?Flume是一個高可用的、高可靠的、分布式的海量日志采集、聚合和傳輸的系統。

1.安裝Flume

下載：http://www.apache.org/dist/flume/

hadoop@dblab:/usr/local$ sudo wget http://www.apache.org/dist/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz

hadoop@dblab:/usr/local$ sudo tar -zxvf apache-flume-1.7.0-bin.tar.gz?

hadoop@dblab:/usr/local$ sudo mv apache-flume-1.7.0-bin ./flume

2.配置環境變量

hadoop@dblab:/usr/local$ sudo vim ~/.bashrc

export FLUME_HOME=/usr/local/flume

export FLUME_CONF_DIR=$FLUME_HOME/conf

export JAVA_HOME=/usr/lib/jvm/default-java

export PATH=$PATH:$HIVE_HOME/bin

hadoop@dblab:/usr/local$ source ~/.bashrc

hadoop@dblab:/usr/local/flume/conf$ mv flume-env.sh.template? flume-env.sh

hadoop@dblab:/usr/local/flume/conf$ sudo vim? ?flume-env.sh??

#在flume-env.sh文件開頭加入如下語句：? ? ? ? ? ? ? ? ? ??

export JAVA_HOME=/usr/lib/jvm/default-java

3.啟動Flume

hadoop@dblab:/usr/local/flume$ cd /usr/local/flume

hadoop@dblab:/usr/local/flume$ ./bin/flume-ng version

錯誤: 找不到或無法加載主類 org.apache.flume.tools.GetJavaProperty

Flume 1.7.0

Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git

Revision: 511d868555dd4d16e6ce4fedc72c2d1454546707

Compiled by bessbd on Wed Oct 12 20:51:10 CEST 2016

From source with checksum 0d21b3ffdc55a07e1d08875872c00523

hadoop@dblab:/usr/local/flume$ cd /usr/local/hbase/conf

hadoop@dblab:/usr/local/hbase/conf$ sudo vim hbase-env.sh? ?

#export HBASE_CLASSPATH=/usr/local/hadoop/conf #注釋該行，即解決上述問題