Cloudera CDH：初步使用，上传文件到HDFS（shakespeare.txt）

Posted on: 2017年9月10日 2021年4月9日
Categories: Hadoop
Tags: CDH, Cloudera, Hadoop, HDFS, shakespeare, upload, 大数据, 莎士比亚

查看集群的运行情况：

elephant Node：

[root@elephant ~]# chkconfig | grep --color cloudera

Note: This output shows SysV services only and does not include native
      systemd services. SysV configuration data might be overridden by native
      systemd configuration.

      If you want to list systemd services use 'systemctl list-unit-files'.
      To see services enabled on particular target use
      'systemctl list-dependencies [target]'.

cloudera-scm-agent      0:off   1:off   2:off   3:on    4:on    5:on    6:off
[root@elephant ~]# 
[root@elephant ~]# service cloudera-scm-agent status
● cloudera-scm-agent.service - LSB: Cloudera SCM Agent
   Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-agent; bad; vendor preset: disabled)
   Active: active (exited) since Sat 2017-09-09 15:45:31 CST; 1 day 1h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 7473 ExecStart=/etc/rc.d/init.d/cloudera-scm-agent start (code=exited, status=0/SUCCESS)

Sep 09 15:45:28 elephant systemd[1]: Starting LSB: Cloudera SCM Agent...
Sep 09 15:45:28 elephant su[7490]: (to root) root on none
Sep 09 15:45:31 elephant cloudera-scm-agent[7473]: Starting cloudera-scm-agent: [  OK  ]
Sep 09 15:45:31 elephant systemd[1]: Started LSB: Cloudera SCM Agent.
[root@elephant ~]# 
[root@elephant ~]# jps
19080 DataNode
19082 NameNode
4092 Jps
21132 NodeManager
[root@elephant ~]# 
[root@elephant ~]# ps -ef | grep --color NAMENODE
root      4121  3990  0 17:30 pts/0    00:00:00 grep --color=auto --color NAMENODE
hdfs     19082  7713  1 01:17 ?        00:13:38 /usr/java/jdk1.8.0_144/bin/java -Dproc_namenode -Xmx1000m -Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop-hdfs -Dhadoop.log.file=hadoop-cmf-hdfs-NAMENODE-elephant.log.out -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xms1008730112 -Xmx1008730112 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/hdfs_hdfs-NAMENODE-2fa0da2f21e3ef0ca5de3412da39a122_pid19082.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode
[root@elephant ~]#

[root@elephant ~]# chkconfig | grep --color cloudera

Note: This output shows SysV services only and does not include native

systemd services. SysV configuration data might be overridden by native

systemd configuration.

If you want to list systemd services use 'systemctl list-unit-files'.

To see services enabled on particular target use

'systemctl list-dependencies [target]'.

cloudera-scm-agent 0:off 1:off 2:off 3:on 4:on 5:on 6:off

[root@elephant ~]#

[root@elephant ~]# service cloudera-scm-agent status

● cloudera-scm-agent.service - LSB: Cloudera SCM Agent

Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-agent; bad; vendor preset: disabled)

Active: active (exited) since Sat 2017-09-09 15:45:31 CST; 1 day 1h ago

Docs: man:systemd-sysv-generator(8)

Process: 7473 ExecStart=/etc/rc.d/init.d/cloudera-scm-agent start (code=exited, status=0/SUCCESS)

Sep 09 15:45:28 elephant systemd[1]: Starting LSB: Cloudera SCM Agent...

Sep 09 15:45:28 elephant su[7490]: (to root) root on none

Sep 09 15:45:31 elephant cloudera-scm-agent[7473]: Starting cloudera-scm-agent: [ OK ]

Sep 09 15:45:31 elephant systemd[1]: Started LSB: Cloudera SCM Agent.

[root@elephant ~]#

[root@elephant ~]# jps

19080 DataNode

19082 NameNode

4092 Jps

21132 NodeManager

[root@elephant ~]#

[root@elephant ~]# ps -ef | grep --color NAMENODE

root 4121 3990 0 17:30 pts/0 00:00:00 grep --color=auto --color NAMENODE

hdfs 19082 7713 1 01:17 ? 00:13:38 /usr/java/jdk1.8.0_144/bin/java -Dproc_namenode -Xmx1000m -Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop-hdfs -Dhadoop.log.file=hadoop-cmf-hdfs-NAMENODE-elephant.log.out -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xms1008730112 -Xmx1008730112 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/hdfs_hdfs-NAMENODE-2fa0da2f21e3ef0ca5de3412da39a122_pid19082.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode

[root@elephant ~]#

解压样例数据，并上传HDFS

[root@elephant ~]# cd /home/training/training_materials/admin/data/
[root@elephant data]# ls -ltr | grep shakespeare
-rw-r--r-- 1 training training  2017653 Sep  9 10:46 shakespeare.txt.gz
[root@elephant data]# 
[root@elephant data]# gunzip shakespeare.txt.gz 
[root@elephant data]# 
[root@elephant data]# hdfs dfs -put shakespeare.txt /tmp
[root@elephant data]# 

[root@elephant data]# hdfs dfs -ls /tmp
Found 3 items
drwxrwxrwx   - hdfs   supergroup          0 2017-09-10 17:33 /tmp/.cloudera_health_monitoring_canary_files
drwxrwxrwt   - mapred hadoop              0 2017-09-10 01:19 /tmp/logs
-rw-r--r--   3 root   supergroup    5447165 2017-09-10 17:32 /tmp/shakespeare.txt
[root@elephant data]#