HadoopCategorySubscribe
CDH Lab：Exploring Hadoop Configuration Settings
节点：Elephant
查看Hadoop的进程：
YUM：tree
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
[root@elephant process]# yum install -y tree Loaded plugins: fastestmirror, langpacks ISO | 3.6 kB 00:00:00 base | 3.6 kB 00:00:00 cloudera-cm | 2.9 kB 00:00:00 cloudera-manager | 951 B 00:00:00 extras | 3.4 kB 00:00:00 updates | 3.4 kB 00:00:00 Loading mirror speeds from cached hostfile * base: mirrors.cn99.com * extras: mirrors.163.com * updates: mirrors.163.com Resolving Dependencies --> Running transaction check ---> Package tree.x86_64 0:1.6.0-10.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================================================== Package Arch Version Repository Size ======================================================================================================================================================================== Installing: tree x86_64 1.6.0-10.el7 ISO 46 k Transaction Summary ======================================================================================================================================================================== Install 1 Package Total download size: 46 k Installed size: 87 k Downloading packages: tree-1.6.0-10.el7.x86_64.rpm | 46 kB 00:00:02 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : tree-1.6.0-10.el7.x86_64 1/1 Verifying : tree-1.6.0-10.el7.x86_64 1/1 Installed: tree.x86_64 0:1.6.0-10.el7 Complete! [root@elephant process]# |
查看包含了Hadoop配置的目录的目录结构：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 |
[root@elephant ~]# cd /var/run/cloudera-scm-agent/process/ [root@elephant process]# [root@elephant process]# ls -ltr total 0 drwxr-x--x 3 hdfs hdfs 520 Sep 12 01:02 23-hdfs-NAMENODE drwxr-x--x 3 hdfs hdfs 400 Sep 12 01:02 21-hdfs-DATANODE drwxr-x--x 3 yarn hadoop 520 Sep 12 01:28 45-yarn-NODEMANAGER drwxr-x--x 3 yarn hadoop 500 Sep 12 01:28 63-yarn-NODEMANAGER drwxr-xr-x 4 root root 100 Sep 12 01:29 ccdeploy_hadoop-conf_etchadoopconf.cloudera.yarn_6856422516434392646 drwxr-xr-x 4 root root 100 Sep 12 01:29 ccdeploy_hadoop-conf_etchadoopconf.cloudera.hdfs_-5020453740185475409 [root@elephant process]# [root@elephant process]# tree . ├── 21-hdfs-DATANODE │ ├── cloudera-monitor.properties │ ├── cloudera-stack-monitor.properties │ ├── config.zip │ ├── core-site.xml │ ├── event-filter-rules.json │ ├── hadoop-metrics2.properties │ ├── hadoop-policy.xml │ ├── hdfs.keytab │ ├── hdfs-site-refreshable.xml │ ├── hdfs-site.xml │ ├── http-auth-signature-secret │ ├── log4j.properties │ ├── logs │ │ ├── stderr.log │ │ └── stdout.log │ ├── proc.json │ ├── redaction-rules.json │ ├── ssl-client.xml │ ├── ssl-server.xml │ └── supervisor.conf ├── 23-hdfs-NAMENODE │ ├── cloudera_manager_agent_fencer.py │ ├── cloudera_manager_agent_fencer_secret_key.txt │ ├── cloudera-monitor.properties │ ├── cloudera-stack-monitor.properties │ ├── config.zip │ ├── core-site.xml │ ├── dfs_hosts_allow.txt │ ├── dfs_hosts_exclude.txt │ ├── event-filter-rules.json │ ├── hadoop-metrics2.properties │ ├── hadoop-policy.xml │ ├── hdfs.keytab │ ├── hdfs-site.xml │ ├── http-auth-signature-secret │ ├── log4j.properties │ ├── logs │ │ ├── stderr.log │ │ └── stdout.log │ ├── navigator.client.properties │ ├── proc.json │ ├── redaction-rules.json │ ├── ssl-client.xml │ ├── ssl-server.xml │ ├── supervisor.conf │ ├── topology.map │ └── topology.py ├── 45-yarn-NODEMANAGER │ ├── cloudera-monitor.properties │ ├── cloudera-stack-monitor.properties │ ├── config.zip │ ├── container-executor.cfg │ ├── core-site.xml │ ├── event-filter-rules.json │ ├── hadoop-metrics2.properties │ ├── hadoop-policy.xml │ ├── hdfs-site.xml │ ├── http-auth-signature-secret │ ├── libleveldbjni-64-1-1328973290730439446.8 │ ├── log4j.properties │ ├── logs │ │ ├── stderr.log │ │ └── stdout.log │ ├── mapred-site.xml │ ├── process_timestamp │ ├── proc.json │ ├── redaction-rules.json │ ├── ssl-client.xml │ ├── ssl-server.xml │ ├── supervisor.conf │ ├── topology.map │ ├── topology.py │ ├── yarn.keytab │ └── yarn-site.xml ├── 63-yarn-NODEMANAGER │ ├── cloudera-monitor.properties │ ├── cloudera-stack-monitor.properties │ ├── config.zip │ ├── container-executor.cfg │ ├── core-site.xml │ ├── event-filter-rules.json │ ├── hadoop-metrics2.properties │ ├── hadoop-policy.xml │ ├── hdfs-site.xml │ ├── http-auth-signature-secret │ ├── libleveldbjni-64-1-6895623587167114991.8 │ ├── log4j.properties │ ├── logs │ │ ├── stderr.log │ │ └── stdout.log │ ├── mapred-site.xml │ ├── proc.json │ ├── redaction-rules.json │ ├── ssl-client.xml │ ├── ssl-server.xml │ ├── supervisor.conf │ ├── topology.map │ ├── topology.py │ ├── yarn.keytab │ └── yarn-site.xml ├── ccdeploy_hadoop-conf_etchadoopconf.cloudera.hdfs_-5020453740185475409 │ ├── __cloudera_metadata__ │ ├── hadoop-conf │ │ ├── __cloudera_generation__ │ │ ├── __cloudera_metadata__ │ │ ├── core-site.xml │ │ ├── hadoop-env.sh │ │ ├── hdfs-site.xml │ │ ├── log4j.properties │ │ ├── ssl-client.xml │ │ ├── topology.map │ │ └── topology.py │ └── logs │ ├── stderr.log │ └── stdout.log └── ccdeploy_hadoop-conf_etchadoopconf.cloudera.yarn_6856422516434392646 ├── __cloudera_metadata__ ├── logs │ ├── stderr.log │ └── stdout.log └── yarn-conf ├── __cloudera_generation__ ├── __cloudera_metadata__ ├── container-executor.cfg ├── core-site.xml ├── hadoop-env.sh ├── hdfs-site.xml ├── log4j.properties ├── mapred-site.xml ├── ssl-client.xml ├── topology.map ├── topology.py └── yarn-site.xml 14 directories, 120 files [root@elephant process]# |
注意，
elepha[……]
- Posted on:
-
Author:
Adamhuan
- Categories: Hadoop
- Tags: CDH, Cloudera, Configuration, Hadoop, Lab, 笔记, 配置
- Leave a comment: CDH Lab：Exploring Hadoop Configuration Settings
CDH Lab：Add service – Spark
该实验为CDH增加一个新服务：Spark。
Follow This Step：
这里，只选择【monkey】，是依照【实验手册】的要求，而非强制。
Adjust Spark Memory settings:[……]
- Posted on:
-
Author:
Adamhuan
- Categories: Hadoop
- Tags: Add service, CDH, Hadoop, Spark
- Leave a comment: CDH Lab：Add service – Spark
CDH Lab：MapReduce – word count
版权声明还是要的，需要注意的是，本文是我在看官方实验教材的时候的笔记记录，… 用于更好的理解知识点，也为了让自己日后翻阅与查找比较便捷。
以后，标明为【CDH Lab】的日志，均为Cloudera的实验笔记或者读书笔记。
参考文献：Cloudera 官方实验手册
下面，开始：[……]
Hadoop CDH：关于HDFS的文件存储
创建目录：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
[root@elephant ~]# id hdfs uid=989(hdfs) gid=985(hdfs) groups=985(hdfs),987(hadoop) [root@elephant ~]# [root@elephant ~]# su - hdfs [hdfs@elephant ~]$ [hdfs@elephant ~]$ hdfs dfs -ls / Found 3 items drwxrwxrwt - hdfs supergroup 0 2017-09-10 17:32 /tmp drwxr-xr-x - hdfs supergroup 0 2017-09-10 01:19 /user drwxr-xr-x - hdfs supergroup 0 2017-09-11 17:32 /usr [hdfs@elephant ~]$ [hdfs@elephant ~]$ hdfs dfs -ls /user Found 1 items drwxrwxrwx - mapred hadoop 0 2017-09-10 01:36 /user/history [hdfs@elephant ~]$ [hdfs@elephant ~]$ hdfs dfs -mkdir /user/training [hdfs@elephant ~]$ [hdfs@elephant ~]$ hdfs dfs -ls /user Found 2 items drwxrwxrwx - mapred hadoop 0 2017-09-10 01:36 /user/history drwxr-xr-x - hdfs supergroup 0 2017-09-11 17:34 /user/training [hdfs@elephant ~]$ [hdfs@elephant ~]$ hdfs dfs -chown training /user/training [hdfs@elephant ~]$ [hdfs@elephant ~]$ hdfs dfs -ls /user Found 2 items drwxrwxrwx - mapred hadoop 0 2017-09-10 01:36 /user/history drwxr-xr-x - training supergroup 0 2017-09-11 17:34 /user/training [hdfs@elephant ~]$ [hdfs@elephant ~]$ exit logout [root@elephant ~]# [root@elephant ~]# su - training Last login: Sat Sep 9 12:53:39 CST 2017 from connecttocluster on pts/0 [training@elephant ~]$ [training@elephant ~]$ hdfs dfs -mkdir weblog [training@elephant ~]$ [training@elephant ~]$ hdfs dfs -ls Found 1 items drwxr-xr-x - training supergroup 0 2017-09-11 18:17 weblog [training@elephant ~]$ [training@elephant ~]$ hdfs dfs -ls /user/training Found 1 items drwxr-xr-x - training supergroup 0 2017-09-11 18:17 /user/training/weblog [training@elephant ~]$ |
NameNode UI：
节点【elephant】
http://10.158.1.97:50070
在NameNode访问文件目录：
可以看到，当前没有任何文件。
解压[……]
- Posted on:
-
Author:
Adamhuan
- Categories: Hadoop
- Tags: Hadoop, HDFS
- Leave a comment: Hadoop CDH：关于HDFS的文件存储