单机安装hadoop-2.9.2+apache-hive-2.3.4-bin

  • 2019-04-22
  • 229
  • 0
                  <span>标签:<a href='http://www.mamicode.com/so/1/manually' title='manually'>manually</a>&nbsp;&nbsp;&nbsp;<a href='http://www.mamicode.com/so/1/service' title='service'>service</a>&nbsp;&nbsp;&nbsp;<a href='http://www.mamicode.com/so/1/store' title='store'>store</a>&nbsp;&nbsp;&nbsp;<a href='http://www.mamicode.com/so/1/href' title='href'>href</a>&nbsp;&nbsp;&nbsp;<a href='http://www.mamicode.com/so/1/chdir' title='chdir'>chdir</a>&nbsp;&nbsp;&nbsp;<a href='http://www.mamicode.com/so/1/char' title='char'>char</a>&nbsp;&nbsp;&nbsp;<a href='http://www.mamicode.com/so/1/1.4' title='1.4'>1.4</a>&nbsp;&nbsp;&nbsp;<a href='http://www.mamicode.com/so/1/int' title='int'>int</a>&nbsp;&nbsp;&nbsp;<a href='http://www.mamicode.com/so/1/hosts' title='hosts'>hosts</a>&nbsp;&nbsp;&nbsp;</p>system:<br/>centos7.5<br/>hostname:<br/>hadoop1<br/>soft:<br/>hadoop-2.9.2<br/>apache-hive-2.3.4-bin<br/>jdk-8u201-linux-x64<br/>mysql5.7《安装略》</p> <p>设置静态ip地址<br/>略<br/>添加主机与ip映射<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="23514c4c57634b42474c4c5312">[email protected]</a> ~]# cat /etc/hosts<br/>127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4<br/>::1         localhost localhost.localdomain localhost6 localhost6.localdomain6<br/>192.168.3.76    hadoop1<br/>关闭防火墙:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="dcaeb3b3a89cb4bdb8b3b3aced">[email protected]</a> ~]# systemctl stop firewalld<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="c3b1acacb783aba2a7acacb3f2">[email protected]</a> ~]# systemctl disable firewalld<br/>关闭seLinux<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="3d4f5252497d555c5952524d0c">[email protected]</a> ~]# egrep -v &quot;^#|^$&quot; /etc/selinux/config <br/>SELINUX=disabled<br/>其它参数设置:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="2e5c41415a6e464f4a41415e1f">[email protected]</a> ~]# sysctl -w vm.max_map_count=262144<br/>vm.max_map_count = 262144</p> <pre><code>[[email protected] ~]# egrep -v "^#|^$" /etc/security/limits.conf  *       soft    nofile      65536 *       hard    nofile      131072 *       soft    nproc       65536 *       hard    nproc       65536</code></pre> <p>安装java:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="384a57574c7850595c57574809">[email protected]</a> opt]# ls<br/>jdk-8u201-linux-x64.rpm<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="4d3f2222390d252c2922223d7c">[email protected]</a> opt]# rpm -ih jdk-8u201-linux-x64.rpm <br/>warning: jdk-8u201-linux-x64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY<br/>################################# [100%]<br/>Updating / installing...<br/>################################# [100%]<br/>Unpacking JAR files...<br/>tools.jar...<br/>plugin.jar...<br/>javaws.jar...<br/>deploy.jar...<br/>rt.jar...<br/>jsse.jar...<br/>charsets.jar...<br/>localedata.jar...</p> <pre><code>[[email protected] opt]# java -version java version "1.8.0_201" Java(TM) SE Runtime Environment (build 1.8.0_201-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)    </code></pre> <p>建立hadoop账户《987654321》:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="daa8b5b5ae9ab2bbbeb5b5aaeb">[email protected]</a> opt]# useradd hadoop<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="ddafb2b2a99db5bcb9b2b2adec">[email protected]</a> opt]# passwd hadoop</p> <p>设置sudo权限<br/>略</p> <p>重启系统:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="b6c4d9d9c2f6ded7d2d9d9c687">[email protected]</a> ~]# reboot</p> <p>以下是有hadoop操作:</p> <table> <thead> <tr> <th>设置ssh免密码登录:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="1971787d7676695971787d76766928">[email protected]</a> ~]$ ssh-keygen <br/>Generating public/private rsa key pair.<br/>Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): <br/>Enter passphrase (empty for no passphrase): <br/>Enter same passphrase again: <br/>Your identification has been saved in /home/hadoop/.ssh/id_rsa.<br/>Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.<br/>The key fingerprint is:<br/>6e:d0:62:99:3c:02:10:cf:00:5a:71:3c:6f:82:67:94 <a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="177f7673787867577f767378786726">[email protected]</a><br/>The key‘s randomart image is:<br/>+--[ RSA 2048]----+<br/></th> <th>*.oo..</th> <th><br/></th> <th>.* .E</th> <th><br/></th> <th>. +o o</th> <th><br/></th> <th>..+.o+</th> <th><br/></th> <th>o.oO S</th> <th><br/></th> <th>o =</th> <th><br/></th> <th>o</th> <th><br/></th> <th>.</th> </tr> </thead> <tbody> </tbody> </table> <pre><code>+-----------------+ [[email protected] ~]$ ssh-copy-id -i .ssh/id_rsa.pub [email protected]</code></pre> <p>hadoop搭建:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="3644595942765e575259594607">[email protected]</a> hadoop]# su - hadoop<br/>Last login: Fri Apr 19 20:56:04 CST 2019 on pts/0<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="2e464f4a41415e6e464f4a41415e1f">[email protected]</a> ~]$ ls<br/>hadoop-2.9.2.tar.gz<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="ef878e8b80809faf878e8b80809fde">[email protected]</a> ~]$ tar -zxf hadoop-2.9.2.tar.gz <br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="630b02070c0c13230b02070c0c1352">[email protected]</a> ~]$ ls<br/>hadoop-2.9.2  hadoop-2.9.2.tar.gz</p> <p>配置hadoop-env.sh<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="452d24212a2a35052d24212a2a3574">[email protected]</a> ~]$ vim hadoop-2.9.2/etc/hadoop/hadoop-env.sh<br/>#export JAVA_HOME=${JAVA_HOME}<br/>export JAVA_HOME=/usr/java/jdk1.8.0_201-amd64</p> <p>配置core-site.xml<br/>vim hadoop-2.9.2/etc/hadoop/core-site.xml<br/>&lt;configuration&gt;<br/>&lt;!-- 指定HADOOP所使用的文件系统schema(URI),HDFS的老大(NameNode)的地址 --&gt;<br/>&lt;property&gt;<br/>&lt;name&gt;fs.defaultFS&lt;/name&gt;<br/>&lt;value&gt;hdfs://192.168.3.76:9000&lt;/value&gt;<br/>&lt;/property&gt;<br/>&lt;!-- 指定hadoop运行时产生临时文件的存储目录 --&gt;<br/>&lt;property&gt;<br/>&lt;name&gt;hadoop.tmp.dir&lt;/name&gt;<br/>&lt;value&gt;/home/hadoop/tmp&lt;/value&gt;<br/>&lt;/property&gt;</p> <pre><code>[[email protected] ~]$ mkdir tmp</code></pre> <p>配置hdfs-site.xml<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="fb939a9f94948bbb939a9f94948bca">[email protected]</a> ~]$ vim hadoop-2.9.2/etc/hadoop/hdfs-site.xml<br/>&lt;property&gt;<br/>&lt;name&gt;dfs.namenode.name.dir&lt;/name&gt;<br/>&lt;value&gt;/home/hadoop/dfs/namenode&lt;/value&gt;<br/>&lt;final&gt;true&lt;/final&gt;<br/>&lt;/property&gt;<br/>&lt;property&gt;<br/>&lt;name&gt;dfs.datanode.data.dir&lt;/name&gt;<br/>&lt;value&gt;/home/hadoop/dfs/datanode&lt;/value&gt;<br/>&lt;final&gt;true&lt;/final&gt;<br/>&lt;/property&gt;<br/>&lt;property&gt;<br/>&lt;name&gt;dfs.http.address&lt;/name&gt;<br/>&lt;value&gt;192.168.3.76:50070&lt;/value&gt;<br/>&lt;description&gt;The address and the base port where the dfs namenode web ui will listen on.If the port is 0 then the server will start on a free port&lt;/description&gt;<br/>&lt;/property&gt;<br/>&lt;!-- 指定HDFS副本的数量 --&gt;<br/>&lt;property&gt;<br/>&lt;name&gt;dfs.replication&lt;/name&gt;<br/>&lt;value&gt;1&lt;/value&gt;<br/>&lt;/property&gt;<br/>&lt;property&gt;<br/>&lt;name&gt;dfs.permissions&lt;/name&gt;<br/>&lt;value&gt;false&lt;/value&gt;<br/>&lt;/property&gt;</p> <p>建目录:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="81e9e0e5eeeef1c1e9e0e5eeeef1b0">[email protected]</a> ~]$ mkdir dfs/datanode -p<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="2f474e4b40405f6f474e4b40405f1e">[email protected]</a> ~]$ mkdir dfs/namenode -p</p> <p>配置mapred-site.xml<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="610900050e0e11210900050e0e1150">[email protected]</a> ~]$ cp hadoop-2.9.2/etc/hadoop/mapred-site.xml.template hadoop-2.9.2/etc/hadoop/mapred-site.xml<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="244c45404b4b54644c45404b4b5415">[email protected]</a> ~]$ vim hadoop-2.9.2/etc/hadoop/mapred-site.xml<br/>&lt;!-- 指定mr运行在yarn上 --&gt;<br/>&lt;property&gt;<br/>&lt;name&gt;mapreduce.framework.name&lt;/name&gt;<br/>&lt;value&gt;yarn&lt;/value&gt;<br/>&lt;/property&gt;<br/>&lt;property&gt;<br/>&lt;name&gt;mapred.job.tracker&lt;/name&gt;<br/>&lt;value&gt;hdfs://192.168.3.76:9001&lt;/value&gt;<br/>&lt;/property&gt;<br/>&lt;property&gt;<br/>&lt;name&gt;mapred.system.dir&lt;/name&gt;<br/>&lt;value&gt;file:/home/hadoop/mapred/system&lt;/value&gt;<br/>&lt;final&gt;true&lt;/final&gt;<br/>&lt;/property&gt;<br/>&lt;property&gt;<br/>&lt;name&gt;mapred.local.dir&lt;/name&gt;<br/>&lt;value&gt;file:/home/hadoop/mapred/local&lt;/value&gt;<br/>&lt;final&gt;true&lt;/final&gt;<br/>&lt;/property&gt;<br/>建立目录:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="a5cdc4c1cacad5e5cdc4c1cacad594">[email protected]</a> ~]$ mkdir mapred/local -p<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="1d757c7972726d5d757c7972726d2c">[email protected]</a> ~]$ mkdir mapred/system -p</p> <p>配置yarn-site.xml<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="345c55505b5b44745c55505b5b4405">[email protected]</a> ~]$ vim hadoop-2.9.2/etc/hadoop/yarn-site.xml<br/>&lt;!-- 指定YARN的老大(ResourceManager)的地址 --&gt;<br/>&lt;property&gt;<br/>&lt;name&gt;yarn.resourcemanager.hostname&lt;/name&gt;<br/>&lt;value&gt;192.168.3.76&lt;/value&gt;<br/>&lt;/property&gt;<br/>&lt;!-- reducer获取数据的方式 --&gt;<br/>&lt;property&gt;<br/>&lt;name&gt;yarn.nodemanager.aux-services&lt;/name&gt;<br/>&lt;value&gt;mapreduce_shuffle&lt;/value&gt;<br/>&lt;/property&gt;</p> <p>格式化hdfs<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="177f7673787867577f767378786726">[email protected]</a> ~]$ hadoop-2.9.2/bin/hdfs namenode -format<br/>出现下面第二行的successfully,那么表名成功<br/>19/04/19 21:24:20 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1608387477-192.168.3.76-1555680260442<br/>19/04/19 21:24:20 INFO common.Storage: Storage directory /home/hadoop/tmp/dfs/name has been successfully formatted.<br/>19/04/19 21:24:20 INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression<br/>19/04/19 21:24:20 INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 325 bytes saved in 0 seconds .<br/>19/04/19 21:24:20 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid &gt;= 0<br/>19/04/19 21:24:20 INFO namenode.NameNode: SHUTDOWN_MSG: <br/>/<strong><strong><strong><strong><strong><strong><strong><strong><strong><strong><strong><strong><strong><strong>****</strong></strong></strong></strong></strong></strong></strong></strong></strong></strong></strong></strong></strong></strong><br/>SHUTDOWN_MSG: Shutting down NameNode at hadoop1/192.168.3.76<br/><strong><strong><strong><strong><strong><strong><strong><strong><strong><strong><strong><strong><strong><strong>****</strong></strong></strong></strong></strong></strong></strong></strong></strong></strong></strong></strong></strong></strong>/<br/>启动并测试hdfs:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="402821242f2f30002821242f2f3071">[email protected]</a> ~]$ hadoop-2.9.2/sbin/start-all.sh<br/>This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh<br/>Starting namenodes on [hadoop1]<br/>hadoop1: starting namenode, logging to /home/hadoop/hadoop-2.9.2/logs/hadoop-hadoop-namenode-hadoop1.out<br/>localhost: starting datanode, logging to /home/hadoop/hadoop-2.9.2/logs/hadoop-hadoop-datanode-hadoop1.out<br/>Starting secondary namenodes [0.0.0.0]<br/>0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop-2.9.2/logs/hadoop-hadoop-secondarynamenode-hadoop1.out<br/>starting yarn daemons<br/>starting resourcemanager, logging to /home/hadoop/hadoop-2.9.2/logs/yarn-hadoop-resourcemanager-hadoop1.out<br/>localhost: starting nodemanager, logging to /home/hadoop/hadoop-2.9.2/logs/yarn-hadoop-nodemanager-hadoop1.out</p> <p>检测:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="107871747f7f60507871747f7f6021">[email protected]</a> ~]$ jps<br/>4705 SecondaryNameNode<br/>4865 ResourceManager<br/>4386 NameNode<br/>5157 NodeManager<br/>5318 Jps<br/>4488 DataNode</p> <p>实例测试:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="dab2bbbeb5b5aa9ab2bbbeb5b5aaeb">[email protected]</a> ~]$ ll<br/>total 357872<br/>drwxrwxr-x  4 hadoop hadoop        36 Apr 19 22:03 dfs<br/>drwxr-xr-x 10 hadoop hadoop       150 Apr 19 21:27 hadoop-2.9.2<br/>-rw-r--r--  1 hadoop hadoop 366447449 Apr 19 20:56 hadoop-2.9.2.tar.gz<br/>drwxrwxr-x  4 hadoop hadoop        31 Apr 19 21:52 mapred<br/>-rw-r--r--  1 hadoop hadoop     11323 Apr 19 22:11 qqqq.xlsx<br/>drwxrwxr-x  4 hadoop hadoop        35 Apr 19 22:06 tmp<br/>这里的文件名必须要以‘/’开头,暂时只了解是hdfs是以绝对路径为基础,因为没有 ‘-cd’这样的命令支持<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="7d151c1912120d3d151c1912120d4c">[email protected]</a> ~]$ hadoop-2.9.2/bin/hdfs dfs -mkdir /input<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="3e565f5a51514e7e565f5a51514e0f">[email protected]</a> ~]$ hadoop-2.9.2/bin/hdfs dfs -put qqqq.xlsx /input<br/>也可以查看此时新建的input目录里面有什么<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="bbd3dadfd4d4cbfbd3dadfd4d4cb8a">[email protected]</a> ~]$ hadoop-2.9.2/bin/hdfs dfs -ls /<br/>Found 1 items<br/>drwxr-xr-x   - hadoop supergroup          0 2019-04-19 22:14 /input<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="b5ddd4d1dadac5f5ddd4d1dadac584">[email protected]</a> ~]$ hadoop-2.9.2/bin/hdfs dfs -ls /input<br/>Found 1 items<br/>-rw-r--r--   1 hadoop supergroup      11323 2019-04-19 22:14 /input/qqqq.xlsx<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="8ce4ede8e3e3fccce4ede8e3e3fcbd">[email protected]</a> ~]$ hadoop-2.9.2/bin/hadoop jar hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar grep /input /output ‘dfs[a-z.]+‘</p> <p>19/04/19 22:23:05 INFO mapreduce.Job: Job job_1555682785585_0001 completed successfully<br/>19/04/19 22:23:23 INFO mapreduce.Job: Job job_1555682785585_0002 completed successfully</p> <p>结果:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="711910151e1e01311910151e1e0140">[email protected]</a> ~]$ hadoop-2.9.2/bin/hdfs dfs -ls /<br/>Found 4 items<br/>drwxr-xr-x   - hadoop supergroup          0 2019-04-19 22:14 /input<br/>drwxr-xr-x   - hadoop supergroup          0 2019-04-19 22:23 /output<br/>drwx------   - hadoop supergroup          0 2019-04-19 22:22 /tmp<br/>drwxr-xr-x   - hadoop supergroup          0 2019-04-19 22:22 /user<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="90f8f1f4ffffe0d0f8f1f4ffffe0a1">[email protected]</a> ~]$ hadoop-2.9.2/bin/hdfs dfs -ls /output<br/>Found 2 items<br/>-rw-r--r--   1 hadoop supergroup          0 2019-04-19 22:23 /output/_SUCCESS<br/>-rw-r--r--   1 hadoop supergroup          0 2019-04-19 22:23 /output/part-r-00000</p> <p>检测二:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="6901080d0606192901080d06061958">[email protected]</a> ~]$ ll<br/>total 357876<br/>drwxrwxr-x  4 hadoop hadoop        36 Apr 19 22:03 dfs<br/>drwxr-xr-x 10 hadoop hadoop       150 Apr 19 21:27 hadoop-2.9.2<br/>-rw-r--r--  1 hadoop hadoop 366447449 Apr 19 20:56 hadoop-2.9.2.tar.gz<br/>drwxrwxr-x  4 hadoop hadoop        31 Apr 19 21:52 mapred<br/>-rw-r--r--  1 hadoop hadoop     11323 Apr 19 22:11 qqqq.xlsx<br/>drwxrwxr-x  4 hadoop hadoop        35 Apr 19 22:06 tmp<br/>-rw-rw-r--  1 hadoop hadoop       213 Apr 19 22:30 www.text<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="274f4643484857674f464348485716">[email protected]</a> ~]$ cat www.text <br/><a href="http://blog.csdn.net/u012342408/article/details/50520696" rel="nofollow">http://blog.csdn.net/u012342408/article/details/50520696</a><br/><a href="http://blog.csdn.net/hitwengqi/article/details/8008203" rel="nofollow">http://blog.csdn.net/hitwengqi/article/details/8008203</a><br/><a href="http://blog.csdn.net/zl007700/article/details/50533675" rel="nofollow">http://blog.csdn.net/zl007700/article/details/50533675</a><br/><a href="https://www.cnblogs.com/yanglf/p/4020555.html" rel="nofollow">https://www.cnblogs.com/yanglf/p/4020555.html</a></p> <p>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="a9c1c8cdc6c6d9e9c1c8cdc6c6d998">[email protected]</a> ~]$ hadoop-2.9.2/bin/hdfs dfs -put www.text /input<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="80e8e1e4efeff0c0e8e1e4efeff0b1">[email protected]</a> ~]$ hadoop-2.9.2/bin/hdfs dfs -ls /input<br/>Found 2 items<br/>-rw-r--r--   1 hadoop supergroup      11323 2019-04-19 22:14 /input/qqqq.xlsx<br/>-rw-r--r--   1 hadoop supergroup        213 2019-04-19 22:31 /input/www.text</p> <p>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="aec6cfcac1c1deeec6cfcac1c1de9f">[email protected]</a> ~]$ hadoop-2.9.2/bin/hadoop jar hadoop-2.9.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar grep /input/www.text /www ‘dfs[a-z.]+‘</p> <p>19/04/19 22:33:33 INFO mapreduce.Job: Job job_1555682785585_0004 completed successfully<br/>19/04/19 22:33:51 INFO mapreduce.Job: Job job_1555682785585_0005 completed successfully</p> <p>结果:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="620a03060d0d12220a03060d0d1253">[email protected]</a> ~]$ hadoop-2.9.2/bin/hdfs dfs -ls /www<br/>Found 2 items<br/>-rw-r--r--   1 hadoop supergroup          0 2019-04-19 22:33 /www/_SUCCESS<br/>-rw-r--r--   1 hadoop supergroup          0 2019-04-19 22:33 /www/part-r-00000</p> <p>查看安装包:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="b6c4d9d9c2f6ded7d2d9d9c687">[email protected]</a> hive]# su - hadoop<br/>Last login: Sat Apr 20 15:46:10 CST 2019 on pts/1<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="dcb4bdb8b3b3ac9cb4bdb8b3b3aced">[email protected]</a> ~]$ ll<br/>total 589016<br/>-rw-r--r--  1 hadoop hadoop 232234292 Apr 20 15:53 apache-hive-2.3.4-bin.tar.gz<br/>drwxrwxr-x  4 hadoop hadoop        36 Apr 19 22:03 dfs<br/>drwxr-xr-x 10 hadoop hadoop       150 Apr 19 21:27 hadoop-2.9.2<br/>-rw-r--r--  1 hadoop hadoop 366447449 Apr 19 20:56 hadoop-2.9.2.tar.gz<br/>drwxrwxr-x  4 hadoop hadoop        31 Apr 19 21:52 mapred<br/>-rw-r--r--  1 hadoop hadoop   4452049 Apr 20 15:53 mysql-connector-java-5.1.47.tar.gz<br/>-rw-r--r--  1 hadoop hadoop     11323 Apr 19 22:11 qqqq.xlsx<br/>drwxrwxr-x  4 hadoop hadoop        35 Apr 19 22:06 tmp<br/>-rw-rw-r--  1 hadoop hadoop       213 Apr 19 22:30 www.text<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="513930353e3e21113930353e3e2160">[email protected]</a> ~]$ tar -zxf apache-hive-2.3.4-bin.tar.gz <br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="1a727b7e75756a5a727b7e75756a2b">[email protected]</a> ~]$ tar -zxf mysql-connector-java-5.1.47.tar.gz</p> <p>设置hive变量:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="224a43464d4d52624a43464d4d5213">[email protected]</a> ~]$ egrep -v &quot;^#|^$&quot; .bashrc <br/>if [ -f /etc/bashrc ]; then<br/>. /etc/bashrc<br/>fi<br/>export HIVE_HOME=/home/hadoop/apache-hive-2.3.4-bin<br/>export PATH=$PATH:$HIVE_HOME/bin</p> <pre><code>[[email protected] ~]$ source .bashrc</code></pre> <p>建立配置文件:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="157d74717a7a65557d74717a7a6524">[email protected]</a> conf]$ pwd<br/>/home/hadoop/apache-hive-2.3.4-bin/conf<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="a3cbc2c7ccccd3e3cbc2c7ccccd392">[email protected]</a> conf]$ cp hive-env.sh.template hive-env.sh<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="660e0702090916260e070209091657">[email protected]</a> conf]$ cp hive-default.xml.template hive-site.xml<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="761e1712191906361e171219190647">[email protected]</a> conf]$ cp hive-log4j2.properties.template hive-log4j2.properties<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="c7afa6a3a8a8b787afa6a3a8a8b7f6">[email protected]</a> conf]$ cp hive-exec-log4j2.properties.template hive-exec-log4j2.properties</p> <p>配置hive-site.xml,修改一下几行:</p> <p>&lt;property&gt;<br/>&lt;name&gt;hive.exec.scratchdir&lt;/name&gt;<br/>&lt;value&gt;/home/hadoop/tmp/hive-${user.name}&lt;/value&gt;<br/>&lt;description&gt;HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/&lt;username&gt; is created, with ${hive.scratch.dir.permission}.&lt;/description&gt;<br/>&lt;/property&gt;</p> <p>&lt;property&gt;<br/>&lt;name&gt;hive.exec.local.scratchdir&lt;/name&gt;<br/>&lt;value&gt;/home/hadoop/tmp/${user.name}&lt;/value&gt;<br/>&lt;description&gt;Local scratch space for Hive jobs&lt;/description&gt;<br/>&lt;/property&gt;</p> <p>&lt;property&gt;<br/>&lt;name&gt;hive.downloaded.resources.dir&lt;/name&gt;<br/>&lt;value&gt;/home/hadoop/tmp/hive/resources&lt;/value&gt;<br/>&lt;description&gt;Temporary local directory for added resources in the remote file system.&lt;/description&gt;<br/>&lt;/property&gt;</p> <p>&lt;property&gt;<br/>&lt;name&gt;hive.server2.logging.operation.log.location&lt;/name&gt;<br/>&lt;value&gt;/home/hadoop/tmp/${user.name}/operation_logs&lt;/value&gt;<br/>&lt;description&gt;Top level directory where operation logs are stored if logging functionality is enabled&lt;/description&gt;<br/>&lt;/property&gt;</p> <p>&lt;property&gt;<br/>&lt;name&gt;hive.querylog.location&lt;/name&gt;<br/>&lt;value&gt;/home/hadoop/tmp/${user.name}&lt;/value&gt;<br/>&lt;description&gt;Location of Hive run time structured log file&lt;/description&gt;<br/>&lt;/property&gt;</p> <p>&lt;property&gt;<br/>&lt;name&gt;javax.jdo.option.ConnectionURL&lt;/name&gt;<br/>&lt;value&gt;jdbc:mysql://192.168.3.76:3306/hive_metadata?&amp;createDatabaseIfNotExist=true&amp;characterEncoding=UTF-8&amp;useSSL=false&lt;/value&gt;<br/>&lt;description&gt;<br/>JDBC connect string for a JDBC metastore.<br/>To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.<br/>For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.<br/>&lt;/description&gt;<br/>&lt;/property&gt;</p> <p>&lt;property&gt;<br/>&lt;name&gt;javax.jdo.option.ConnectionUserName&lt;/name&gt;<br/>&lt;value&gt;root&lt;/value&gt;<br/>&lt;description&gt;Username to use against metastore database&lt;/description&gt;<br/>&lt;/property&gt;</p> <p>&lt;property&gt;<br/>&lt;name&gt;javax.jdo.option.ConnectionPassword&lt;/name&gt;<br/>&lt;value&gt;Wd#GDrf142D&lt;/value&gt;<br/>&lt;description&gt;password to use against metastore database&lt;/description&gt;<br/>&lt;/property&gt;</p> <p>&lt;property&gt;<br/>&lt;name&gt;javax.jdo.option.ConnectionDriverName&lt;/name&gt;<br/>&lt;value&gt;com.mysql.jdbc.Driver&lt;/value&gt;<br/>&lt;description&gt;Driver class name for a JDBC metastore&lt;/description&gt;<br/>&lt;/property&gt;</p> <p>&lt;property&gt;<br/>&lt;name&gt;datanucleus.schema.autoCreateAll&lt;/name&gt;<br/>&lt;value&gt;true&lt;/value&gt;<br/>&lt;description&gt;Auto creates necessary schema on a startup if one doesn‘t exist. Set this to false, after creating it once.To enable auto create also set hive.metastore.schema.verification=false. Auto creation is not recommended for production use cases, run schematool command instead.&lt;/description&gt;<br/>&lt;/property&gt;</p> <p>&lt;property&gt;<br/>&lt;name&gt;hive.metastore.schema.verification&lt;/name&gt;<br/>&lt;value&gt;false&lt;/value&gt;<br/>&lt;description&gt;<br/>Enforce metastore schema version consistency.<br/>True: Verify that version information stored in is compatible with one from Hive jars.  Also disable automatic<br/>schema migration attempt. Users are required to manually migrate schema after Hive upgrade which ensures<br/>proper metastore schema migration. (Default)<br/>False: Warn if the version information stored in metastore doesn‘t match with one from in Hive jars.<br/>&lt;/description&gt;<br/>&lt;/property&gt;</p> <p>配置hive-env.sh:</p> <h1>Set HADOOP_HOME to point to a specific hadoop install directory</h1> <pre><code># HADOOP_HOME=${bin}/../../hadoop HADOOP_HOME=/home/hadoop/hadoop-2.9.2  # Hive Configuration Directory can be controlled by: # export HIVE_CONF_DIR= export HIVE_CONF_DIR=/home/hive/apache-hive-2.3.4-bin/conf</code></pre> <p>加载mysql驱动<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="0b636a6f64647b4b636a6f64647b3a">[email protected]</a> ~]$ ll<br/>total 231144<br/>drwxrwxr-x 10 hive hive      4096 Apr 20 14:21 apache-hive-2.3.4-bin<br/>-rw-r--r--  1 hive hive 232234292 Apr 20 14:16 apache-hive-2.3.4-bin.tar.gz<br/>-rw-r--r--  1 hive hive   4452049 Apr 20 15:23 mysql-connector-java-5.1.47.tar.gz<br/>drwxrwxr-x  3 hive hive        17 Apr 20 15:07 tmp<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="94fcfde2f1d4fcf5f0fbfbe4a5">[email protected]</a> ~]$ tar -zxf mysql-connector-java-5.1.47.tar.gz <br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="fb939a9f94948bbb939a9f94948bca">[email protected]</a> ~]$ cp mysql-connector-java-5.1.47/mysql-connector-java-5.1.47.jar apache-hive-2.3.4-bin/lib/</p> <p>为Hive创建HDFS目录<br/>在 Hive 中创建表之前需要使用以下 HDFS 命令创建 /tmp 和 /user/hive/warehouse (hive-site.xml 配置文件中属性项 hive.metastore.warehouse.dir 的默认值) 目录并给它们赋写权限<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="224a43464d4d52624a43464d4d5213">[email protected]</a> ~]$ hadoop-2.9.2/bin/hdfs dfs -mkdir tmp<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="d4bcb5b0bbbba494bcb5b0bbbba4e5">[email protected]</a> ~]$ hadoop-2.9.2/bin/hdfs dfs -mkdir -p /user/hive/warehouse<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="0b636a6f64647b4b636a6f64647b3a">[email protected]</a> ~]$ hadoop-2.9.2/bin/hdfs dfs -chmod g+w /user/hive/warehouse<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="fc949d9893938cbc949d9893938ccd">[email protected]</a> ~]$ hadoop-2.9.2/bin/hdfs dfs -chmod g+w tmp/</p> <p>给mysql创建用户hive/密码hive&lt;在没有使用mysql数据库root账户的情况下使用&gt;:<br/>$ mysql -u root -p #密码已设为123456<br/>mysql&gt; CREATE USER ‘hive‘@‘localhost‘ IDENTIFIED BY &quot;hive&quot;;<br/>mysql&gt; grant all privileges on <em>.</em> to <a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="523a3b2437123e3d31333e3a3d2126">[email protected]</a> identified by ‘hive‘;</p> <p>更改root远程访问:<br/>mysql&gt; use mysql;<br/>mysql&gt; update user set host = ‘%‘ where user = ‘root‘;<br/>mysql&gt; flush privileges;<br/>mysql&gt; select host, user from user;<br/>+-----------+---------------+<br/>| host      | user          |<br/>+-----------+---------------+<br/>| %         | root          |<br/>| localhost | mysql.session |<br/>| localhost | mysql.sys     |<br/>+-----------+---------------+</p> <p>运行Hive<br/>在命令行运行 hive 命令时必须保证 HDFS 已经启动。可以使用 start-dfs.sh 来启动 HDFS。</p> <p>从 Hive 2.1 版本开始, 我们需要先运行 schematool 命令来执行初始化操作。</p> <pre><code>[[email protected] ~]$ apache-hive-2.3.4-bin/bin/schematool -dbType mysql -initSchema SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/hadoop/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Metastore connection URL:    jdbc:mysql://192.168.3.76:3306/hive_metadata?&createDatabaseIfNotExist=true&characterEncoding=UTF-8&useSSL=false Metastore Connection Driver :    com.mysql.jdbc.Driver Metastore connection User:   root Starting metastore schema initialization to 2.3.0 Initialization script hive-schema-2.3.0.mysql.sql Initialization script completed schemaTool completed</code></pre> <p>启动hive并测试:<br/>[<a href="http://www.mamicode.com/cdn-cgi/l/email-protection" data-cfemail="b8d0d9dcd7d7c8f8d0d9dcd7d7c889">[email protected]</a> ~]$ apache-hive-2.3.4-bin/bin/hive<br/>which: no hbase in (/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/hadoop/.local/bin:/home/hadoop/bin:/home/hadoop/apache-hive-2.3.4-bin/bin)<br/>SLF4J: Class path contains multiple SLF4J bindings.<br/>SLF4J: Found binding in [jar:file:/home/hadoop/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]<br/>SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]<br/>SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" rel="nofollow">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation.<br/>SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]</p> <pre><code>Logging initialized using configuration in file:/home/hadoop/apache-hive-2.3.4-bin/conf/hive-log4j2.properties Async: true Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. hive> show tables; OK Time taken: 3.236 seconds hive> show databases; OK default Time taken: 0.055 seconds, Fetched: 1 row(s) hive> </code></pre> <p>简答的hive语句测试:<br/>建表:<br/>hive&gt; CREATE TABLE IF NOT EXISTS test (id INT,name STRING)ROW FORMAT DELIMITED FIELDS TERMINATED BY &quot; &quot; LINES TERMINATED BY &quot;n&quot;;<br/>OK<br/>Time taken: 0.524 seconds<br/>hive&gt; insert into test values(1,‘张三‘);<br/>WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.<br/>Query ID = hadoop_20190420163725_0be10015-72ae-4642-b2c4-311aaeaacaa8<br/>Total jobs = 3<br/>Launching Job 1 out of 3<br/>Number of reduce tasks is set to 0 since there‘s no reduce operator<br/>Starting Job = job_1555738609578_0001, Tracking URL = <a href="http://hadoop1:8088/proxy/application_1555738609578_0001/" rel="nofollow">http://hadoop1:8088/proxy/application_1555738609578_0001/</a><br/>Kill Command = /home/hadoop/hadoop-2.9.2/bin/hadoop job  -kill job_1555738609578_0001<br/>Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0<br/>2019-04-20 16:37:38,182 Stage-1 map = 0%,  reduce = 0%<br/>2019-04-20 16:37:43,443 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.28 sec<br/>MapReduce Total cumulative CPU time: 2 seconds 280 msec<br/>Ended Job = job_1555738609578_0001<br/>Stage-4 is selected by condition resolver.<br/>Stage-3 is filtered out by condition resolver.<br/>Stage-5 is filtered out by condition resolver.<br/>Moving data to directory hdfs://192.168.3.76:9000/user/hive/warehouse/test/.hive-staging_hive_2019-04-20_16-37-25_672_7073846121967206245-1/-ext-10000<br/>Loading data to table default.test<br/>MapReduce Jobs Launched: <br/>Stage-Stage-1: Map: 1   Cumulative CPU: 2.28 sec   HDFS Read: 4249 HDFS Write: 77 SUCCESS<br/>Total MapReduce CPU Time Spent: 2 seconds 280 msec<br/>OK<br/>Time taken: 19.356 seconds<br/>hive&gt; select * from test;<br/>OK<br/>1   张三<br/>Time taken: 0.352 seconds, Fetched: 1 row(s)

单机安装hadoop-2.9.2+apache-hive-2.3.4-bin

标签:manually   service   store   href   chdir   char   1.4   int   hosts   

原文地址:https://blog.51cto.com/1054054/2382121

评论

还没有任何评论,你来说两句吧