标签：impala hello cloudera hdfs linux
Cloudera Impala 官方教程 《Impala Tutorial》，讲解了Impala一些基本操作，但操作步骤前后缺少连贯性，本文节W选《Impala Tutorial》中的部分示例，从零开始讲解了一个完整示例：创建表、加载数据、查询数据。提供了一个入门级教程，通过本文的操作，向Impala说“Hello World”。
本文假设你已经具备了安装好的Impala环境，环境搭建可以参考： CDH5上安装Hive,HBase,Impala,Spark等服务
Impala Tutorial中示例的登录用户名为cloudera，但Cloudera Manager 5.0.2 安装时并没有自动在主机节点（例如：h1.worker.com)上创建cloudera用户，为了和Impala Tutorial 中示例一致， 需要手工创建cloudera用户和组。
以root用户身份登录主机节点（例如：h1.worker.com)，先检查下是否存在cloudera用户，执行如下的命令：
[root@h1 home]# cat /etc/passwd | grep cloudera cloudera-scm:x:496:493:Cloudera Manager:/var/run/cloudera-scm-server:/sbin/nologin上面显示不存在cloudera用户。如果存在，则不需要进行下面的创建用户步骤了。
创建cloudera用户和组，并设置密码为cloudera：
[root@h1 home]# groupadd cloudera [root@h1 home]# useradd -g cloud era cloudera [root@h1 home]# passwd cloudera Changing password for user cloudera.an New password: BAD PASSWORD: it is based on a dictionary word Retype new password: passwd: all authentication tokens updated successfully.
我们需要在HDFS上新建/user/cloudera文件夹，并将这个文件夹的所有者修改为cloudera，这需要HDFS的超级用户才有权限执行这些操作。HDFS的超级用户即运行name node进程的用户。宽泛的讲，如果你启动了name node，你就是超级用户。通过Cloudera Manager 5安装环境的超级用户名为：hdfs
切换到HDFS的超级用户，先检查是否存在 /user/cloudera 文件夹，如果不存在则创建。
[root@h1 home]# su - hdfs -bash-4.1$ hdfs dfs -ls /user Found 7 items drwx------ - hdfs supergroup 0 2014-06-26 08:44 /user/hdfs drwxrwxrwx - mapred hadoop 0 2014-06-20 10:10 /user/history drwxrwxr-t - hive hive 0 2014-06-20 10:13 /user/hive drwxrwxr-x - impala impala 0 2014-06-20 10:18 /user/impala drwxrwxr-x - oozie oozie 0 2014-06-20 10:15 /user/oozie drwxr-x--x - spark spark 0 2014-06-20 10:08 /user/spark drwxrwxr-x - sqoop2 sqoop 0 2014-06-20 10:16 /user/sqoop2在HDFS上创建 /user/cloudera 目录，设置目录的所有者和组为cloudera
-bash-4.1$ hdfs dfs -mkdir -p /user/cloudera -bash-4.1$ hdfs dfs -chown cloudera:cloudera /user/cloudera -bash-4.1$ hdfs dfs -ls /user Found 8 items drwxr-xr-x - cloudera cloudera 0 2014-06-26 09:05 /user/cloudera drwx------ - hdfs supergroup 0 2014-06-26 08:44 /user/hdfs drwxrwxrwx - mapred hadoop 0 2014-06-20 10:10 /user/history drwxrwxr-t - hive hive 0 2014-06-20 10:13 /user/hive drwxrwxr-x - impala impala 0 2014-06-20 10:18 /user/impala drwxrwxr-x - oozie oozie 0 2014-06-20 10:15 /user/oozie drwxr-x--x - spark spark 0 2014-06-20 10:08 /user/spark drwxrwxr-x - sqoop2 sqoop 0 2014-06-20 10:16 /user/sqoop2
经过以上的操作已经具备了运行 Impala Tutorial中示例的条件。
[root@h1 ~]# su - cloudera [cloudera@h1 ~]$ whoami cloudera [cloudera@h1 ~]$ hdfs dfs -ls /user Found 8 items drwxr-xr-x - cloudera cloudera 0 2014-06-26 09:05 /user/cloudera drwx------ - hdfs supergroup 0 2014-06-26 08:44 /user/hdfs drwxrwxrwx - mapred hadoop 0 2014-06-20 10:10 /user/history drwxrwxr-t - hive hive 0 2014-06-20 10:13 /user/hive drwxrwxr-x - impala impala 0 2014-06-20 10:18 /user/impala drwxrwxr-x - oozie oozie 0 2014-06-20 10:15 /user/oozie drwxr-x--x - spark spark 0 2014-06-20 10:08 /user/spark drwxrwxr-x - sqoop2 sqoop 0 2014-06-20 10:16 /user/sqoop2 [cloudera@h1 ~]$ hdfs dfs -mkdir -p /user/cloudera/sample_data/tab1 /user/cloudera/sample_data/tab2 [cloudera@h1 ~]$
通过以上的操作，就创建了存放TAB1 和 TAB2表数据的目录。
tab1.csv:
1,true,123.123,2012-10-24 08:55:00 2,false,1243.5,2012-10-25 13:40:00 3,false,24453.325,2008-08-22 09:33:21.123 4,false,243423.325,2007-05-12 22:32:21.33454 5,true,243.325,1953-04-22 09:11:33tab2.csv:
1,true,12789.123 2,false,1243.5 3,false,24453.325 4,false,2423.3254 5,true,243.325 60,false,243565423.325 70,true,243.325 80,false,243423.325 90,true,243.325
执行下面的命令将两个 .csv 文件放入单独的 HDFS 目录：
[cloudera@h1 testdata]$ pwd /home/cloudera/testdata [cloudera@h1 testdata]$ ll total 8 -rw-rw-r--. 1 cloudera cloudera 193 Jun 27 08:33 tab1.csv -rw-rw-r--. 1 cloudera cloudera 158 Jun 27 08:34 tab2.csv [cloudera@h1 testdata]$ hdfs dfs -put tab1.csv /user/cloudera/sample_data/tab1 [cloudera@h1 testdata]$ hdfs dfs -ls /user/cloudera/sample_data/tab1 Found 1 items -rw-r--r-- 3 cloudera cloudera 193 2014-06-27 08:35 /user/cloudera/sample_data/tab1/tab1.csv [cloudera@h1 testdata]$ hdfs dfs -put tab2.csv /user/cloudera/sample_data/tab2 [cloudera@h1 testdata]$ hdfs dfs -ls /user/cloudera/sample_data/tab2 Found 1 items -rw-r--r-- 3 cloudera cloudera 158 2014-06-27 08:36 /user/cloudera/sample_data/tab2/tab2.csv [cloudera@h1 testdata]$每个数据文件的名称不重要。事实上，当 Impala 第一次检测数据目录的内容时，它认为目录下的所有文件都是表中的数据文件，无论目录下有多少文件，无论什么样的文件名。
使用 impala-shell 命令创建表，可以用交互式创建，也可以用 SQL 脚本。
下面的例子演示创建了三个表。每个表中的列都使用了不同的数据类型，如 Boolean 或 integer。 例子还包括了如何格式数据的命令，例如列以逗号分隔，这样从 .csv 文件导入数据。我们已经有了存放在 HDFS 目录树中的包含数据的 .csv 文件，我们给表指定了包含对应 .csv 文件的路径位置。Impala 认为这些目录下的所有文件里的所有数据都是表里的数据。
table_setup.sql 文件包含如下内容:
DROP TABLE IF EXISTS tab1; -- The EXTERNAL clause means the data is located outside the central location for Impala data files -- and is preserved when the associated Impala table is dropped. We expect the data to already -- exist in the directory specified by the LOCATION clause. CREATE EXTERNAL TABLE tab1 ( id INT, col_1 BOOLEAN, col_2 DOUBLE, col_3 TIMESTAMP ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/user/cloudera/sample_data/tab1'; DROP TABLE IF EXISTS tab2; -- TAB2 is an external table, similar to TAB1. CREATE EXTERNAL TABLE tab2 ( id INT, col_1 BOOLEAN, col_2 DOUBLE ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/user/cloudera/sample_data/tab2'; DROP TABLE IF EXISTS tab3; -- Leaving out the EXTERNAL clause means the data will be managed -- in the central Impala data directory tree. Rather than reading -- existing data files when the table is created, we load the -- data after creating the table. CREATE TABLE tab3 ( id INT, col_1 BOOLEAN, col_2 DOUBLE, month INT, day INT ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
执行 table_setup.sql 脚本，使用：
impala-shell -i 172.16.230.152 -f table_setup.sql
操作步骤如下：
[cloudera@h1 testdata]$ pwd /home/cloudera/testdata [cloudera@h1 testdata]$ ll total 12 -rw-rw-r--. 1 cloudera cloudera 193 Jun 27 08:33 tab1.csv -rw-rw-r--. 1 cloudera cloudera 158 Jun 27 08:34 tab2.csv -rw-rw-r--. 1 cloudera cloudera 1106 Jun 27 08:49 table_setup.sql [cloudera@h1 testdata]$ impala-shell -i 172.16.230.152 -f table_setup.sql Starting Impala Shell without Kerberos authentication Connected to 172.16.230.152:21000 Server version: impalad version 1.3.1-cdh5 RELEASE (build ) ... ... Returned 0 row(s) in 0.28s [cloudera@h1 testdata]$
[cloudera@h1 testdata]$ impala-shell -i 172.16.230.152 Starting Impala Shell without Kerberos authentication Connected to 172.16.230.152:21000 Server version: impalad version 1.3.1-cdh5 RELEASE (build ) Welcome to the Impala shell. Press TAB twice to see a list of available commands. Copyright (c) 2012 Cloudera, Inc. All rights reserved. (Shell build version: Impala Shell v1.3.1-cdh5 () built on Mon Jun 9 09:30:26 PDT 2014) [172.16.230.152:21000] > show tables; Query: show tables +------+ | name | +------+ | tab1 | | tab2 | | tab3 | +------+ Returned 3 row(s) in 0.01s [172.16.230.152:21000] > describe tab1; Query: describe tab1 +-------+-----------+---------+ | name | type | comment | +-------+-----------+---------+ | id | int | | | col_1 | boolean | | | col_2 | double | | | col_3 | timestamp | | +-------+-----------+---------+ Returned 4 row(s) in 6.85s [172.16.230.152:21000] > quit; Goodbye [cloudera@h1 testdata]$
登录impala-shell，执行如下的sql语句：
SELECT * FROM tab1;
SELECT * FROM tab2 LIMIT 5;
SELECT tab2.*
FROM tab2,
(SELECT tab1.col_1, MAX(tab2.col_2) AS max_col2
FROM tab2, tab1
WHERE tab1.id = tab2.id
GROUP BY col_1) subquery1
WHERE subquery1.max_col2 = tab2.col_2;
操作步骤如下：
[cloudera@h1 testdata]$ impala-shell -i 172.16.230.152
Starting Impala Shell without Kerberos authentication
Connected to 172.16.230.152:21000
Server version: impalad version 1.3.1-cdh5 RELEASE (build )
Welcome to the Impala shell. Press TAB twice to see a list of available commands.
Copyright (c) 2012 Cloudera, Inc. All rights reserved.
(Shell build version: Impala Shell v1.3.1-cdh5 () built on Mon Jun 9 09:30:26 PDT 2014)
[172.16.230.152:21000] > SELECT * FROM tab1;
Query: select * FROM tab1
+----+-------+------------+-------------------------------+
| id | col_1 | col_2 | col_3 |
+----+-------+------------+-------------------------------+
| 1 | true | 123.123 | 2012-10-24 08:55:00 |
| 2 | false | 1243.5 | 2012-10-25 13:40:00 |
| 3 | false | 24453.325 | 2008-08-22 09:33:21.123000000 |
| 4 | false | 243423.325 | 2007-05-12 22:32:21.334540000 |
| 5 | true | 243.325 | 1953-04-22 09:11:33 |
+----+-------+------------+-------------------------------+
Returned 5 row(s) in 2.39s
[172.16.230.152:21000] > SELECT * FROM tab2 LIMIT 5;
Query: select * FROM tab2 LIMIT 5
+----+-------+-----------+
| id | col_1 | col_2 |
+----+-------+-----------+
| 1 | true | 12789.123 |
| 2 | false | 1243.5 |
| 3 | false | 24453.325 |
| 4 | false | 2423.3254 |
| 5 | true | 243.325 |
+----+-------+-----------+
Returned 5 row(s) in 1.30s
[172.16.230.152:21000] > SELECT tab2.*
> FROM tab2,
> (SELECT tab1.col_1, MAX(tab2.col_2) AS max_col2
> FROM tab2, tab1
> WHERE tab1.id = tab2.id
> GROUP BY col_1) subquery1
> WHERE subquery1.max_col2 = tab2.col_2;
Query: select tab2.* FROM tab2, (SELECT tab1.col_1, MAX(tab2.col_2) AS max_col2 FROM tab2, tab1 WHERE tab1.id = tab2.id GROUP BY col_1) subquery1 WHERE subquery1.max_col2 = tab2.col_2
+----+-------+-----------+
| id | col_1 | col_2 |
+----+-------+-----------+
| 1 | true | 12789.123 |
| 3 | false | 24453.325 |
+----+-------+-----------+
Returned 2 row(s) in 1.02s
[172.16.230.152:21000] > quit;
Goodbye
[cloudera@h1 testdata]$
本文讲解了一个Impala使用的基本示例，提供了一个入门指导，更多的示例参见： Impala Tutorial
本文使用了许多 impala-shell 命令的方法，具体参见 Using the Impala Shell (impala-shell Command)
原创作品，转载请注明出处
http://blog.csdn.net/yangzhaohui168/article/details/35340387
Hello World on Impala,布布扣,bubuko.com
标签：impala hello cloudera hdfs linux
原文地址：http://blog.csdn.net/yangzhaohui168/article/details/35340387
