CentOS 7 安装 Cloudera Manager 6.2.0

-w1676

1. 安装 Cloudera Manager

我用了一台 2 核 8 G 的 云服务器(CentOS 7.6) 安装 Cloudera Manager, 3 台 4 核 8 G 安装 Hadoop。

在 4 台服务器(其中 3 台安装 Hadoop)上,均设置好主机名和 hosts,确保相互之间通过主机名能访问。

1.1 安装 JDK

去 华为 镜像源,下载 JDK open in new window

rpm -ivh jdk-8u202-linux-x64.rpm
1

1.2 安装 Cloudera Manager

下载 安装包open in new window

 yum localinstall cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm
1

1.3 安装数据库

安装 mariadb,用于 Cloud Manager 的数据库

yum install mariadb-server
systemctl enable mariadb
systemctl start mariadb 
1
2
3

初始化数据库

/usr/bin/mysql_secure_installation 
1

设置 MySQL 连接驱动

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar xf mysql-connector-java-5.1.46.tar.gz

mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
1
2
3
4
5
6

1.4 初始化数据

初始化数据库脚本

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'dhcpdotcn';
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'dhcpdotcn';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'dhcpdotcn';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'dhcpdotcn';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'dhcpdotcn';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'dhcpdotcn';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'dhcpdotcn';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'dhcpdotcn';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'dhcpdotcn';
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

执行初始化数据库

mysql -uroot -p<ROOT_PASSWORD> < ./cdh.sql
1

执行初始化

 /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm
 # 上面设置的密码
1
2

1.5 准备 Hadoop 版本源

 cd /opt/cloudera/parcel-repo
 wget https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel
 wget https://archive.cloudera.com/cdh6/6.2.0/parcels/manifest.json
 sha1sum CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel | awk '{ print $1 }' > CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha
 chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*
1
2
3
4
5

1.6 启动 Cloudera Manager

systemctl start cloudera-scm-server
systemctl enable cloudera-scm-server
1
2

等待 1 ~ 2 分钟,访问 :7180 (admin/admin)即可进入安装界面,不着急,直接进入下一个章节,安装 Hadoop 前需要有一些准备事项。

-w1679

这是 Cloudera Manager 可以 安装的集群组件

Cloudera Manager 安装好了,他只是一个管理器,接下来开始安装 Hadoop 组件。

2. 安装 Hadoop

我用了 3 台 4 核 8 G 的 云服务器,CentOS 7.6.

2.1 所有节点 安装 JDK

去 华为 镜像源,下载 JDK open in new window

rpm -ivh jdk-8u202-linux-x64.rpm
1

使用 ansible 操作比较方便: hadoop -a "rpm -ivh /data/cloudrea_manager/jdk-8u202-linux-x64.rpm"

2.2 安装 Cloudera Agent

RPM 地址: https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/

yum localinstall cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm
1

使用 ansible 操作比较方便: ansible hadoop -a "yum localinstall /data/cloudrea_manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm /data/cloudrea_manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm -y"

2.3 初始化环境

echo "vm.swappiness = 10" >> /etc/sysctl.conf # 优化交换分区,默认是 30
sysctl -p
echo never > /sys/kernel/mm/transparent_hugepage/defrag # 透明大页面压缩
echo never > /sys/kernel/mm/transparent_hugepage/enabled # 透明大页面压缩
1
2
3
4

使用 ansible 操作比较方便:hadoop -m lineinfile -a "dest=/etc/sysctl.conf line='vm.swappiness = 10'"

2.4 开始按照 7180 端口 访问的安装指引安装即可

下图中,选择 Public Cloudera repository 即可。

2.5 设置 MySQL 连接驱动

在需要 MySQL 的 角色 所在服务器上,部署驱动。

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar xf mysql-connector-java-5.1.46.tar.gz

mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
1
2
3
4
5
6

按照步骤指引,安装完成。

-w1677

附录:问题排查

因 MySQL 版本导致的 cloudera-scm-server 反复重启失败

最终用默认的 CentOS 自带的 mariadb-5.5.64 解决了。

7180 打开提示 HTTP/1.1 500 Server Error, Jetty

cloudera-scm-server.log 中显示 ScmActive IP address validation failed for, 原来是因为主机名配置错误,修改了主机名,但未配置 /etc/hosts,导致解析到公网的域名。

MySQL 开启 GTID 导致 无法初始化 DB 数据

使用 scm_prepare_database.sh 初始化 Cloudera Manager 的 数据结构时,提示 Unable to create/drop a table. java.sql.SQLException: Statement violates GTID consistency: CREATE TABLE ... SELECT.,意思是 开启了 MySQL 开启了 GTID,导致无法使用 CREATE TABLE ... SELECT. 的句式。

MySQL [(none)]> show global variables like 'gtid_%';
+----------------------------------+-------------------------------------------+
| Variable_name                    | Value                                     |
+----------------------------------+-------------------------------------------+
| gtid_executed                    | cda84aa8-54a2-11ea-bb46-525400de58d1:1-56 |
| gtid_executed_compression_period | 1000                                      |
| gtid_mode                        | ON                                        |
| gtid_owned                       |                                           |
| gtid_purged                      |                                           |
+----------------------------------+-------------------------------------------+
5 rows in set (0.00 sec)
1
2
3
4
5
6
7
8
9
10
11

因为 云 DB 无法关闭 GTID,所以直接在本地安装 MariaDB。

Agent 失联,无法获取状态

从 Agent 日志 /var/log/cloudera-scm-agent/cloudera-scm-agent.log 找到 报错 Error, CM server guid updated, expected 3fa66639-c4f2-4810-9dce-xxx, received xxx,删除 cm_guid 文件,重启 Agent 即可。

rm -rf /var/lib/cloudera-scm-agent/cm_guid
systemctl restart cloudera-scm-agent
1
2

reference