【Ambari】Docker 安装Ambari 大数据单机版本

一、前期准备

1.1 部署 docker

1.2 部署 docker-compose

1.3 版本说明

二、镜像构建启动

2.1 系统镜像构建

2.2 安装包源镜像构建

2.3 kdc镜像构建

2.4 集群安装

2.5 容器导出为镜像

三、Ubuntu环境安装测试

3.1 环境准备

3.2 集群容器启动

一、前期准备

1.1 部署 docker

# 安装yum-config-manager配置工具
yum -y install yum-utils# 建议使用阿里云yum源：（推荐）
#yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo# 安装docker-ce版本
yum install -y docker-ce# 启动并开机启动
systemctl enable --now docker
docker --version
docker info

1.2 部署 docker-compose

curl -SL https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-composechmod +x /usr/local/bin/docker-compose
docker-compose --version

创建网络

docker network create hadoop-network

1.3 版本说明

组件	版本
操作系统	Centos7.6/Ubuntu20.04.5
ambari	2.7.4
HDP	3.1.4
HDP-GPL	3.1.4
HDP-UTILS	1.1.0.22
JDK	jdk-8u162-linux-x64

如下安装只做测试。

二、镜像构建启动

2.1 系统镜像构建

# 拉取 Centos 镜像
FROM centos:7.6.1810# 作者信息
LABEL  kangll # 安装基本工具
RUN yum update -y && yum install -y \net-tools \vim \wget \curl \lrzsz \ nc \openssl \sed \telnet \openssh-server \httpd \chronyd \zip unzip \openssh-clients \expect \krb5-server \krb5-libs \ krb5-workstation \&& yum clean all# JDK 
RUN mkdir -p /usr/java 
ADD ./config/jdk-8u162-linux-x64.tar.gz /usr/java/
ENV JAVA_HOME=/usr/java/jdk1.8.0_162
ENV CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
ENV PATH=$JAVA_HOME/bin:$PATH# 添加一个名为myuser的用户
RUN useradd -m winner_spark
# 设置myuser用户的密码，这里设置为mypassword
RUN echo 'winner_spark:123456' | chpasswd
# 设置myuser用户为sudoer
RUN echo 'winner_spark ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers# SSH
RUN sed -i '/^HostKey/'d /etc/ssh/sshd_config
RUN echo 'HostKey /etc/ssh/ssh_host_rsa_key' >> /etc/ssh/sshd_config
RUN mkdir /var/run/sshd/
RUN echo 'UseDNS no' >> /etc/ssh/sshd_config
RUN sed -i -e '/pam_loginuid.so/d' /etc/pam.d/sshdCOPY ./config/hosts    /etc/#set default password
RUN echo 'root:123456' | chpasswd
RUN /usr/bin/ssh-keygen -ACOPY ./config/krb5.conf   /etc#声明22端口 
EXPOSE 22RUN mkdir -p /opt && mkdir -p /var/log/supervisord
RUN echo '/usr/sbin/sshd -D' >> /opt/run.sh
RUN chmod +x /opt/run.sh
CMD ["/opt/run.sh"]#
WORKDIR /opt/

系统镜像构建

docker build -f  dockerfile-centos -t centos-jdk:1.0  .

2.2 安装包源镜像构建

dockerfile-httpd

FROM centos-jdk:1.0# centos7.6 JDK1.8
LABEL kangllRUN yum -y install httpd
RUN echo "crushlinux" >/var/www/html/index.html
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime# 添加yum 源
COPY ./repo/ambari.repo   /etc/yum.repos.d/
COPY ./repo/hdp.repo      /etc/yum.repos.d/
COPY ./repo/hdp.gpl.repo  /etc/yum.repos.d/RUN mkdir -p /hadoop/hdp
ADD ./HDP/ambari.tar.gz    /var/www/html
ADD ./HDP/HDP-GPL.tar.gz   /var/www/html
ADD ./HDP/HDP.tar.gz       /var/www/html
ADD ./HDP/HDP-UTILS.tar.gz /var/www/htmlEXPOSE 80CMD ["httpd","-DFOREGROUND"]

源镜像构建

docker build -t ambari:2.7.4  -f dockerfile-ambari  .

2.3 kdc镜像构建

dockerfile-kerberos

FROM centos:7.6.1810RUN yum install -y krb5-server krb5-libs krb5-auth-dialog krb5-workstationRUN yum install -y net-tools telnetCOPY ./config/krb5.conf   /etc
COPY ./config/kdc.conf    /var/kerberos/krb5kdc/
COPY ./config/kadm5.acl   /var/kerberos/krb5kdc/CMD ["/usr/sbin/init"]

服务启动

# 进入容器 启动服务
Systemctl restart krb5kdc
Systemctl restart kadmin

docker-compose-base.yml 启动基础环境

version: '3'
services:mysql:image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/mysql-test:5.7restart: alwaysnetworks:- hadoop-networkcontainer_name: mysqlhostname: mysql-testenvironment:MYSQL_ROOT_PASSWORD: Winner001TZ: Asia/Shanghaiports:- 3306:3306volumes:-  /data:/var/lib/mysql- ./conf/my.cnf:/etc/mysql/my.cnfcommand:--max_connections=1000--character-set-server=utf8mb4--collation-server=utf8mb4_general_ci--default-authentication-plugin=mysql_native_passwordhealthcheck:test: ["CMD-SHELL", "curl --silent localhost:3306 >/dev/null || exit 1"]interval: 10stimeout: 10shttpd:container_name: httpdhostname: httpdimage: httpd:1.0networks:- hadoop-networkports:- "80:80"privileged: truerestart: alwaysvolumes:- "/etc/hosts:/etc/hosts"kdc:image: kerberos:1.0restart: alwaysnetworks:- hadoop-networkcontainer_name: kerberoshostname: kerberos.winnerports:- 88:88- 749:749- 750:750volumes:- /etc/hosts:/etc/hosts- /hadoop/kerberos/krb5kdc:/var/kerberos/krb5kdc- /hadoop/kerberos/keytabs:/etc/security/keytabscommand:- /bin/sh- -c- |/opt/apache/kerberosStart.shnetworks:hadoop-network:external:name: hadoop-network

Ambari启动

# 启动基础环境
docker-compose -f docker-compose-base.yaml up -d# 启动ambari , 在此容器中安装集群
docker-compose -f docker-compose-ambari.yaml up -d

2.4 集群安装

repo地址

节点和秘钥

agent安装

服务安装

由于是单机直接下一步

安装组件

输入密码

数据目录确认

用户确认下一步

服务参数确定

发布

安装完成

2.5 容器导出为像

docker ps -a
docker commit  容器ID ambari-1.0.tar.gz

三、Ubuntu环境安装测试

3.1 环境准备

我们准备Ubuntu 环境，在Ubuntu环境安装Ambari单机，相当于在Centos服务器安装的docker化的Ambari 导出为镜像然后在Ubutntu 环境尝试启动。

准备Ubuntu 环境，版本：Ubuntu 20.04.5 LTS

将镜像 ambari-1.0.tar.gz 加载到Ubuntu 环境

docker load -i ambari-1.0.tar.gz

MySQL元数据库备份还原（Ambari、hive、ranger）

HDFS、kerberos 备份的数据拷贝到Ubuntu服务器并保持数据文件路径不变。

将在centos 安装完成的集群备份元数据库写入到新的库中 , 也就是Ubuntu环境要使用的元数据库。

# 备份的元数据写入新的数据库中
mysql -hmysql-test -uroot -pWinner001 ambari < ambari.sql
mysql -hmysql-test -uroot -pWinner001 ranger < ranger.sql
mysql -hmysql-test -uroot -pWinner001 hive   < hive.sql

备份的kerberos和安装包解压

# 解压到 /usr 路径下，保存的是集群安装的数据
hdp.tar.gz# 在根路径解压保留的kerberos 和HDFS 初始化的元数据
hadoop.tar.gz

基础环境启动docker-compose-base.yaml

version: '3'
services:mysql:image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/mysql-test:5.7restart: alwaysnetworks:- hadoop-networkcontainer_name: mysqlhostname: mysql-testenvironment:MYSQL_ROOT_PASSWORD: Winner001TZ: Asia/Shanghaiports:- 3306:3306volumes:-  /data:/var/lib/mysql- ./conf/my.cnf:/etc/mysql/my.cnfcommand:--max_connections=1000--character-set-server=utf8mb4--collation-server=utf8mb4_general_ci--default-authentication-plugin=mysql_native_passwordhealthcheck:test: ["CMD-SHELL", "curl --silent localhost:3306 >/dev/null || exit 1"]interval: 10stimeout: 10shttpd:container_name: httpdhostname: httpdprivileged: trueimage: httpd:1.0networks:- hadoop-networkports:- "80:80"privileged: truerestart: alwaysvolumes:- "/etc/hosts:/etc/hosts"kdc:image: kerberos:1.0restart: alwaysprivileged: truenetworks:- hadoop-networkcontainer_name: kerberoshostname: kerberos.winnerports:- 888:88- 8749:749- 8750:750volumes:- /etc/hosts:/etc/hosts- /hadoop/kerberos/krb5kdc:/var/kerberos/krb5kdc- /hadoop/kerberos/keytabs:/etc/security/keytabsnetworks:hadoop-network:external:name: hadoop-network

启动

# 启动
docker-compose -f docker-compose-base.yaml up -d# 停止
docker-compose -f docker-compose-base.yaml down

3.2 集群容器启动

镜像

启动容器

version: '3'
services:ambari:image: ambari:2.7.4  # 修改为新的容器container_name: ambarihostname: ambari-serverrestart: alwaysprivileged: truenetworks:- hadoop-networkports:- 8080:8080- 8088:8088- 50070:50070- 50075:50075- 8188:8188- 8042:8042- 10000:10000- 9083:9083- 6080:6080- 16010:16010volumes:- /etc/hosts:/etc/hosts- /usr/hdp:/usr/hdp- /hadoop/hdfs:/hadoop/hdfs- /hadoop/etc/hadoop/:/etc/hadoop/- /hadoop/etc/hbase/:/etc/hbase/- /hadoop/etc/hive/:/etc/hive/- /hadoop/etc/livy2/:/etc/livy2/- /hadoop/etc/spark2/:/etc/spark2/- /hadoop/etc/tez/:/etc/tez/- /hadoop/etc/zookeeper/:/etc/zookeeper/- /hadoop/yarn:/hadoop/yarn- /hadoop/kerberos/krb5kdc:/var/kerberos/krb5kdc- /hadoop/kerberos/keytabs:/etc/security/keytabscommand:- /bin/sh- -c- |/opt/apache/bootstrap.sh
networks:hadoop-network:external:name: hadoop-network

启动

docker-compose -f docker-compose-ambari.yaml up -d

下次安装直接加载镜像，然后执行yaml文件即可

docker load -i ambari-1.0.tar.gz
docker load -i kerberos.tar.gz
docker load -i mysql.tar.gz
docker load -i HDP-REPO.tar.gz

脚本

bootstrap.sh

#!/bin/bashset -xambari_db=ambari
myurl=mysql-test
myuser=root
mypwd=Winner001
myport=3306
mydb=ipvacloud
java_home=/usr/java/jdk1.8.0_162
mysql_driver=/usr/share/java/mysql-connector-java.jarBASEDIR=$(cd "$(dirname "$0")"; pwd)wait_for() {echo Waiting for $1 to listen on $2...while ! nc -z $1 $2; do echo waiting...; sleep 1s; done
}setup_ambari() {# 强制替换文件cp -f /tmp/ambari.properties /etc/ambari-server/conf/ cp -f /tmp/password.dat     /etc/ambari-server/conf/cp -f /tmp/ambari-agent.ini    /etc/ambari-agent/conf/
}# wait for mysql and httpd
#wait_for windp-aio 3306
#wait_for httpd 80# 开始ambari安装
yum -y install ambari-server
yum -y install ambari-agentre=`mysql -h${myurl} -u${myuser} -p${mypwd} -e "show databases"|grep ambari|wc -l`
if [ $re -eq 0 ];then# sql 初始化mysql -h${myurl} -u${myuser} -p${mypwd} < /tmp/init_db.sql mysql -h${myurl} -u${myuser} -p${mypwd} ambari  < /tmp/Ambari-DDL-MySQL-CREATE.sql
fi# 配置
setup_ambari# 启动ambari服务
/usr/sbin/ambari-server start
/usr/sbin/ambari-agent start/usr/sbin/sshd -Dtail -f /var/log/ambari-server/ambari-server.log

参考文章：https://www.cnblogs.com/liugp/p/17489616.html

Ubuntu 环境安装docker: 【云计算】Ubuntu多种安装docker方式_ubuntu安装docker-CSDN博客

【Ambari】Docker 安装Ambari 大数据单机版本

一、前期准备

1.1 部署 docker

1.2 部署 docker-compose

1.3 版本说明

二、镜像构建启动

2.1 系统镜像构建

2.2 安装包源镜像构建

2.3 kdc镜像构建

2.4 集群安装

2.5 容器导出为像

三、Ubuntu环境安装测试

3.1 环境准备

3.2 集群容器启动

最新新闻

热搜词

【Ambari】Docker 安装Ambari 大数据单机版本

一、前期准备

1.1 部署 docker

1.2 部署 docker-compose

1.3 版本说明

二 、镜像构建启动

2.1 系统镜像构建

2.2 安装包源镜像构建

2.3 kdc镜像构建

2.4 集群安装

2.5 容器导出为像

三、Ubuntu环境安装测试

3.1 环境准备

3.2 集群容器启动

最新新闻

热搜词

二、镜像构建启动