码迷,mamicode.com
首页 > 其他好文 > 详细

Centos7 安装部署 Airflow

时间:2019-08-20 18:35:25      阅读:86      评论:0      收藏:0      [点我收藏+]

标签:xpl   log   eps   def   monitor   没有   rpm -ivh   meta   环境   

本人在centos7 的环境下部署,怎么在centos7 下配置静态 IP 关闭防火墙 以及安装jdk在这里不多赘述,

centos7 配置静态ip可以参考:https://www.cnblogs.com/braveym/p/8523100.html  和  https://www.cnblogs.com/braveym/p/9096402.html

 

Airflow 基础安装

1、默认自带python2环境,自行安装pip
sudo yum -y install epel-release
sduo yum install python-pip

技术图片

 

 技术图片

 

 

 

 

 技术图片

技术图片

 

 

2、进行pip的更新,否则很多安装会报错
sudo pip install --upgrade pip
sudo pip install --upgrade setuptools

 

 技术图片

 

 技术图片

 

 

3、安装开发库
sudo yum install python-devel
sudo yum install libevent-devel
sudo yum install mysql-devel

 

技术图片

 

 

 技术图片

 

 技术图片

 

 

4、安装centos7下的mysql

在安装mysql前建议先更新一下源,不然下载老失败

 

打开centos的yum文件夹(需要root权限或者拥有该目录的操作权限才可以)

输入命令cd  /etc/yum.repos.d

技术图片

 

用wget下载repo文件

输入命令sudo wget  http://mirrors.aliyun.com/repo/Centos-7.repo(如果wget命令不生效,说明还没有安装wget工具,输入sudo yum -y install wget 回车进行安装)

技术图片

 

 备份系统原来的repo文件

sudo mv  CentOS-Base.repo CentOS-Base.repo.bak

技术图片

 

 

 

替换系统原理的repo文件

sudo wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.163.com/.help/CentOS7-Base-163.repo

 技术图片

 

 在更新源之前先把自带的mariadb删除,会报类似这样的错误

 

技术图片

 

 

 

 技术图片

[bigdata@bigdata ~]$ rpm -qa|grep mariadb
mariadb-libs-5.5.60-1.el7_5.x86_64
[bigdata@bigdata ~]$ rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64
error: cant create transaction lock on /var/lib/rpm/.rpm.lock (Permission denied)
[bigdata@bigdata ~]$ sudo rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64
[bigdata@bigdata ~]$ rpm -qa|grep mariadb
[bigdata@bigdata ~]$ 

 

 

 

执行yum源更新命令

sudo yum clean all

sudo yum makecache

sudo yum -y update (需要比较长的时间)

 技术图片

 

 

 在https://dev.mysql.com/downloads/mysql/下载mysql的离线安装包

 技术图片

 

 

 下载完之后把包上传到集群

技术图片

 

 解压

技术图片

 

 把以上解压包每一个都安装一遍,但是,安装的时候建议安装顺序来,不要随便拿起一个就安装,不然会报这样的错误

技术图片

 

 所以建议按照下面的顺序进行安装

技术图片


bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-common-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-common-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-common-5.7.27-1.e################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-libs-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-libs-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-libs-5.7.27-1.el7################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-libs-compat-5.7.2################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-embedded-compat-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-embedded-compat-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-embedded-compat-5################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-devel-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-devel-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-devel-5.7.27-1.el################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-client-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-client-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-client-5.7.27-1.e################################# [100%]
[bigdata@bigdata mysql]$ sudo rpm -ivh mysql-community-server-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-server-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:mysql-community-server-5.7.27-1.e################################# [100%]
[bigdata@bigdata mysql]$

 

先关闭mysql服务

技术图片

 

 

修改MySql配置

技术图片


skip-grant-tables #添加这句话,这时候登入mysql就不需要密码

 技术图片

 

启动mysql

技术图片

[bigdata@bigdata mysql]$ sudo systemctl start mysqld.service
[bigdata@bigdata mysql]$ sudo systemctl status mysqld.service
● mysqld.service - MySQL Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-08-20 11:49:49 CST; 3s ago
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
  Process: 4759 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)
  Process: 4679 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
 Main PID: 4762 (mysqld)
    Tasks: 27
   CGroup: /system.slice/mysqld.service
           └─4762 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid

Aug 20 11:49:45 bigdata systemd[1]: Starting MySQL Server...
Aug 20 11:49:49 bigdata systemd[1]: Started MySQL Server.

 

 

登录mysql,输入密码处直接回车就可以

技术图片

 

 退出并停止Mysql服务

技术图片

 

 

 修改配置文件

技术图片

 

设置MySql开机自启

sudo systemctl enable mysqld

 

 

 

5、初始化配置mysql,可以设定允许root远程登录

先通过root登录mysql,密码是刚刚给root用户设置的密码我这里是:root

技术图片

 

 Mysql默认不允许远程登录,我们需要设置下,并且防火墙开放3306端口;(备注 mysql5.7默认密码策略要求密码必须是大小写字母数字特殊字母的组合,至少8位)

 技术图片

可以看到,太简单的密码是不可行的,这里需要注意的!!!

mysql> GRANT ALL PRIVILEGES ON *.* TO root@% IDENTIFIED BY root WITH GRANT OPTION;
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
mysql> GRANT ALL PRIVILEGES ON *.* TO root@% IDENTIFIED BY root WITH GRANT OPTION;
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
mysql> GRANT ALL PRIVILEGES ON *.* TO root@% IDENTIFIED BY Root123 WITH GRANT OPTION;
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
mysql> GRANT ALL PRIVILEGES ON *.* TO root@% IDENTIFIED BY Admin123! WITH GRANT OPTION;
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> exit;
Bye

 

 

对于防火墙开启的情况下,就需要开发3306端口,我这里是把防火墙关闭

技术图片

[bigdata@bigdata mysql]$ sudo firewall-cmd --zone=public --add-port=3306/tcp --permanent
FirewallD is not running
[bigdata@bigdata mysql]$ sudo firewall-cmd --reload
FirewallD is not running

 

 

6、配置Mysql默认编码为UTF-8

 修改/etc/my.cnf配置文件,在[mysqld]下添加编码配置

 技术图片

 

技术图片

character_set_server=utf8
init_connect=SET NAMES utf8

 

 编辑保存完 重启mysql服务: sudo systemctl restart mysqld.service

 技术图片

 

 查看下编码:

 技术图片

[bigdata@bigdata mysql]$ mysql -u root -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.27 MySQL Community Server (GPL)

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type help; or \h for help. Type \c to clear the current input statement.

mysql> show variables like %character%;
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | utf8                       |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | utf8                       |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.01 sec)

mysql> 

 

 

 

7、安装AirFlow

给airflow设置环境变量

sudo vim /etc/profile

技术图片

#airflow
export AIRFLOW_HOME=/opt/modules/airflow
export SLUGIFY_USES_TEXT_UNIDECODE=yes

 

是环境变量生效

技术图片

 

 安装airflow

技术图片

sudo pip install apache-airflow

 

 

可以看到出现以下错误

技术图片

 

 这样解决:

sudo pip install six --user -U
sudo pip install ipython --user -U
sudo pip install --ignore-installed dnspython

  sudo yum install python-devel

  sudo yum install openldap-devel

  sudo pip install --ignore-installed python-ldap

  sudo pip install --ignore-installed enum34

 

 

前面的问题解决了,又开始新的问题

技术图片

 

 这个问题这样解决:

sudo pip install --ignore-installed requests

 

 

最后再重新安装一次airflow,可以看到成功了

技术图片

 

 

8、 设置mysql数据库连接

sudo pip install apache-airflow[mysql]

 

技术图片

技术图片

 

 创建airflow用户,创建airflow数据库并给出所有权限给此用户(注意在设置密码的时候还是前面讲的问题,不能太简单,否则不能通过)

技术图片

mysql> create database airflow;
Query OK, 1 row affected (0.00 sec)

mysql> create user airflow@% identified by AirFlow123!;
Query OK, 0 rows affected (0.02 sec)

mysql> GRANT all privileges on airflow.* TO airflow@%  IDENTIFIED BY AirFlow123!;
Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

 

 

修改airflow配置文件,指向数据库mysql
airflow.cfg 文件通常在~/airflow目录下

 

技术图片

 

 技术图片

 

 技术图片

executor =  LocalExecutor
sql_alchemy_conn = mysql://airflow:AirFlow123!@192.168.189.11/airflow
对应字段解释如下: dialect+driver://username:password@host:port/database

 

 修改mysql的配置文件 

 vim /etc/my.cnf

 添加这一句

技术图片

explicit_defaults_for_timestamp=1

 

 

 重启mysql

技术图片

 

 

 9、初始化airflow并启动web服务

sudo airflow initdb
sudo airflow webserver
sudo airflow scheduler
sudo airflow worker 

技术图片

 

 技术图片

 

 

 在浏览器打开地址 192.168.189.11:8080

技术图片

 

Centos7 安装部署 Airflow

标签:xpl   log   eps   def   monitor   没有   rpm -ivh   meta   环境   

原文地址:https://www.cnblogs.com/braveym/p/11378851.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!