k8s 部署 Mysqld_exporter 以及添加告警规则

最近监控 mysql 数据库，用了 pmm-server、pmm-client 发现监控是真的不太好用，还是用回 prometheus 吧。

部署mysqld_exporter

k8s 部署最新版本的 mysqld_exporter，支持的数据库版本 MySQL >=5.6、MariaDB >= 10.3。

先在数据库创建用户以及授权

CREATE USER 'exporter'@'%' IDENTIFIED BY 'XXXXXXXX' WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';
FLUSH PRIVILEGES;

configmap

mysqld_exporter 的配置文件 configmap，cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:name: mysqld-exporter-confignamespace: monitoring
data:exporter.conf: |[client]user = exporterpassword = xxxxxxxx[client.servers]user = exporterpassword = xxxxxxxx

deployment\service

mysqld_exporter 的 deployment 和 service yaml 文件，deploy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:name: mysqld-exporternamespace: monitoringlabels:app: mysqld-exporter
spec:replicas: 1selector:matchLabels:app: mysqld-exportertemplate:metadata:labels:app: mysqld-exporterspec:containers:- name: mysqld-exporterimage: harbor.wenbo/dockerhub/prom/mysqld-exporter:v0.15.1ports:- containerPort: 9104resources:requests:memory: "1Gi"cpu: "0.5"           volumeMounts:- name: config-volumemountPath: /etc/exporter.confsubPath: exporter.confargs:- --config.my-cnf=/etc/exporter.confvolumes:- name: config-volumeconfigMap:name: mysqld-exporter-config---          
apiVersion: v1
kind: Service
metadata:name: mysqld-exporternamespace: monitoringlabels:app: mysqld-exporter
spec:ports:- port: 9104targetPort: 9104protocol: TCPname: httpselector:app: mysqld-exportertype: NodePort

prometheus 配置

在 prometheus 端，你可以按如下方式设置抓取配置

    - job_name: mysqld # To get metrics about the mysql exporter’s targetsparams:# Not required. Will match value to child in config file. Default value is `client`.auth_module: [client.servers]static_configs:- targets:# All mysql hostnames or unix sockets to monitor.- server1:3306- unix:///run/mysqld/mysqld.socklabels:mysql: alpha-master-72- targets:- server2:3306labels:mysql: alpha-slave1relabel_configs:- source_labels: [__address__]target_label: __param_target- source_labels: [__param_target]target_label: instance- target_label: __address__# The mysqld_exporter host:portreplacement: mysqld-exporter.monitoring.svc.cluster.local:9104

配置告警规则

这里用的是 kube-prometheus。配置了最大连接数、mysql、io thread、sql thread状态等告警规则。mysqld-exporter-prometheusrule.yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:#labels:#prometheus: k8s#role: alert-rulesname: prometheusrule-mysqld-exporter-alertrulesnamespace: monitoring
spec:groups:- name: mysqld-statusrules:- alert: MySQLMaxConnectionsexpr: (mysql_global_status_threads_connected / mysql_global_variables_max_connections) * 100 > 90for: 1mlabels:webhook: 'mysql'name: opsseverity: warningannotations:summary: "MySQL 最大连接数达到了 90%"description: '实例 {{$labels.instance}}，mysql: {{$labels.mysql}}，(目前使用:{{ printf "%0.2f" $value}}%)'- alert: MySQLSlaveIOThreadDownexpr: mysql_slave_status_slave_io_running != 1for: 1mlabels:webhook: "mysql"name: opsseverity: warningannotations:summary: "MySQL Slave I/O 线程停止"description: '实例 {{$labels.instance }}，mysql: {{$labels.mysql}}，I/O 线程已经停止运行.'- alert: MySQLSlaveSQLThreadDownexpr: mysql_slave_status_slave_sql_running != 1for: 1mlabels:webhook: 'mysql'severity: warningname: opsannotations:summary: "MySQL Slave SQL 线程停止"description: "实例 {{$labels.instance }}，mysql: {{$labels.mysql}}，SQL 线程已经停止运行."- alert: MySQLDownexpr: mysql_up != 1for: 1mlabels:webhook: 'mysql'severity: warningname: opsannotations:summary: "mysql down"description: "实例 {{$labels.instance }}, mysql: {{$labels.mysql}} down。"- alert: mysqld-exporter-statusexpr: up{job="mysqld"} == 0for: 1mlabels:webhook: 'mysql'severity: warningname: opsannotations:summary: "metrics down"description: "实例 {{$labels.instance }},mysql: {{$labels.mysql}} 数据源异常。"