目录
一、概述
1.1 promethues简介
1.2 JMX Exporter简介
二、监控SparkHistoryServer实现
一、概述
1.1 promethues简介
promethues采集数据的方法很多,常用的是通过各种exporter去主机采集,然后有些程序是没有相关的exporter,所以有些时候会通过脚本采集服务的数据,还有一些程序的metrics自身也没有暴露,通过 脚本采集的方法也不常用,工作中java程序又比较常见,所以有人开发了jmx_prometheus_javaagent.jar专门用来采集java程序metrics的。
1.2 JMX Exporter简介
Java Management Extensions,JMX 是管理 Java 的一种扩展框架,JMX Exporter 基于此框架读取 JVM 的运行时状态。JMX Exporter 利用 Java 的 JMX 机制来读取 JVM 运行时的监控数据,然后将其转换为 Prometheus 可辨识的 metrics 格式,以便让 Prometheus 对其进行监控采集。
JMX Exporter 提供启动独立进程及 JVM 进程内启动(in-process)两种方式暴露 JVM 监控指标:
1. 启动独立进程 JVM 启动时指定参数,暴露 JMX 的 RMI 接口。JMX Exporter 调用 RMI 获取 JVM 运行时状态数据,转换为 Prometheus metrics 格式,并暴露端口让 Prometheus 采集。
2. JVM 进程内启动(in-process) JVM 启动时指定参数,通过 javaagent 的形式运行 JMX Exporter 的 jar 包,进程内读取 JVM 运行时状态数据,转换为 Prometheus metrics 格式,并暴露端口让 Prometheus 采集。
二、监控SparkHistoryServer实现
prometheus.yml
# my global config
global:scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.# scrape_timeout is set to the global default (10s).# Alertmanager configuration
alerting:alertmanagers:- static_configs:- targets:- localhost:9093# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:# - "first_rules.yml"# - "second_rules.yml"- "alert_rules/*.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:- job_name: 'prometheus'# metrics_path defaults to '/metrics'# scheme defaults to 'http'.static_configs:- targets: ['localhost:9090']- job_name: 'grafana'# metrics_path defaults to '/metrics'# scheme defaults to 'http'.static_configs:- targets: ['localhost:3000']- job_name: 'alertmanager'# metrics_path defaults to '/metrics'# scheme defaults to 'http'.static_configs:- targets: ['localhost:9093']- job_name: 'sparkhistoryserver' # historyserverfile_sd_configs:- files:- configs/sparkhistoryserver.json
configs/sparkhistoryserver.json 监控目标
[root@ddp01 prometheus]# cat configs/sparkhistoryserver.json
[{"targets":["ddp01:18082"]}
]
alert_rules/spark3.yml 定义告警规则
[root@ddp01 prometheus]# cat alert_rules/spark3.yml
groups:
- name: SPARK3# rules:定义规则rules:# alert:告警规则的名称- alert: SparkHistoryServer进程存活expr: up{job="sparkhistoryserver"} != 1for: 15slabels:# severity: 指定告警级别。有三种等级,分别为warning、critical和emergency。严重等级依次递增。severity: exceptionclusterId: 1serviceRoleName: SparkHistoryServerannotations:# summary描述告警的概要信息# description用于描述告警的详细信息。summary: 重新启动description: "{{ $labels.job }}的{{ $labels.instance }}实例产生告警"
增加 SPARK_HISTORY_OPTS 环境变量, 也就是将spark程序与监控插件绑定
[root@ddp01 datasophon]# cat spark3/conf/spark-env.sh
export SPARK_DIST_CLASSPATH=$(/opt/datasophon/hadoop-3.3.3/bin/hadoop classpath)
export HADOOP_CONF_DIR=/opt/datasophon/hadoop-3.3.3/etc/hadoop
export YARN_CONF_DIR=/opt/datasophon/hadoop-3.3.3/etc/hadoop
export SPARK_CLASSPATH=/opt/datasophon/spark-3.1.3/carbonlib/*
export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -javaagent:/opt/datasophon/spark-3.1.3/jmx/jmx_prometheus_javaagent-0.16.1.jar=18082:/opt/datasophon/spark-3.1.3/jmx/prometheus_config.yml"
jmx_prometheus_javaagent-0.16.1.jar 包位置
查看进程状态
rules
targats
原文链接:prometheus通过jmx_prometheus_javaagent-.jar采集java程序的metrics_jmx promethues-CSDN博客