YARN

YARN

YARN 的组件:

  • ResourceManager(Master)
  • NodeManager(Slave)

common properties

# YARN 程序依赖的路径
yarn.application.classpath  /conf,/*,...

# YARN nodemanager 日志存放路径,对应 Spark standalone Env : SPARK_WORKER_DIR
yarn.nodemanager.log-dirs   /hadoop/yarn/log,/mnt/disk1/hadoop/yarn/log

# YARN Application web 地址
yarn.resourcemanager.webapp.address 0.0.0.0:8088

Enable CPU Scheduling and CPU Isolation- HDP

step1

From the Ambari web interface, select theYARNservice, and then select theConfig->Settings->CPU->Node. EnableCPU SchedulingandCPU Isolation.

step2

From the Ambari web interface, select the YARN service, and then select theConfig->Advancedtab. Search and check properties below, if the properties don’t exist, just Add Property in Custom yarn-site

yarn.nodemanager.local-dirs=/hadoop/yarn/local
yarn.nodemanager.log-dirs=/hadoop/yarn/log

# DefaultResourceCalculator only uses Memory while DominantResourceCalculator uses Dominant-resource to 
# compare multi-dimensional resources such as Memory, CPU etc.
yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor
yarn.nodemanager.linux-container-executor.group=hadoop
yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler
yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/yarn
yarn.nodemanager.linux-container-executor.cgroups.mount=false
yarn.nodemanager.linux-container-executor.cgroups.mount-path=/cgroup
# 以 hive 用户执行提交的任务
# 注意:不能指定 banned.users 列表中的用户,默认 banned.users=hdfs,yarn,mapred,bin
# 上面的参数可以通过 /etc/hadoop/conf/container-executor.cfg 查看
yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user=hive

在每个NodeManager节点执行下面的操作,路径规则${yarn.nodemanager.linux-container-executor.cgroups.mount-path/${resources_name}/${yarn.nodemanager.linux-container-executor.cgroups.hierarchy}}

# Configure cgroups hierarchy
mkdir -p /sys/fs/cgroup/cpu/yarn
chown -R yarn /sys/fs/cgroup/cpu/yarn
mkdir -p /sys/fs/cgroup/memory/yarn
chown -R yarn /sys/fs/cgroup/memory/yarn
mkdir -p /sys/fs/cgroup/blkio/yarn
chown -R yarn /sys/fs/cgroup/blkio/yarn
mkdir -p /sys/fs/cgroup/net_cls/yarn
chown -R yarn /sys/fs/cgroup/net_cls/yarn

# 保证 yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user 指定的用户
# 有读写 yarn.nodemanager.local-dirs 和 yarn.nodemanager.log-dirs 的权限
chmod -R 777 /hadoop/yarn/local
chmod -R 777 /hadoop/yarn/log

YARN 日志管理

YARN Application 日志分为两部分:jobhistorycontainer日志。

  • jobhistory:应用程序运行日志,包括应用程序启动时间、结束时间,每个任务的启动时间、结束时间,各种 counter 信息等。
  • container:包含 ApplicationMaster 日志和普通 Task 日志,比如用户自己输出的日志。日志目录位于${yarn.nodemanager.log-dirs}/application_${appid}/container_${container_id},container 日志目录下包含三个日志文件:stdout,stderr,syslog

YARN 环境变量和日志存放目录相关的还有export YARN_LOG_DIR=/$USER

打赏一个呗

取消

感谢您的支持,我会继续努力的!

扫码支持
扫码支持
扫码打赏,一毛也是爱

打开支付宝扫一扫,即可进行扫码打赏哦