YARN Components
YARN 的组件:
- ResourceManager(Master)
- NodeManager(Slave)
YARN Schedule
Common Properties
# YARN 程序依赖的路径
yarn.application.classpath /conf,/*,...
# YARN nodemanager 日志存放路径，对应 Spark standalone Env : SPARK_WORKER_DIR
yarn.nodemanager.log-dirs /hadoop/yarn/log,/mnt/disk1/hadoop/yarn/log
# YARN Application web 地址
yarn.resourcemanager.webapp.address 0.0.0.0:8088
YARN Logs
YARN Application 日志分为两部分：jobhistory和container日志。
jobhistory：应用程序运行日志，包括应用程序启动时间、结束时间，每个任务的启动时间、结束时间，各种 counter 信息等。container：包含 ApplicationMaster 日志和普通 Task 日志，比如用户自己输出的日志。日志目录位于${yarn.nodemanager.log-dirs}/application_${appid}/container_${container_id}，container 日志目录下包含三个日志文件：stdout,stderr,syslog。
YARN 环境变量和日志存放目录相关的还有export YARN_LOG_DIR=/$USER
Enable CPU Scheduling and CPU Isolation- HDP
- Hadoop Capacity Scheduler
- Using CGroups with YARN
- Enabling CGroups - HDP
- Enable CPU scheduling - HDP
- Configuring YARN container executor
- why container-executor.cfg is overwritten when rest
- HDP 2.3.4 min_user_id in container-executor.cfg does not take effect
step1
From the Ambari web interface, select theYARNservice, and then select theConfig->Settings->CPU->Node. EnableCPU SchedulingandCPU Isolation.
step2
From the Ambari web interface, select the YARN service, and then select theConfig->Advancedtab. Search and check properties below, if the properties don’t exist, just Add Property in Custom yarn-site
yarn.nodemanager.local-dirs=/hadoop/yarn/local
yarn.nodemanager.log-dirs=/hadoop/yarn/log
# DefaultResourceCalculator only uses Memory while DominantResourceCalculator uses Dominant-resource to
# compare multi-dimensional resources such as Memory, CPU etc.
yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor
yarn.nodemanager.linux-container-executor.group=hadoop
yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler
yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/yarn
yarn.nodemanager.linux-container-executor.cgroups.mount=false
yarn.nodemanager.linux-container-executor.cgroups.mount-path=/cgroup
# 以 hive 用户执行提交的任务
# 注意：不能指定 banned.users 列表中的用户，默认 banned.users=hdfs,yarn,mapred,bin
# 上面的参数可以通过 /etc/hadoop/conf/container-executor.cfg 查看
yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user=hive
在每个NodeManager节点执行下面的操作，路径规则${yarn.nodemanager.linux-container-executor.cgroups.mount-path/${resources_name}/${yarn.nodemanager.linux-container-executor.cgroups.hierarchy}}
# Configure cgroups hierarchy
mkdir -p /sys/fs/cgroup/cpu/yarn
chown -R yarn /sys/fs/cgroup/cpu/yarn
mkdir -p /sys/fs/cgroup/memory/yarn
chown -R yarn /sys/fs/cgroup/memory/yarn
mkdir -p /sys/fs/cgroup/blkio/yarn
chown -R yarn /sys/fs/cgroup/blkio/yarn
mkdir -p /sys/fs/cgroup/net_cls/yarn
chown -R yarn /sys/fs/cgroup/net_cls/yarn
# 保证 yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user 指定的用户
# 有读写 yarn.nodemanager.local-dirs 和 yarn.nodemanager.log-dirs 的权限
chmod -R 777 /hadoop/yarn/local
chmod -R 777 /hadoop/yarn/log
