周海汉?2013.7.24 http://abloz.com 假如设备链接层次分3层，第一层交换机d1下面连多个交换机rk1,rk2,rk3,rk4,. 每个交换机对应一个机架。 d1(rk1(hs11,hs12,),rk2(hs21,hs22,), rk3(hs31,hs32,),rk4(hs41,hs42,),) 可以用程序或脚本完成由host到设备的映射
周海汉?2013.7.24
http://abloz.com
假如设备链接层次分3层，第一层交换机d1下面连多个交换机rk1,rk2,rk3,rk4,…. 每个交换机对应一个机架。
d1(rk1(hs11,hs12,…),rk2(hs21,hs22,…), rk3(hs31,hs32,…),rk4(hs41,hs42,…),…)
可以用程序或脚本完成由host到设备的映射。比如，用python，生成一个topology.py：
然后在core-site.xml中配置
topology.script.file.name
/home/hadoop/hadoop-1.1.2/conf/topology.py
the script name that should be invoked to resolve dns names to
networktopology names. example: the script would take host.foo.bar as an
argument, and return /rack1 as the output.
python机架脚本：
[hadoop@hs11 conf]$ cat topology.py
#!/usr/bin/env python
”’
this script used by hadoop to determine network/rack topology. it
should be specified in hadoop-site.xml via topology.script.file.name
property.
topology.script.file.name
/home/hadoop/hadoop-1.1.2/conf/topology.py
to generate dict:
for i in range(xx):
#print “\”hs%d\”:\”/rk%d/hs%d\”,”%(i,(i-1)/10,i)
print “\”hs%d\”:\”/rk%d\”,”%(i,(i-1)/10)
andy 2013.7.23
”’
import sys
from string import join
default_rack = ‘/rk0′;
rack_map = {
“hs11″:”/rk1″,
“hs12″:”/rk1″,
“hs13″:”/rk1″,
“hs14″:”/rk1″,
“hs15″:”/rk1″,
“hs16″:”/rk1″,
“hs17″:”/rk1″,
“hs18″:”/rk1″,
“hs19″:”/rk1″,
“hs20″:”/rk1″,
“hs21″:”/rk2″,
“hs22″:”/rk2″,
“hs23″:”/rk2″,
“hs24″:”/rk2″,
“hs25″:”/rk2″,
“hs26″:”/rk2″,
“hs27″:”/rk2″,
“hs28″:”/rk2″,
“hs29″:”/rk2″,
“hs30″:”/rk2″,
“hs31″:”/rk3″,
“hs32″:”/rk3″,
“hs33″:”/rk3″,
“hs34″:”/rk3″,
“hs35″:”/rk3″,
“hs36″:”/rk3″,
“hs37″:”/rk3″,
“hs38″:”/rk3″,
“hs39″:”/rk3″,
“hs40″:”/rk3″,
“hs41″:”/rk4″,
“hs42″:”/rk4″,
“hs43″:”/rk4″,
“hs44″:”/rk4″,
“hs45″:”/rk4″,
“hs46″:”/rk4″,
…
“10.10.20.11″:”/rk1″,
“10.10.20.12″:”/rk1″,
“10.10.20.13″:”/rk1″,
“10.10.20.14″:”/rk1″,
“10.10.20.15″:”/rk1″,
“10.10.20.16″:”/rk1″,
“10.10.20.17″:”/rk1″,
“10.10.20.18″:”/rk1″,
“10.10.20.19″:”/rk1″,
“10.10.20.20″:”/rk1″,
“10.10.20.21″:”/rk2″,
“10.10.20.22″:”/rk2″,
“10.10.20.23″:”/rk2″,
“10.10.20.24″:”/rk2″,
“10.10.20.25″:”/rk2″,
“10.10.20.26″:”/rk2″,
“10.10.20.27″:”/rk2″,
“10.10.20.28″:”/rk2″,
“10.10.20.29″:”/rk2″,
“10.10.20.30″:”/rk2″,
“10.10.20.31″:”/rk3″,
“10.10.20.32″:”/rk3″,
“10.10.20.33″:”/rk3″,
“10.10.20.34″:”/rk3″,
“10.10.20.35″:”/rk3″,
“10.10.20.36″:”/rk3″,
“10.10.20.37″:”/rk3″,
“10.10.20.38″:”/rk3″,
“10.10.20.39″:”/rk3″,
“10.10.20.40″:”/rk3″,
“10.10.20.41″:”/rk4″,
“10.10.20.42″:”/rk4″,
“10.10.20.43″:”/rk4″,
“10.10.20.44″:”/rk4″,
“10.10.20.45″:”/rk4″,
“10.10.20.46″:”/rk4″,
…
}
if len(sys.argv)==1:
print default_rack
else:
print join([rack_map.get(i, default_rack) for i in sys.argv[1:]],” “)
原来这个程序我返回的是
“hs11″:”/rk1/hs11″,
结果执行mapreduce程序时报如下错误：
total mapreduce jobs = 1
launching job 1 out of 1
number of reduce tasks is set to 0 since there’s no reduce operator
starting job = job_201307241502_0003, tracking url = http://hs11:50030/jobdetails.jsp?jobid=job_201307241502_0003
kill command = /home/hadoop/hadoop-1.1.2/libexec/../bin/hadoop job? -kill job_201307241502_0003
hadoop job information for stage-1: number of mappers: 0; number of reducers: 0
2013-07-24 18:38:11,854 stage-1 map = 100%,? reduce = 100%
ended job = job_201307241502_0003 with errors
error during job, obtaining debugging information…
job tracking url: http://hs11:50030/jobdetails.jsp?jobid=job_201307241502_0003
failed: execution error, return code 2 from org.apache.hadoop.hive.ql.exec.mapredtask
mapreduce jobs launched:
job 0:? hdfs read: 0 hdfs write: 0 fail
total mapreduce cpu time spent: 0 msec
通过http://hs11:50030/jobdetails.jsp?jobid=job_201307241502_0002?可以看到：
job initialization failed:
java.lang.nullpointerexception
at?org.apache.hadoop.mapred.jobtracker.resolveandaddtotopology(jobtracker.java:2751)
at?org.apache.hadoop.mapred.jobinprogress.createcache(jobinprogress.java:578)
at?org.apache.hadoop.mapred.jobinprogress.inittasks(jobinprogress.java:750)
at org.apache.hadoop.mapred.jobtracker.initjob(jobtracker.java:3775)
at?org.apache.hadoop.mapred.eagertaskinitializationlistener$initjob.run(eagertaskinitializationlistener.java:90)
at?java.util.concurrent.threadpoolexecutor$worker.runtask(threadpoolexecutor.java:886)
at?java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:908)
at java.lang.thread.run(thread.java:662)
原来系统在配置机架敏感时，并不需要在脚本中返回设备ns或hostname，系统会自动添加。改为上面的topology.py后，系统执行正确。
相关博文：
hadoop 打印配置变量hadoop 中的 classnotfoundexceptionhadoop ubuntu集群安装 原文地址：hadoop 配置机架感知, 感谢原作者分享。