你好,游客 登录 注册 搜索
背景:
阅读新闻

windows配置hadoop-1.1.0的伪分布式环境(续)

[日期:2013-03-24] 来源:  作者: [字体: ]

在前一篇文章中,介绍了一写常见问题的解决方法。

 

但是,当我重装系统,再次按照前面一篇文章( http://winseclone.iteye.com/blog/1734737 ) 安装cygwin和hadoop-1时,发现伪分布式环境使用mapred时,总是报错。(忘了,但是好像当时没有遇到过这种情况。就当是安装win8送给自己的礼物吧!)。

怀疑了很多东西,配置有问题,重新自定hadoop.tmp.dir,把hadoop-1.1.0换成hadoop-1.0.0等等。

 

错误日志如下:

 

$ hhadoop fs -rmr /test/output ; hhadoop jar hadoop-examples-1.0.0.jar wordcount /test/input /test/output
Deleted hdfs://WINSE:9000/test/output
13/03/23 22:46:07 INFO input.FileInputFormat: Total input paths to process : 1
13/03/23 22:46:08 INFO mapred.JobClient: Running job: job_201303232144_0002
13/03/23 22:46:09 INFO mapred.JobClient:  map 0% reduce 0%
13/03/23 22:46:16 INFO mapred.JobClient: Task Id : attempt_201303232144_0002_m_000002_0, Status : FAILED
java.lang.Throwable: Child Error
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:272)
Caused by: java.io.IOException: Task process exit with nonzero status of -1.
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:259)

13/03/23 22:46:16 WARN mapred.JobClient: Error reading task outputhttp://WINSE:50060/tasklog?plaintext=true&attemptid=attempt_201303232144_0002_m_000002_0&filter=stdout
13/03/23 22:46:16 WARN mapred.JobClient: Error reading task outputhttp://WINSE:50060/tasklog?plaintext=true&attemptid=attempt_201303232144_0002_m_000002_0&filter=stderr
13/03/23 22:46:22 INFO mapred.JobClient: Task Id : attempt_201303232144_0002_m_000002_1, Status : FAILED

 

经过修改原来,不断的修改,加入sysout打印,算是最终找出程序出现错误的地方!

org.apache.hadoop.mapred.DefaultTaskController.java #launchTask
org.apache.hadoop.mapred.JvmManager.java #runChild
org.apache.hadoop.mapred.TaskRunner.java #launchJvmAndWait

 

org.apache.hadoop.fs.FileUtil.java #checkReturnValue
org.apache.hadoop.fs.RawLocalFileSystem.java  #setPermission  #mkdirs

 

发现在org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(Path)方法中,建立文件的路径方法检查attempt_201303232144_0002_m_000001_0是否为文件夹会失败!

而在cygwin中查看:

 

Winseliu@WINSE ~/hadoop/logs/userlogs/job_201303232144_0002
$ ll
总用量 9
lrwxrwxrwx 1 Winseliu None  89 3月  23 22:46 attempt_201303232144_0002_m_000001_0 -> /cluster/mapred/local/userlogs/job_201303232144_0002/attempt_201303232144_0002_m_000001_0
lrwxrwxrwx 1 Winseliu None  89 3月  23 22:46 attempt_201303232144_0002_m_000001_1 -> /cluster/mapred/local/userlogs/job_201303232144_0002/attempt_201303232144_0002_m_000001_1
lrwxrwxrwx 1 Winseliu None  89 3月  23 22:46 attempt_201303232144_0002_m_000001_2 -> /cluster/mapred/local/userlogs/job_201303232144_0002/attempt_201303232144_0002_m_000001_2
lrwxrwxrwx 1 Winseliu None  89 3月  23 22:46 attempt_201303232144_0002_m_000001_3 -> /cluster/mapred/local/userlogs/job_201303232144_0002/attempt_201303232144_0002_m_000001_3
lrwxrwxrwx 1 Winseliu None  89 3月  23 22:46 attempt_201303232144_0002_m_000002_0 -> /cluster/mapred/local/userlogs/job_201303232144_0002/attempt_201303232144_0002_m_000002_0
lrwxrwxrwx 1 Winseliu None  89 3月  23 22:46 attempt_201303232144_0002_m_000002_1 -> /cluster/mapred/local/userlogs/job_201303232144_0002/attempt_201303232144_0002_m_000002_1
lrwxrwxrwx 1 Winseliu None  89 3月  23 22:46 attempt_201303232144_0002_m_000002_2 -> /cluster/mapred/local/userlogs/job_201303232144_0002/attempt_201303232144_0002_m_000002_2
lrwxrwxrwx 1 Winseliu None  89 3月  23 22:46 attempt_201303232144_0002_m_000002_3 -> /cluster/mapred/local/userlogs/job_201303232144_0002/attempt_201303232144_0002_m_000002_3
-rwxr-xr-x 1 Winseliu None 404 3月  23 22:46 job-acls.xml

 

对于linux来说,这些就是引用到另一个文件夹,它本身应该也是文件夹!但是window的jdk不认识这些东西!

 

  public boolean mkdirs(Path f) throws IOException {
    Path parent = f.getParent();
    File p2f = pathToFile(f);
    return (parent == null || mkdirs(parent)) &&
      (p2f.mkdir() || p2f.isDirectory());
  }

 

所以在判断p2f.isDirectory()返回false,然后会抛出IOException,最终以-1的状态退出Map Child的程序!

 

使用org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskAttemptID, boolean)方法来制定了输出的日志输出的位置。在最终执行的会在shell命令中把输出的sysout和syserr输出到日志文件中。userlogs的父目录是使用hadoop.log.dir系统属性来配置的!

 

mapred.DefaultTaskController.launchTask()

|--mapred.TaskLog.buildCommandLine()

 

临时解决方法:

把hadoop.log.dir定位到真正mapred日志的目录( mapred.local.dir : ${hadoop.tmp.dir}/mapred/local )!

 

export HADOOP_LOG_DIR=/cluster/mapred/local

 

把windows的/cluster映射到cygwin(linux)的/cluster:

 

Winseliu@WINSE ~/hadoop
$ ll /cygdrive/c | grep cluster
drwxr-xr-x+ 1 Winseliu         None                      0 3月  24 00:08 cluster

Winseliu@WINSE ~/hadoop
$ ll / | grep cluster
lrwxrwxrwx   1 Winseliu None     19 3月  23 09:39 cluster -> /cygdrive/c/cluster

 

但是,运行wordcount的例子时,还是不正常!查看tasktracker的日志时,发现有String装成Integer的NumberFormatException异常!

修改org.apache.hadoop.mapred.JvmManager.JvmManagerForType.JvmRunner.kill()方法。添加pidStr为空字符串的检查!

 

String pidStr = jvmIdToPid.get(jvmId);
if (pidStr != null && !pidStr.isEmpty()) {

 

然后,终于看到Finish咯!在/test/output/part-r-00000中也看到了结果。

 

 

其他一些简化处理,即配置文件:

 

alias startCluster="~/hadoop/bin/start-all.sh"
alias stopCluster="~/hadoop/bin/stop-all.sh; ~/hadoop/bin/stop-all.sh"

alias hhadoop="~/hadoop/bin/hadoop"

 

Winseliu@WINSE ~
$ ll | grep hadoop
lrwxrwxrwx  1 Winseliu None    12 3月  23 10:44 hadoop -> hadoop-1.0.0
drwx------+ 1 Winseliu None     0 3月  24 00:06 hadoop-1.0.0

 

<!-- core-site.xml -->

<configuration>

<property>
<name>fs.default.name</name>
<value>hdfs://WINSE:9000</value>
</property>

<property>
<name>hadoop.tmp.dir</name>
<value>/cluster</value>
</property>

</configuration>

<!-- hdfs-site.xml -->

<configuration>

<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<property>
  <name>dfs.permissions</name>
  <value>false</value>
</property>

<property>
  <name>dfs.permissions.supergroup</name>
  <value>None</value>
</property>

<property>
<name>dfs.safemode.extension</name>
<value>1000</value>
</property>

</configuration>

<!-- mapred-site.xml -->

<configuration>

<property>
<name>mapred.job.tracker</name>
<value>WINSE:9001</value>
</property>

</configuration>

 

关于查看启动的进程,看可以通过任务管理器来查看:



 
 

 

  • 大小: 56.6 KB




收藏 推荐 打印 | 录入:admin | 阅读:
相关新闻