QA@IT

hadoopでストリーミングで処理してエラー

1785 PV

お世話になります
Hadoopのサンプルで、青空文庫から、吾輩は猫であるをとってきて、それをmecabで分割した後
PHPで、MapperとReducerを作り、処理をしてるのですが、うまく行かなくて、困っております

このサイトを参考にやってます
http://d.hatena.ne.jp/stellaqua/20090305/1236222223

server:hadoop shiratsu$ hadoop jar /usr/local/Cellar/hadoop/1.1.2/libexec/contrib/streaming/hadoop-streaming-1.1.2.jar -input inputs/wagahaiwa_nekodearu_wakati.txt -output outputs -mapper 'php map.php' -reducer 'php reduce.php'
packageJobJar: [/tmp/hadoop-shiratsu/hadoop-unjar3146907696531556152/] [] /var/folders/35/8cpd2xpx7755kmrtlxd8c8j00000gn/T/streamjob590423568834373421.jar tmpDir=null
14/01/22 01:19:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/01/22 01:19:21 WARN snappy.LoadSnappy: Snappy native library not loaded
14/01/22 01:19:21 INFO mapred.FileInputFormat: Total input paths to process : 1
14/01/22 01:19:22 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-shiratsu/mapred/local]
14/01/22 01:19:22 INFO streaming.StreamJob: Running job: job_201401220106_0008
14/01/22 01:19:22 INFO streaming.StreamJob: To kill this job, run:
14/01/22 01:19:22 INFO streaming.StreamJob: /usr/local/Cellar/hadoop/1.1.2/libexec/bin/../bin/hadoop job  -Dmapred.job.tracker=localhost:9001 -kill job_201401220106_0008
14/01/22 01:19:22 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201401220106_0008
14/01/22 01:19:23 INFO streaming.StreamJob:  map 0%  reduce 0%
14/01/22 01:20:26 INFO streaming.StreamJob:  map 100%  reduce 100%
14/01/22 01:20:26 INFO streaming.StreamJob: To kill this job, run:
14/01/22 01:20:26 INFO streaming.StreamJob: /usr/local/Cellar/hadoop/1.1.2/libexec/bin/../bin/hadoop job  -Dmapred.job.tracker=localhost:9001 -kill job_201401220106_0008
14/01/22 01:20:26 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201401220106_0008
14/01/22 01:20:26 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201401220106_0008_m_000001

ググっても有効な回答が見つからず、ちょっとお手上げ気味です
よろしくお願いいたします

ウォッチ

この質問への回答やコメントをメールでお知らせします。