ANKITBHATNAGAR | 1 Nov 15:56 2010

Re: Hadoop Reduce is never called


This is looks suspicious - 

10/10/30 11:54:08 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found
in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
core-default.xml, mapred-default.xml and hdfs-default.xml respectively

Ankit
--

-- 
View this message in context: http://old.nabble.com/Hadoop-Reduce-is-never-called-tp30094575p30105474.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Cameron_ | 2 Nov 17:46 2010
Picon
Picon

Re: Hadoop Reduce is never called


The deprecated warning is not the issue - This message appears only because I
am running it in an eclipse environment - when run from the command line it
is not there. 

ANKITBHATNAGAR wrote:
> 
> This is looks suspicious - 
> 
> 10/10/30 11:54:08 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
> found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
> use core-site.xml, mapred-site.xml and hdfs-site.xml to override
> properties of core-default.xml, mapred-default.xml and hdfs-default.xml
> respectively
> 
> 
> Ankit
> 

--

-- 
View this message in context: http://old.nabble.com/Hadoop-Reduce-is-never-called-tp30094575p30115480.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

ANKITBHATNAGAR | 2 Nov 19:21 2010

Re: Hadoop Reduce is never called


10/10/30 11:54:10 INFO mapred.JobClient:     Reduce output records=32242
10/10/30 11:54:10 INFO mapred.JobClient:     Spilled Records=64484
10/10/30 11:54:10 INFO mapred.JobClient:     Map output bytes=1840378
10/10/30 11:54:10 INFO mapred.JobClient:     Combine input records=0
10/10/30 11:54:10 INFO mapred.JobClient:     Map output records=32242
10/10/30 11:54:10 INFO mapred.JobClient:     Reduce input records=32242 

These bold lines suggest that infact reduce was called.
Could you debug and check the reducer?
--

-- 
View this message in context: http://old.nabble.com/Hadoop-Reduce-is-never-called-tp30094575p30116293.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

ANKITBHATNAGAR | 2 Nov 19:29 2010

Re: Hadoop Reduce is never called


Also just noticed that your extension is not generic ?

extends Reducer{

should be extends Reducer<...>{

--

-- 
View this message in context: http://old.nabble.com/Hadoop-Reduce-is-never-called-tp30094575p30116358.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Cameron_ | 2 Nov 21:43 2010
Picon
Picon

Re: Hadoop Reduce is never called


Thanks for your help - I found the problem.  The signature for my reduce
function had Iterator instead of Iterable.  Without the right signature, it
was calling the reduce method from the superclass and not mine.

ANKITBHATNAGAR wrote:
> 
> Also just noticed that your extension is not generic ?
>  
> extends Reducer{
> 
> 
> should be extends Reducer<...>{
> 
> 
> 
> 

--

-- 
View this message in context: http://old.nabble.com/Hadoop-Reduce-is-never-called-tp30094575p30117511.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.


Gmane