headers
Iwona Bialynicka-Birula | 7 May 2011 22:56
Favicon

Problem debugging MapReduce job under Windows

Hello,

I am trying to debug a Hadoop MapReduce job under Eclipse in Windows and I am running into a problem when the
Hadoop framework tries to set up the staging directory (see the stack trace below) and change its
permissions to 700. It is doing it in RawLocalFileSystem.setPermissions by calling
java.io.File.setReadable(false, false), but under Windows this always fails, since you're not
allowed to set readable to false. Is there a workaround for this to enable debugging in a Windows
environment, or is such a scenario simply not possible?

(I am using Windows 7, Hadoop 0.20.2, Java 6. I have Cygwin installed, but in this case it is not using it,
since it is making the system call directly.)

Thanks,
Iwona

java.io.IOException: Failed to set permissions of path:
file:/tmp/hadoop-iwonabb/mapred/staging/iwonabb-1931875024/.staging to 0700
                at org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:526)
                at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:500)
                at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:310)
                at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
                at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)
                at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:799)
                at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:793)
                at java.security.AccessController.doPrivileged(Native Method)
                at javax.security.auth.Subject.doAs(Unknown Source)
                at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
                at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:793)
                at org.apache.hadoop.mapreduce.Job.submit(Job.java:465)
                at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:495)
(Continue reading)

Permalink | Reply |
headers
hadoopfan | 8 May 2011 02:15
Favicon

Re:Problem debugging MapReduce job under Windows

hadoop dfs -chmod 777 /tmp

At 2011-05-08 04:56:22，"Iwona Bialynicka-Birula" <iwona.bb <at> atigeo.com> wrote:

>Hello,
>
>I am trying to debug a Hadoop MapReduce job under Eclipse in Windows and I am running into a problem when the
Hadoop framework tries to set up the staging directory (see the stack trace below) and change its
permissions to 700. It is doing it in RawLocalFileSystem.setPermissions by calling
java.io.File.setReadable(false, false), but under Windows this always fails, since you're not
allowed to set readable to false. Is there a workaround for this to enable debugging in a Windows
environment, or is such a scenario simply not possible?
>
>(I am using Windows 7, Hadoop 0.20.2, Java 6. I have Cygwin installed, but in this case it is not using it,
since it is making the system call directly.)
>
>Thanks,
>Iwona
>
>java.io.IOException: Failed to set permissions of path:
file:/tmp/hadoop-iwonabb/mapred/staging/iwonabb-1931875024/.staging to 0700
>                at org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:526)
>                at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:500)
>                at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:310)
>                at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
>                at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)
>                at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:799)
>                at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:793)
>                at java.security.AccessController.doPrivileged(Native Method)
>                at javax.security.auth.Subject.doAs(Unknown Source)
(Continue reading)

Permalink | Reply |
headers
Iwona Bialynicka-Birula | 8 May 2011 05:28
Favicon

RE: Re:Problem debugging MapReduce job under Windows

Thanks for the reply, but the command you suggest refers to the Hadoop file system and I am having trouble
with the local file system - in particular a subset of hadoop.tmp.dir. Also, even if I grant full
permissions to this folder initially, Hadoop still tries to set them to 700, which is not allowed under Windows.

Thanks,
Iwona

-----Original Message-----
From: hadoopfan [mailto:shhaijava@...] 
Sent: Saturday, May 07, 2011 5:16 PM
To: common-user@...
Subject: Re:Problem debugging MapReduce job under Windows

hadoop dfs -chmod 777 /tmp

At 2011-05-08 04:56:22，"Iwona Bialynicka-Birula" <iwona.bb <at> atigeo.com> wrote:

>Hello,
>
>I am trying to debug a Hadoop MapReduce job under Eclipse in Windows and I am running into a problem when the
Hadoop framework tries to set up the staging directory (see the stack trace below) and change its
permissions to 700. It is doing it in RawLocalFileSystem.setPermissions by calling
java.io.File.setReadable(false, false), but under Windows this always fails, since you're not
allowed to set readable to false. Is there a workaround for this to enable debugging in a Windows
environment, or is such a scenario simply not possible?
>
>(I am using Windows 7, Hadoop 0.20.2, Java 6. I have Cygwin installed, but in this case it is not using it,
since it is making the system call directly.)
>
>Thanks,
(Continue reading)

Permalink | Reply |
headers
Itzhak Pan | 9 May 2011 03:12
Picon

Re: RE: Re:Problem debugging MapReduce job under Windows

I encountered the same problem. It is because the app fails to call 'chmod'
in windows. My solution is to include cygwin directory in PATH.

在 2011 5 8 11:29，"Iwona Bialynicka-Birula" <iwona.bb <at> atigeo.com>写道：
> Thanks for the reply, but the command you suggest refers to the Hadoop
file system and I am having trouble with the local file system - in
particular a subset of hadoop.tmp.dir. Also, even if I grant full
permissions to this folder initially, Hadoop still tries to set them to 700,
which is not allowed under Windows.
>
> Thanks,
> Iwona
>
> -----Original Message-----
> From: hadoopfan [mailto:shhaijava@...]
> Sent: Saturday, May 07, 2011 5:16 PM
> To: common-user@...
> Subject: Re:Problem debugging MapReduce job under Windows
>
> hadoop dfs -chmod 777 /tmp
>
>
>
> At 2011-05-08 04:56:22，"Iwona Bialynicka-Birula" <iwona.bb <at> atigeo.com>
wrote:
>
>>Hello,
>>
>>I am trying to debug a Hadoop MapReduce job under Eclipse in Windows and I
am running into a problem when the Hadoop framework tries to set up the
(Continue reading)

Permalink | Reply |
headers
Iwona Bialynicka-Birula | 9 May 2011 18:28
Favicon

RE: RE: Re:Problem debugging MapReduce job under Windows

I do have Cygwin on the path, but in this case, it is not using chmod to set permissions, but
File.setReadable. Below is what RawLocalFileSystem.setPermissions looks like. It only uses chmod
(hence Cygwin) if group != other (lines 490-494), but this is not the case for 700 permissions, so it does it
using File.setReadable and fails (line 499). I guess I could edit this class, but I'm just wondering if
there is a proper way of dealing with this without editing the Hadoop library.

481	 <at> Override
482	  public void setPermission(Path p, FsPermission permission
483	      ) throws IOException {
484	    FsAction user = permission.getUserAction();
485	    FsAction group = permission.getGroupAction();
486	    FsAction other = permission.getOtherAction();
487	    
488	    File f = pathToFile(p);
489	    
490	    // Fork chmod if group and other permissions are different...
491	    if (group != other) {
492	      execSetPermission(f, permission);
493	      return;
494	    }
495	    
496	    boolean rv = true;
497	    
498	    // read perms
499	    rv = f.setReadable(group.implies(FsAction.READ), false);

Thanks,
Iwona

-----Original Message-----
(Continue reading)

Permalink | Reply |
headers
Iwona Bialynicka-Birula | 9 May 2011 22:43
Favicon

RE: RE: Re:Problem debugging MapReduce job under Windows

It turns out that this problem is specific to a beta Cloudera Hadoop build I was using: 0.20.2-737. It does
not reproduce in CDH3u0, so all is good now.

Thanks for all the suggestions,
Iwona

-----Original Message-----
From: Iwona Bialynicka-Birula [mailto:iwona.bb <at> atigeo.com] 
Sent: Monday, May 09, 2011 9:29 AM
To: common-user <at> hadoop.apache.org
Subject: RE: RE: Re:Problem debugging MapReduce job under Windows

I do have Cygwin on the path, but in this case, it is not using chmod to set permissions, but
File.setReadable. Below is what RawLocalFileSystem.setPermissions looks like. It only uses chmod
(hence Cygwin) if group != other (lines 490-494), but this is not the case for 700 permissions, so it does it
using File.setReadable and fails (line 499). I guess I could edit this class, but I'm just wondering if
there is a proper way of dealing with this without editing the Hadoop library.

481	 <at> Override
482	  public void setPermission(Path p, FsPermission permission
483	      ) throws IOException {
484	    FsAction user = permission.getUserAction();
485	    FsAction group = permission.getGroupAction();
486	    FsAction other = permission.getOtherAction();
487	    
488	    File f = pathToFile(p);
489	    
490	    // Fork chmod if group and other permissions are different...
491	    if (group != other) {
492	      execSetPermission(f, permission);
(Continue reading)

Permalink | Reply |
headers
Raja Nagendra Kumar | 15 Jun 2011 08:53
Gravatar

Re: Problem debugging MapReduce job under Windows

Hi,

We are see the same error in  hadoop-0.20.203.0rc1.tar

We tried to to windows path with Cygwin and tried the chmod to 777 etc.. but 
none works.. it continues to throw the below error..

java.io.IOException: Failed to set permissions of path: file:/tmp/hadoop-nagkuma
r/mapred/staging/nagkumar-803773577/.staging to 0700
        at org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFile
System.java:525)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSys
tem.java:499)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.jav
a:318)
        at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:18
3)
        at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmi
ssionFiles.java:116)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:797)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:791)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1059)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:7
91)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:465)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:494)
        at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
(Continue reading)

Permalink | Reply |
headers
Sal | 23 Jun 2011 01:08
Picon
Favicon

Re: Problem debugging MapReduce job under Windows

I had the same issue.  I installed the previous stable version of Hadoop
(0.20.2), and it worked fine.  I hope this helps.

-Sal
Permalink | Reply |
headers
Hardik Akbari | 10 Mar 2012 13:46
Picon

Re: Problem debugging MapReduce job under Windows

Sal <sals99 <at> ...> writes:

> 
> I had the same issue.  I installed the previous stable version of Hadoop
> (0.20.2), and it worked fine.  I hope this helps.
> 
> -Sal
> 
> 

Thanks. It helped. worked for 0.20.2
Permalink | Reply |

Gmane