Apache Hadoop user mailing list

Christian Stucchio | 25 Jan 2011 05:45

Hadoop is ignoring my hadoop.tmp.dir variable

Greetings,

I'm encountering the following problem with hadoo
p (0.20.2) - for whatever reason, hadoop seems to be completely
ignoring my hadoop.tmp.dir variable. I expect I've just made some
stupid mistake, but I'm a bit stuck. Can anyone offer any suggestions?

From my core-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/media/hadoop-tmp/</value>
  </property>
...

However, when I run a job, the spill data is dumped to
/tmp/hadoop-hadoopuser, and printing out the
configuration.get("hadoop.tmp.dir") reveals that that is indeed the
temporary directory.

I believe this is the cause of another error I am running into:

11/01/24 20:46:17 WARN mapred.LocalJobRunner: job_local_0002
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_local_0002/attempt_local_0002_m_000000_0/output/file.out
in any of the configured

(Continue reading)

headers

Harsh J | 25 Jan 2011 07:00

Re: Hadoop is ignoring my hadoop.tmp.dir variable

If your environment is properly setup to read your configuration files
instead of the defaults in JAR (which is /tmp/hadoop-{user.name}, as
you're seeing it), you should get the right property values in effect.

Also, instead of modifying "hadoop.tmp.dir", you may also
hand-configure "mapred.local.dir" to specify local directories to be
used by tasks, etc.

On Tue, Jan 25, 2011 at 10:15 AM, Christian Stucchio
<stucchio@...> wrote:
> Greetings,
>
> I'm encountering the following problem with hadoo
> p (0.20.2) - for whatever reason, hadoop seems to be completely
> ignoring my hadoop.tmp.dir variable. I expect I've just made some
> stupid mistake, but I'm a bit stuck. Can anyone offer any suggestions?
>
> From my core-site.xml:
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <configuration>
>  <property>
>    <name>hadoop.tmp.dir</name>
>    <value>/media/hadoop-tmp/</value>
>  </property>
> ...
>
> However, when I run a job, the spill data is dumped to

(Continue reading)

headers

Saurabh Dutta | 25 Jan 2011 07:01

RE: Hadoop is ignoring my hadoop.tmp.dir variable

Hi,

/media is generally a directory where your mount files systems on removable media like CD-ROM drives,
floppy disks, and Zip drives.

Did you try using another directory instead of /media, say /home/directory-where-user-has-permission/tmp

See if this works for you.

Thanks,
Saurabh

-----Original Message-----
From: Christian Stucchio [mailto:stucchio@...]
Sent: Tuesday, January 25, 2011 10:15 AM
To: common-user@...
Subject: Hadoop is ignoring my hadoop.tmp.dir variable

Greetings,

I'm encountering the following problem with hadoo
p (0.20.2) - for whatever reason, hadoop seems to be completely
ignoring my hadoop.tmp.dir variable. I expect I've just made some
stupid mistake, but I'm a bit stuck. Can anyone offer any suggestions?

From my core-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

(Continue reading)