Mahesh Shinde | 22 Jul 08:20 2011
Picon

FW: Question about property "fs.default.name"


Hi,

I have basic question on property "fs.default.name" is that I am not able to open NameNode URL when I set
fs.default.name=file:/// .
If we define not use HDFS as our file system then how hadoop deals with File system of local?..
Please reply.

Mahesh Shinde | Systems
Engineer
mahesh_shinde@...<mailto:mahesh_shinde@...>|
Cell: +918308321501 | Tel: +91-20-30235194
Persistent Systems Ltd. | 20 Glorious Years | www.persistentsys.com<http://www.persistentsys.com/>

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent
Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are
not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this
message. If you have received this communication in error, please notify the sender and delete all copies
of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

Harsh J | 23 Jul 23:38 2011

Re: FW: Question about property "fs.default.name"

Mahesh,

The NameNode is a component of HDFS. It does not apply to other FSes
Hadoop can use. You can browse file:/// (are you planning to use NFS,
etc.?) with a regular file browser.

Besides this, am not sure I even got your question. Why don't you try
out how Hadoop deals with a local fs by running a few MR jobs with
that config?

On Fri, Jul 22, 2011 at 11:50 AM, Mahesh Shinde
<Mahesh_Shinde@...> wrote:
>
>
> Hi,
>
> I have basic question on property "fs.default.name" is that I am not able to open NameNode URL when I set
fs.default.name=file:/// .
> If we define not use HDFS as our file system then how hadoop deals with File system of local?..
> Please reply.
>
> Mahesh Shinde | Systems Engineer
>
mahesh_shinde@...<mailto:mahesh_shinde@...>|
Cell: +918308321501 | Tel: +91-20-30235194
> Persistent Systems Ltd. | 20 Glorious Years | www.persistentsys.com<http://www.persistentsys.com/>
>
>
> DISCLAIMER
> ==========
(Continue reading)

Uma Maheswara Rao G 72686 | 24 Jul 05:30 2011

Re: FW: Question about property "fs.default.name"


Hi Mahesh,

When starting the NN, it will throw exception with your provided configuration. please check the code
snippet below where exactly validation will happen.

in NameNode:
 public static InetSocketAddress getAddress(URI filesystemURI) {
    String authority = filesystemURI.getAuthority();
    if (authority == null) {
      throw new IllegalArgumentException(String.format(
          "Invalid URI for NameNode address (check %s): %s has no authority.",
          FileSystem.FS_DEFAULT_NAME_KEY, filesystemURI.toString()));
    }
    if (!FSConstants.HDFS_URI_SCHEME.equalsIgnoreCase(
        filesystemURI.getScheme())) {
      throw new IllegalArgumentException(String.format(
          "Invalid URI for NameNode address (check %s): %s is not of scheme '%s'.",
          FileSystem.FS_DEFAULT_NAME_KEY, filesystemURI.toString(),
          FSConstants.HDFS_URI_SCHEME));
    }

Since NN is specific to HDFS, it will expect scheme as hdfs. Otherwise it will throw the exception.

If you develop your own file system, then clients will have intelligence to connect to that file system,
based on your provided configuration for fs.default.name.

Coming to  clients side, if you pass file:/// as fs uri, it will not try to connect to DFS because your passed fs
is related to local. So, it will create LocalFileSystem instead of DistributedFilesystem.

(Continue reading)


Gmane