dmitri garbuzov | 29 Apr 01:15 2010
Picon

hbase with multiple interfaces

Hi all,

I'm trying to set up hbase on a cluster where each machine has multiple 
interfaces.  Hdfs worked right away since I could just set the name node 
to listen on 0.0.0.0, but hbase region servers seem to pick one of the 
interfaces at random, ignoring the hbase.regionserver.dns.interface 
option.  Is the dns.interface setting supposed to make hbase bind to a 
specific interface? Is there anything special I need to do to get this 
to work?

dmitri

Ryan Rawson | 29 Apr 01:31 2010
Picon

Re: hbase with multiple interfaces

HBase does bind to 0.0.0.0:60020.  It uses the hostname to report this
to META and that is how other people find the region servers.

Are you not seeing this?

On Wed, Apr 28, 2010 at 4:15 PM, dmitri garbuzov <dgarbuzov@...> wrote:
> Hi all,
>
> I'm trying to set up hbase on a cluster where each machine has multiple
> interfaces.  Hdfs worked right away since I could just set the name node to
> listen on 0.0.0.0, but hbase region servers seem to pick one of the
> interfaces at random, ignoring the hbase.regionserver.dns.interface option.
>  Is the dns.interface setting supposed to make hbase bind to a specific
> interface? Is there anything special I need to do to get this to work?
>
> dmitri
>

dmitri garbuzov | 29 Apr 01:54 2010
Picon

Re: hbase with multiple interfaces

Doesn't look like it's binding to 0.0.0.0

sudo lsof -i :60020
COMMAND   PID   USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
java    22530 hadoop   37u  IPv4 81808593      0t0  TCP 
sns12-virt.CS.Princeton.EDU:60020 (LISTEN)

sudo netstat -an | grep 60020
tcp        0      0 128.112.7.112:60020         
0.0.0.0:*                   LISTEN

Ryan Rawson wrote:
> HBase does bind to 0.0.0.0:60020.  It uses the hostname to report this
> to META and that is how other people find the region servers.
>
> Are you not seeing this?
>
> On Wed, Apr 28, 2010 at 4:15 PM, dmitri garbuzov
<dgarbuzov@...> wrote:
>   
>> Hi all,
>>
>> I'm trying to set up hbase on a cluster where each machine has multiple
>> interfaces.  Hdfs worked right away since I could just set the name node to
>> listen on 0.0.0.0, but hbase region servers seem to pick one of the
>> interfaces at random, ignoring the hbase.regionserver.dns.interface option.
>>  Is the dns.interface setting supposed to make hbase bind to a specific
>> interface? Is there anything special I need to do to get this to work?
>>
>> dmitri
(Continue reading)

Ryan Rawson | 29 Apr 01:58 2010
Picon

Re: hbase with multiple interfaces

If you are using the "hbase.regionserver.dns.interface" option this is
what would happen - binding to a specific interface.

What is your config looking like?

On Wed, Apr 28, 2010 at 4:54 PM, dmitri garbuzov <dgarbuzov@...> wrote:
> Doesn't look like it's binding to 0.0.0.0
>
> sudo lsof -i :60020
> COMMAND   PID   USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
> java    22530 hadoop   37u  IPv4 81808593      0t0  TCP
> sns12-virt.CS.Princeton.EDU:60020 (LISTEN)
>
> sudo netstat -an | grep 60020
> tcp        0      0 128.112.7.112:60020         0.0.0.0:*
> LISTEN
>
>
> Ryan Rawson wrote:
>>
>> HBase does bind to 0.0.0.0:60020.  It uses the hostname to report this
>> to META and that is how other people find the region servers.
>>
>> Are you not seeing this?
>>
>> On Wed, Apr 28, 2010 at 4:15 PM, dmitri garbuzov <dgarbuzov@...>
>> wrote:
>>
>>>
>>> Hi all,
(Continue reading)

dmitri garbuzov | 29 Apr 02:11 2010
Picon

Re: hbase with multiple interfaces

I'm getting the same results without hbase.regionserver.dns.interface 
set and with it explicitly set  to "default"

hbase-site.xml:
<configuration>
    <property>
        <name>hbase.rootdir</name>
        <value>hdfs://sns12.cs.princeton.edu:8020/hbase</value>
    </property>
    <property>
        <name>hbase.master</name>
        <value>sns12-virt.CS.Princeton.EDU:60000</value>
    </property>
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    <property>
        <name>hbase.zookeeper.quorum</name>

<value>sns12-virt.CS.Princeton.EDU,sns13-virt.CS.Princeton.EDU,sns14-virt.CS.Princeton.EDU</value>
    </property>
</configuration>

Ryan Rawson wrote:
> If you are using the "hbase.regionserver.dns.interface" option this is
> what would happen - binding to a specific interface.
>
> What is your config looking like?
>
(Continue reading)

Ryan Rawson | 29 Apr 02:19 2010
Picon

Re: hbase with multiple interfaces

Ah yes I was reading my netstat wrong, we have the same deal.

So hbase will use hostname to determine which IP you wish to
communicate on, then use that to bind and report to the master.

The setting     <name>hbase.regionserver.dns.interface</name>
should work the same as the similar setting in Hadoop.   If it does
not, sounds like a bug.

On Wed, Apr 28, 2010 at 5:11 PM, dmitri garbuzov <dgarbuzov@...> wrote:
> I'm getting the same results without hbase.regionserver.dns.interface set
> and with it explicitly set  to "default"
>
> hbase-site.xml:
> <configuration>
>   <property>
>       <name>hbase.rootdir</name>
>       <value>hdfs://sns12.cs.princeton.edu:8020/hbase</value>
>   </property>
>   <property>
>       <name>hbase.master</name>
>       <value>sns12-virt.CS.Princeton.EDU:60000</value>
>   </property>
>   <property>
>       <name>hbase.cluster.distributed</name>
>       <value>true</value>
>   </property>
>   <property>
>       <name>hbase.zookeeper.quorum</name>
>
(Continue reading)

Michael Segel | 29 Apr 13:39 2010
Picon

RE: hbase with multiple interfaces


Is sns12-virt the actual machine name? 
What do you get if you type in uname -a?

The problem with Hadoop and of course HBase is that they determine their own IP network based on the
machine's actual name, so that even if you have multiple interfaces, the nodes will choose the interface
that matches the machine name. (IMHO this is a defect that should be fixed.) 

The use case is that you may want to expose a 1GBe nic to the outside world for access to the cloud and then use a
10GBe interface for inter-cloud communication.

Just IMHO.

-Mike

> Date: Wed, 28 Apr 2010 20:11:37 -0400
> From: dgarbuzov@...
> To: hbase-user@...
> CC: jterrace@...
> Subject: Re: hbase with multiple interfaces
> 
> I'm getting the same results without hbase.regionserver.dns.interface 
> set and with it explicitly set  to "default"
> 
> hbase-site.xml:
> <configuration>
>     <property>
>         <name>hbase.rootdir</name>
>         <value>hdfs://sns12.cs.princeton.edu:8020/hbase</value>
>     </property>
(Continue reading)

Stack | 29 Apr 16:56 2010
Picon

Re: hbase with multiple interfaces

On Thu, Apr 29, 2010 at 4:39 AM, Michael Segel
<michael_segel@...> wrote:
> The problem with Hadoop and of course HBase is that they determine their own IP network based on the
machine's actual name, so that even if you have multiple interfaces, the nodes will choose the interface
that matches the machine name. (IMHO this is a defect that should be fixed.)
>

I made HBASE-2502 to cover this issue.
Thanks Michael,
St.Ack


Gmane