headers
Ryan Smith | 1 Mar 2009 13:49
Picon

Re: Exception when scanning table

Hi guys,

http://pastebin.com/m1c7b01da

I am also getting the ScannerTimeoutException on HBase/Hadoop 0.19.0.  I am
loading millions of records in 20GB of data in a mapreduce job.  Just trying
to map (write) one text file to one hbase table.  This job would complete
fine on 0.18.x   This is the first time ive tried to run this on 0.19.0.
Ive repeated this test under several 0.19.0 clouds and i keep getting this
ScannerTimeoutException.   I am going to test it once again on 0.18.x to
make sure that is the problem.

-alphaomega

On Sun, Mar 1, 2009 at 3:39 AM, Liu Yan <gzbigegg.ml@...> wrote:

> I have 2 tables, each with about 350K rows (I did "count" on them).
>
> For the first table, the scan kept running for about 5 hours before failing
> out. For the second table, it took about 5 minutes to see the same error.
>
> How to increase the timeout, I searched for "timeout" or "scan" keyword
> under the ./conf folder, didn't find anything too helpful.
>
> Regards,
> Yan
>
> 2009/3/1 stack <stack@...>
>
> > Before you hit the exception, had it been scanning fine?
(Continue reading)

Permalink | Reply |
headers
stack | 4 Mar 2009 07:47

Re: Exception when scanning table

Let us know how it did on 0.18.0 Ryan.

Basically, the message is saying that the interval between invocations of
next by the client was longer than the 60 seconds default.  Here is the
relevant property:

  <property>
    <name>hbase.regionserver.lease.period</name>
    <value>60000</value>
    <description>HRegion server lease period in milliseconds. Default is
    60 seconds. Clients must report in within this period else they are
    considered dead.</description>
  </property>

You could try upping this timeout.

Otherwise, was the server struggling at this time?  You could enable metrics
-- the jvm ones in particular -- or enable gc logging by adding jvm parms to
HBASE_OPTS in hbase-env.sh to see if its struggling doing a full gc (See
also end of troubleshooting section for a suggestion that you use a
different garbage collector).  Do you see instances of 'slept ten times
longer than expected...' in your logs before the scanner timeout or anything
that might give a clue about the server being hung up in any way?

What about the client.  Could it have been stalled by competing processes on
same machine?

I don't think this code changed between 0.18 and 0.19 but if you find that
it works on 0.18, then for sure something is up.
(Continue reading)

Permalink | Reply |
headers
Liu Yan | 4 Mar 2009 07:23
Picon

Re: Exception when scanning table

Can I just ignore this error? I assume it is only a scanner error. What
caused this? Memory, disk space, max-file, I checked all these, but seems
very unlikly caused these conditions.

Regards,
Yan

2009/3/1 Ryan Smith <ryan.justin.smith@...>

> Hi guys,
>
> http://pastebin.com/m1c7b01da
>
> I am also getting the ScannerTimeoutException on HBase/Hadoop 0.19.0.  I am
> loading millions of records in 20GB of data in a mapreduce job.  Just
> trying
> to map (write) one text file to one hbase table.  This job would complete
> fine on 0.18.x   This is the first time ive tried to run this on 0.19.0.
> Ive repeated this test under several 0.19.0 clouds and i keep getting this
> ScannerTimeoutException.   I am going to test it once again on 0.18.x to
> make sure that is the problem.
>
> -alphaomega
>
>
>
> On Sun, Mar 1, 2009 at 3:39 AM, Liu Yan <gzbigegg.ml@...> wrote:
>
> > I have 2 tables, each with about 350K rows (I did "count" on them).
> >
(Continue reading)

Permalink | Reply |
headers
stack | 4 Mar 2009 07:49

Re: Exception when scanning table

Well, it messes up your client, right?  Its not completing its scan?

Is your client processing the return from hbase and taking longer than one
minute to complete?  If so, its lease on the remote server will have
timedout in this time.  Otherwise, do any of questions above to Ryan apply
to you?

Thanks,
St.Ack

On Tue, Mar 3, 2009 at 10:23 PM, Liu Yan <gzbigegg.ml@...> wrote:

> Can I just ignore this error? I assume it is only a scanner error. What
> caused this? Memory, disk space, max-file, I checked all these, but seems
> very unlikly caused these conditions.
>
> Regards,
> Yan
>
> 2009/3/1 Ryan Smith <ryan.justin.smith@...>
>
> > Hi guys,
> >
> > http://pastebin.com/m1c7b01da
> >
> > I am also getting the ScannerTimeoutException on HBase/Hadoop 0.19.0.  I
> am
> > loading millions of records in 20GB of data in a mapreduce job.  Just
> > trying
> > to map (write) one text file to one hbase table.  This job would complete
(Continue reading)

Permalink | Reply |
headers
Ryan Smith | 16 Mar 2009 19:57
Picon

Re: Exception when scanning table

Yes, 0.18.3 hadoop and 0.18.1 hbase loads the same data just fine.  The only
issues I have with 0.18.x is a few map tasks that timeout on one of the many
processing MR jobs, but that is normal, i just have to set the timeout
higher.  the data itself loads into hbase tables just fine.
Obviously im not using the same code, but its logically the same, the only
difference is using the new MapReduce API classes in 0.19

As far as ScannerTimeoutException in hbase 0.19.0, ill test it again on
hbase 0.19.1 once its released and let you know.

-alphaomega

On Wed, Mar 4, 2009 at 2:49 AM, stack <stack@...> wrote:

> Well, it messes up your client, right?  Its not completing its scan?
>
> Is your client processing the return from hbase and taking longer than one
> minute to complete?  If so, its lease on the remote server will have
> timedout in this time.  Otherwise, do any of questions above to Ryan apply
> to you?
>
> Thanks,
> St.Ack
>
> On Tue, Mar 3, 2009 at 10:23 PM, Liu Yan <gzbigegg.ml@...> wrote:
>
> > Can I just ignore this error? I assume it is only a scanner error. What
> > caused this? Memory, disk space, max-file, I checked all these, but seems
> > very unlikly caused these conditions.
> >
(Continue reading)

Permalink | Reply |

Gmane