Mohit Anchlia | 26 May 18:56 2011
Picon

Help with pigsetup

I sent this to pig apache user mailing list but have got no response.
Not sure if that list is still active.

thought I will post here if someone is able to help me.

I am in process of installing and learning pig. I have a hadoop
cluster and when I try to run pig in mapreduce mode it errors out:

Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1

Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. Failed to create DataStorage

java.lang.RuntimeException: Failed to create DataStorage
       at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
       at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
       at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
       at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
       at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
       at org.apache.pig.PigServer.<init>(PigServer.java:226)
       at org.apache.pig.PigServer.<init>(PigServer.java:215)
       at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
       at org.apache.pig.Main.run(Main.java:452)
       at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Call to dsdb1/172.18.60.96:54310
failed on local exception: java.io.EOFException
       at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
       at org.apache.hadoop.ipc.Client.call(Client.java:743)
       at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
(Continue reading)

Harsh J | 26 May 19:00 2011

Re: Help with pigsetup

I think Jonathan Coveney's reply on user <at> pig answered your question.
Its basically an issue of hadoop version differences between the one
Pig 0.8.1 release got bundled with vs. Hadoop 0.20.203 release which
is newer.

On Thu, May 26, 2011 at 10:26 PM, Mohit Anchlia
<mohitanchlia@...> wrote:
> I sent this to pig apache user mailing list but have got no response.
> Not sure if that list is still active.
>
> thought I will post here if someone is able to help me.
>
> I am in process of installing and learning pig. I have a hadoop
> cluster and when I try to run pig in mapreduce mode it errors out:
>
> Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1
>
> Error before Pig is launched
> ----------------------------
> ERROR 2999: Unexpected internal error. Failed to create DataStorage
>
> java.lang.RuntimeException: Failed to create DataStorage
>       at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
>       at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
>       at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
>       at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
>       at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
>       at org.apache.pig.PigServer.<init>(PigServer.java:226)
>       at org.apache.pig.PigServer.<init>(PigServer.java:215)
>       at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
(Continue reading)

Mohit Anchlia | 26 May 19:04 2011
Picon

Re: Help with pigsetup

For some reason I don't see that reply from Jonathan in my Inbox. I'll
try to google it.

What should be my next step in that case? I can't use pig then?

On Thu, May 26, 2011 at 10:00 AM, Harsh J <harsh@...> wrote:
> I think Jonathan Coveney's reply on user <at> pig answered your question.
> Its basically an issue of hadoop version differences between the one
> Pig 0.8.1 release got bundled with vs. Hadoop 0.20.203 release which
> is newer.
>
> On Thu, May 26, 2011 at 10:26 PM, Mohit Anchlia
<mohitanchlia@...> wrote:
>> I sent this to pig apache user mailing list but have got no response.
>> Not sure if that list is still active.
>>
>> thought I will post here if someone is able to help me.
>>
>> I am in process of installing and learning pig. I have a hadoop
>> cluster and when I try to run pig in mapreduce mode it errors out:
>>
>> Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1
>>
>> Error before Pig is launched
>> ----------------------------
>> ERROR 2999: Unexpected internal error. Failed to create DataStorage
>>
>> java.lang.RuntimeException: Failed to create DataStorage
>>       at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
>>       at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
(Continue reading)

Jonathan Coveney | 26 May 19:06 2011
Picon

Re: Help with pigsetup

I'll repost it here then :)

"Here is what I had to do to get pig running with a different version of
Hadoop (in my case, the cloudera build but I'd try this as well):

build pig-withouthadoop.jar by running "ant jar-withouthadoop". Then, when
you run pig, put the pig-withouthadoop.jar on your classpath as well as your
hadoop jar. In my case, I found that scripts only worked if I additionally
manually registered the antlr jar:

register /path/to/pig/build/ivy/lib/Pig/antlr-runtime-3.2.jar;"

2011/5/26 Mohit Anchlia <mohitanchlia@...>

> For some reason I don't see that reply from Jonathan in my Inbox. I'll
> try to google it.
>
> What should be my next step in that case? I can't use pig then?
>
> On Thu, May 26, 2011 at 10:00 AM, Harsh J <harsh@...> wrote:
> > I think Jonathan Coveney's reply on user <at> pig answered your question.
> > Its basically an issue of hadoop version differences between the one
> > Pig 0.8.1 release got bundled with vs. Hadoop 0.20.203 release which
> > is newer.
> >
> > On Thu, May 26, 2011 at 10:26 PM, Mohit Anchlia <mohitanchlia@...>
> wrote:
> >> I sent this to pig apache user mailing list but have got no response.
> >> Not sure if that list is still active.
> >>
(Continue reading)

Mohit Anchlia | 26 May 19:18 2011
Picon

Re: Help with pigsetup

On Thu, May 26, 2011 at 10:06 AM, Jonathan Coveney
<jcoveney@...> wrote:
> I'll repost it here then :)
>
> "Here is what I had to do to get pig running with a different version of
> Hadoop (in my case, the cloudera build but I'd try this as well):

>
> build pig-withouthadoop.jar by running "ant jar-withouthadoop". Then, when
> you run pig, put the pig-withouthadoop.jar on your classpath as well as your
> hadoop jar. In my case, I found that scripts only worked if I additionally
> manually registered the antlr jar:

Thanks Jonathan! I will give it a shot.

>
> register /path/to/pig/build/ivy/lib/Pig/antlr-runtime-3.2.jar;"

Is this a windows command? Sorry, have not used this before.

>
> 2011/5/26 Mohit Anchlia <mohitanchlia@...>
>
>> For some reason I don't see that reply from Jonathan in my Inbox. I'll
>> try to google it.
>>
>> What should be my next step in that case? I can't use pig then?
>>
>> On Thu, May 26, 2011 at 10:00 AM, Harsh J <harsh@...> wrote:
>> > I think Jonathan Coveney's reply on user <at> pig answered your question.
(Continue reading)

Mohit Anchlia | 26 May 19:55 2011
Picon

Re: Help with pigsetup

I've built pig-withouthadoop.jar and have copied it to my linux box.
Now how do I put hadoop-core-0.20.203.0.jar and pig-withouthadoop.jar
in the classpath. Is it by using CLASSPATH variable?

On Thu, May 26, 2011 at 10:18 AM, Mohit Anchlia
<mohitanchlia@...> wrote:
> On Thu, May 26, 2011 at 10:06 AM, Jonathan Coveney
<jcoveney@...> wrote:
>> I'll repost it here then :)
>>
>> "Here is what I had to do to get pig running with a different version of
>> Hadoop (in my case, the cloudera build but I'd try this as well):
>
>>
>> build pig-withouthadoop.jar by running "ant jar-withouthadoop". Then, when
>> you run pig, put the pig-withouthadoop.jar on your classpath as well as your
>> hadoop jar. In my case, I found that scripts only worked if I additionally
>> manually registered the antlr jar:
>
> Thanks Jonathan! I will give it a shot.
>
>>
>> register /path/to/pig/build/ivy/lib/Pig/antlr-runtime-3.2.jar;"
>
> Is this a windows command? Sorry, have not used this before.
>
>>
>> 2011/5/26 Mohit Anchlia <mohitanchlia@...>
>>
>>> For some reason I don't see that reply from Jonathan in my Inbox. I'll
(Continue reading)

Mohit Anchlia | 26 May 20:20 2011
Picon

Re: Help with pigsetup

I added to PIG_CLASSPATH and went past the error but now I get a
different error. Looks like I need to add some other jars but not sure
which one.

export PIG_CLASSPATH=$HADOOP_CONF_DIR:$HADOOP_HOME/hadoop-core-0.20.203.0.jar:$PIG_HOME/../pig-withouthadoop.jar

ERROR 2998: Unhandled internal error.
org/apache/commons/configuration/Configuration

java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
        at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
        at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
        at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
        at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
        at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
        at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:109)
        at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
        at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:196)
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:116)
        at org.apache.pig.impl.PigContext.connect(PigContext.java:187)
        at org.apache.pig.PigServer.<init>(PigServer.java:243)
        at org.apache.pig.PigServer.<init>(PigServer.java:228)
        at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:46)
(Continue reading)

Mohit Anchlia | 26 May 21:01 2011
Picon

Re: Help with pigsetup

I added all the jars in the classpath in HADOOP_HOME/lib and now I get
to the grunt prompt. Will try the tutorials and see how it behaves :)

Thanks for your help!

On Thu, May 26, 2011 at 9:56 AM, Mohit Anchlia
<mohitanchlia@...> wrote:
> I sent this to pig apache user mailing list but have got no response.
> Not sure if that list is still active.
>
> thought I will post here if someone is able to help me.
>
> I am in process of installing and learning pig. I have a hadoop
> cluster and when I try to run pig in mapreduce mode it errors out:
>
> Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1
>
> Error before Pig is launched
> ----------------------------
> ERROR 2999: Unexpected internal error. Failed to create DataStorage
>
> java.lang.RuntimeException: Failed to create DataStorage
>       at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
>       at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
>       at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
>       at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
>       at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
>       at org.apache.pig.PigServer.<init>(PigServer.java:226)
>       at org.apache.pig.PigServer.<init>(PigServer.java:215)
>       at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
(Continue reading)


Gmane