Technical discussion about the MICO ORB ()

Rob Ratcliff | 18 Oct 2011 17:04

idl compiled multi-threaded core dumps when loading CORBA.idl

Hi,

I compiled MICO with multi-thread enabled (and CSIV2). When I attempt to load the CORBA.idl into the
interface repository using
idl --feed-ir CORBA.idl

I get the following stack trace from the core dump:

(gdb) where
#0  0x0013a416 in __kernel_vsyscall ()
#1  0x003052f1 in raise () from /lib/libc.so.6
#2  0x00306d5e in abort () from /lib/libc.so.6
#3  0x007f6b1b in ?? () from /usr/lib/libstdc++.so.6
#4  0x007f6b53 in std::terminate() () from /usr/lib/libstdc++.so.6
#5  0x007f6cd2 in __cxa_throw () from /usr/lib/libstdc++.so.6
#6  0x00d0dc60 in CORBA::BAD_OPERATION::_throwit (this=0x9239be8) at ../include/mico/sysexc.h:36
#7  0x00d3d6ac in mico_throw (r=0xbffee144) at ../include/mico/throw.h:88
#8  mico_sii_throw (r=0xbffee144) at ../include/mico/throw.h:130
#9  0x00d41488 in CORBA::Contained_stub::id (this=0x9250a80) at ir.cc:331
#10 0x081ab7dc in IRCopy::is_included_def (this=0xbffee3b0, src=0x9250a80) at ir-copy.cc:123
#11 0x081b0d8b in IRCopy::copy (this=0xbffee3b0, src=0x92bd658, feed_included_defs=true) at ir-copy.cc:153
#12 0x081b0f4d in IRCopy::copy (this=0xbffee3b0, src=0x925cfe0, feed_included_defs=true) at ir-copy.cc:181
#13 0x081b0f4d in IRCopy::copy (this=0xbffee3b0, src=0x925bff0, feed_included_defs=true) at ir-copy.cc:181
#14 0x081b0f4d in IRCopy::copy (this=0xbffee3b0, src=0x9213910, feed_included_defs=true) at ir-copy.cc:181
#15 0x081b11bf in IRCopier (db=..., params=..., repo=0x922a860, cont=0x9213910) at ir-copy.cc:1628
#16 0x08089b23 in main (argc=17159044, argv=0x105d40c) at main.cc:357

I've attached my MakeVars for reference.

I'm just beginning to troubleshoot it, but if anybody has some idea why this would be happening, please let

(Continue reading)

headers

Rob Ratcliff | 19 Oct 2011 15:31

Re: idl compiled multi-threaded core dumps when loading CORBA.idl

Here's another update: I recompiled mico with threads disabled and noticed that the single-threaded idl
program crashed when
communicating with the multi-threaded ird, but not with the single-threaded ird. So it appears that the
crash is related to enabling
threads in the ird.

On 10/18/2011 10:04 AM, Rob Ratcliff wrote:
> Hi,
>
> I compiled MICO with multi-thread enabled (and CSIV2). When I attempt to load the CORBA.idl into the
interface repository using
> idl --feed-ir CORBA.idl
>
> I get the following stack trace from the core dump:
>
> (gdb) where
> #0  0x0013a416 in __kernel_vsyscall ()
> #1  0x003052f1 in raise () from /lib/libc.so.6
> #2  0x00306d5e in abort () from /lib/libc.so.6
> #3  0x007f6b1b in ?? () from /usr/lib/libstdc++.so.6
> #4  0x007f6b53 in std::terminate() () from /usr/lib/libstdc++.so.6
> #5  0x007f6cd2 in __cxa_throw () from /usr/lib/libstdc++.so.6
> #6  0x00d0dc60 in CORBA::BAD_OPERATION::_throwit (this=0x9239be8) at ../include/mico/sysexc.h:36
> #7  0x00d3d6ac in mico_throw (r=0xbffee144) at ../include/mico/throw.h:88
> #8  mico_sii_throw (r=0xbffee144) at ../include/mico/throw.h:130
> #9  0x00d41488 in CORBA::Contained_stub::id (this=0x9250a80) at ir.cc:331
> #10 0x081ab7dc in IRCopy::is_included_def (this=0xbffee3b0, src=0x9250a80) at ir-copy.cc:123
> #11 0x081b0d8b in IRCopy::copy (this=0xbffee3b0, src=0x92bd658, feed_included_defs=true) at ir-copy.cc:153
> #12 0x081b0f4d in IRCopy::copy (this=0xbffee3b0, src=0x925cfe0, feed_included_defs=true) at ir-copy.cc:181
> #13 0x081b0f4d in IRCopy::copy (this=0xbffee3b0, src=0x925bff0, feed_included_defs=true) at ir-copy.cc:181

(Continue reading)

headers

Rob Ratcliff | 19 Oct 2011 16:23

Re: idl compiled multi-threaded core dumps when loading CORBA.idl

FYI:

Here's the debug output from idl just prior to core dumping:

IIOPProxy::add_invoke: rec=0xd0e8328, id=0xf056878, msgid=38143)
MICO::GIOPConn::output (CORBA::Buffer *b)
b: 0xcdf8b50
Out Data 47 49 4f 50 01 00 01 00 44 00 00 00 00 00 00 00 GIOP....D.......
ff 94 00 00 01 00 00 00 17 00 00 00 2f 31 36 30 ��........../160
37 35 2f 31 33 31 39 30 33 31 38 39 36 2f 30 2f 75/1319031896/0/
5f 32 39 00 0e 00 00 00 5f 73 65 74 5f 63 6f 6e _29....._set_con
74 65 78 74 73 00 00 00 00 00 00 00 00 00 00 00 texts...........
MICO::GIOPConn::input_ready ()
conn: 0x99c2af8
ev: GIOPConnCallback::InputReady
t_mod: 1
pool:
conn: 
_activerefs: 3
In Data 47 49 4f 50 01 00 01 01 0c 00 00 00 00 00 00 00 GIOP............
ff 94 00 00 00 00 00 00 ��......
IIOP: incoming data from inet:192.168.1.102:9005
GIOP: incoming Reply from inet:192.168.1.102:9005 for msgid 38143 status is 0
ORB::get_invoke (MsgId=38143)
IIOPProxy::pull_invoke: id=0xf056878, rec = 0xd0e8328
IIOPProxy::handle_invoke_reply: rec=0xd0e8328)
IIOPProxy::del_invoke: rec = 0xd0e8328
MICO::IIOPProxy::exec_invoke_reply (obj=0, *req=0xbfe6ce28, *conn=0x99c2af8)
GIOPConn::deref: 0x99c2af8, refcnt: 1, activerefs: 3
ORB::wait for 0xf056878

(Continue reading)

headers

Karel Gardas | 20 Oct 2011 13:04

Re: idl compiled multi-threaded core dumps when loading CORBA.idl

Hi Rob,

indeed, this would point to some race-condition in ird itself. However 
even part or not majority of interface repository (at least this 
contained in libmicoir) should be thread-safe as this is also used in 
ObjectWall[1] and its generally thread-safe and tested a lot under hight 
load in multi-threading setup on multi-cpu servers. So what you perhaps 
see is relict of single-threaded MICO which should be fixed. If you do 
have already a patch for this or some attempt on it, please send it to 
here for review.

Thanks!
Karel

[1]: http://www.objectsecurity.com/en-products-objectwall.html

On 10/19/11 03:31 PM, Rob Ratcliff wrote:
> Here's another update: I recompiled mico with threads disabled and noticed that the single-threaded idl
program crashed when
> communicating with the multi-threaded ird, but not with the single-threaded ird. So it appears that the
crash is related to enabling
> threads in the ird.
>
> On 10/18/2011 10:04 AM, Rob Ratcliff wrote:
>> Hi,
>>
>> I compiled MICO with multi-thread enabled (and CSIV2). When I attempt to load the CORBA.idl into the
interface repository using
>> idl --feed-ir CORBA.idl

(Continue reading)

headers

Rob Ratcliff | 24 Oct 2011 06:35

Re: idl compiled multi-threaded core dumps when loading CORBA.idl

Karel,

I haven't found the root cause of the problem yet.  I'm running both the "idl --feed-ir" command and the "ird"
in multi-threaded
mode. I've noticed that I can make a slight change to a method signature such as:

typedef sequence<string> Items; (defined in an include file)

...
interface Test {
...
    Items getItems();
...
}

vs.
...
interface Test {
...
    sequence<string> getItems();
...
}

The second will work and the first will not. I haven't been able to distill it to a small test case, since
deleting the
"non-offensive" IDL from the files before loading it will fix the problem with the formerly "offensive" IDL.

But, thankfully, for a given IDL file, the  "Bad Operation" call will always happen at the same place in the
IDL. For one case I'm
debugging, it happens during the second pass of IRCopy::copy command during the "is_included_def(cs)"

(Continue reading)

headers

Rob Ratcliff | 26 Oct 2011 05:25

Re: idl compiled multi-threaded core dumps when loading CORBA.idl

Here's more info on the problem:

When I defined the method in the IDL as:

   JobIds getFailedJobIds();

with JobIds defined in an included file as:
typedef JobIds sequence<string>;

The definition for this method when evaluated here appears to be a "Container" with a repository ID of
"IDL:omg.org/CORBA/OperationDef:1.00"

class IRCopy {
...
bool IRCopy::copy (CORBA::Container_ptr src,  bool feed_included_defs) {
  ....
   dummy2 = CORBA::Container::_narrow (contents[i]);
    if (!CORBA::is_nil (dummy2)) {
      if (!copy (dummy2, _feed_included_defs)) {
    return false;
      }
    }
...

which leads to the exception of:

 uncaught MICO exception: IDL:omg.org/CORBA/BAD_OPERATION:1.0 (0, not-completed)

when an attempt is made to call id() on the dummy2 reference.

(Continue reading)

headers

Rob Ratcliff | 26 Oct 2011 05:55

Re: idl compiled multi-threaded core dumps when loading CORBA.idl


Even though OperationDef is not a Container, a Container_stub is used. I wonder if the "!strcmp" logic is
incorrect when there is an
OperationDef?

ir.cc
...
CORBA::Container::_narrow( CORBA::Object_ptr _obj )
{
  CORBA::Container_ptr _o;
  if( !CORBA::is_nil( _obj ) ) {
    void *_p;
    if( (_p = _obj->_narrow_helper( "IDL:omg.org/CORBA/Container:1.0" )))
      return _duplicate( (CORBA::Container_ptr) _p );
    if (!strcmp (_obj->_repoid(), "IDL:omg.org/CORBA/Container:1.0") || _obj->_is_a_remote
("IDL:omg.org/CORBA/Container:1.0")) {
      _o = new CORBA::Container_stub;
      _o->CORBA::Object::operator=( *_obj );
      return _o;
    }
  }
  return _nil();
}

On 10/25/2011 10:25 PM, Rob Ratcliff wrote:
> Here's more info on the problem:
>
> When I defined the method in the IDL as:
>
>    JobIds getFailedJobIds();

(Continue reading)

headers

Rob Ratcliff | 26 Oct 2011 16:26

Re: idl compiled multi-threaded core dumps when loading CORBA.idl

Looks like some lookup table in the ird gets corrupted, since it appears that the "ird" thinks that the given
"OperationDef"  is_a
Container since this code returns "True" where obj is a "OperationDef" and repoid is "IDL:omg.org/CORBA/Container:1.0:

object.cc: (called from ir.cc below)
CORBA::Boolean
CORBA::Object::_is_a_remote (const char *repoid)
{
    // only ask remote if we are a stub
    if (_orbnc()->is_impl (this))
    return FALSE;
    if (!ior)
        return FALSE;
    return _orbnc()->is_a (this, repoid);
}

orb.cc:
...
CORBA::Boolean CORBA::ORB::is_a (Object_ptr obj, const char *repo_id)
...
Request_var req = obj->_request ("_is_a");
    req->add_in_arg ("logical_type_id") <<= repo_id;
    req->result()->value()->set_type (CORBA::_tc_boolean);
    MICO_CATCHANY(req->invoke ());
    if (req->env()->exception())
        mico_throw (*req->env()->exception());
    Boolean res;
    CORBA::Boolean r = (*req->result()->value() >>= Any::to_boolean (res));
    assert (r);

(Continue reading)