gmane.comp.lang.tcl.core

Re: GSoC ShlibMemLoad


> >> They all have only one actual function greet(). This function get pid
> >> of the process which calls it and tries to write it to stdout.
> >> libwrite_simple makes it in the easiest possible way: just uses one
> >> write sys call.
> >> libwrite_global makes it similarly, but a buffer is global now.
> >> libwrite_fun calls another function (fill_str to fill buffer with data)
> >> These three libraries seems to be working fine.
> >> The fourth library libprintf tries to make the same thing easier and
> >> use printf instead of write. Anyway, it does not work. I get a
> >> segmentation fault (the same happens when I substitute printf for any
> >> other functions from stdio.h like puts).
>
> >> I do not know what may be wrong but I will try to work on it. Maybe
> >> the problem is in function load_needed_objects(Obj_Entry *) which is
> >> called by elf_dlopen, but I still do not know why it works for the
> >> first three libraries. They use functions from unistd.h.
>
> >I can't say at the moment. It seems that you haven't committed all the
> >changes yet, so I have can't see what was done.
>
> >For example, it is not clear if you are using the OS dlsym() already, or
> >not.
>
> >Best guess I have right now, is that it might be that your non-working
> >library depends on some other library FOO instead of just libc. And while
> >libc symbols are present the FOO is not loaded (yet) and thus its symbols
> >are not present either?
>
> As far as I know printf function is in libc as well. I also tried to

(Continue reading)

headers

Daniel Hans | 5 Jul 12:03

Re: GSoC ShlibMemLoad

2008/7/4 Andreas Kupries <andreask@...>:
>
>> >> They all have only one actual function greet(). This function get pid
>> >> of the process which calls it and tries to write it to stdout.
>> >> libwrite_simple makes it in the easiest possible way: just uses one
>> >> write sys call.
>> >> libwrite_global makes it similarly, but a buffer is global now.
>> >> libwrite_fun calls another function (fill_str to fill buffer with data)
>> >> These three libraries seems to be working fine.
>> >> The fourth library libprintf tries to make the same thing easier and
>> >> use printf instead of write. Anyway, it does not work. I get a
>> >> segmentation fault (the same happens when I substitute printf for any
>> >> other functions from stdio.h like puts).
>>
>> >> I do not know what may be wrong but I will try to work on it. Maybe
>> >> the problem is in function load_needed_objects(Obj_Entry *) which is
>> >> called by elf_dlopen, but I still do not know why it works for the
>> >> first three libraries. They use functions from unistd.h.
>>
>> >I can't say at the moment. It seems that you haven't committed all the
>> >changes yet, so I have can't see what was done.
>>
>> >For example, it is not clear if you are using the OS dlsym() already, or
>> >not.
>>
>> >Best guess I have right now, is that it might be that your non-working
>> >library depends on some other library FOO instead of just libc. And while
>> >libc symbols are present the FOO is not loaded (yet) and thus its symbols
>> >are not present either?
>>

(Continue reading)

headers

Andreas Kupries | 5 Jul 21:35

Re: GSoC ShlibMemLoad


> > While walking to the office I managed to come up with a number of additional
> > ideas ...
> >
> > printf is different from the other functions in two respects.
> >
> >        It takes a variable number of arguments,
> > and     it operates at a higher IO level than write and consorts.
> >
> >        Both might be a possible source of the problem I guess.
> >        Although I have no idea of how that could be.
> >
> > Things to consider:
> >
> >        - Does using 'sprintf()' (+write) work ?
> >                sprintf is also varargs, but doesn't do any I/O
> sprintf does not work
> >
> >        - Does 'fprintf()' work ?
> >                It is like printf, i.e. varargs and highlevel I/O
> >                However it explicitly takes an output argument.
> >                printf (...) is like fprintf (stdout, ...).
> fprintf does not work
> >        - Does fflush() work?
> >                Highlevel I/O, but no varargs. It is also outside
> >                of the printf family of functions in general.
> fflush surprisingly works...
> Anyway, the problem is not in varargs, because I also tried with
> functions like putchar, puts, etc and they all does not work.

(Continue reading)

headers

Daniel Hans | 6 Jul 22:02

Re: GSoC ShlibMemLoad

2008/7/5 Andreas Kupries <akupries@...>:
>
>> > While walking to the office I managed to come up with a number of additional
>> > ideas ...
>> >
>> > printf is different from the other functions in two respects.
>> >
>> >        It takes a variable number of arguments,
>> > and     it operates at a higher IO level than write and consorts.
>> >
>> >        Both might be a possible source of the problem I guess.
>> >        Although I have no idea of how that could be.
>> >
>> > Things to consider:
>> >
>> >        - Does using 'sprintf()' (+write) work ?
>> >                sprintf is also varargs, but doesn't do any I/O
>> sprintf does not work
>> >
>> >        - Does 'fprintf()' work ?
>> >                It is like printf, i.e. varargs and highlevel I/O
>> >                However it explicitly takes an output argument.
>> >                printf (...) is like fprintf (stdout, ...).
>> fprintf does not work
>> >        - Does fflush() work?
>> >                Highlevel I/O, but no varargs. It is also outside
>> >                of the printf family of functions in general.
>> fflush surprisingly works...
>> Anyway, the problem is not in varargs, because I also tried with
>> functions like putchar, puts, etc and they all does not work.

(Continue reading)

headers

Arnulf Wiedemann | 10 Jul 12:12

Re: GSoC ShlibMemLoad

Am Sonntag, 6. Juli 2008 22:02:19 schrieb Daniel Hans:
> 2008/7/5 Andreas Kupries <akupries@...>:
> >> > While walking to the office I managed to come up with a number of
> >> > additional ideas ...
> >> >
> >> > printf is different from the other functions in two respects.
> >> >
> >> >        It takes a variable number of arguments,
> >> > and     it operates at a higher IO level than write and consorts.
> >> >
> >> >        Both might be a possible source of the problem I guess.
> >> >        Although I have no idea of how that could be.
> >> >
> >> > Things to consider:
> >> >
> >> >        - Does using 'sprintf()' (+write) work ?
> >> >                sprintf is also varargs, but doesn't do any I/O
> >>
> >> sprintf does not work
> >>
> >> >        - Does 'fprintf()' work ?
> >> >                It is like printf, i.e. varargs and highlevel I/O
> >> >                However it explicitly takes an output argument.
> >> >                printf (...) is like fprintf (stdout, ...).
> >>
> >> fprintf does not work
> >>
> >> >        - Does fflush() work?
> >> >                Highlevel I/O, but no varargs. It is also outside
> >> >                of the printf family of functions in general.

(Continue reading)

headers

Daniel Hans | 10 Jul 17:19

Re: GSoC ShlibMemLoad

Thank you very much. I am extremely grateful for his email 

2008/7/10 Arnulf Wiedemann <arnulf@...>:
> Am Sonntag, 6. Juli 2008 22:02:19 schrieb Daniel Hans:
>> 2008/7/5 Andreas Kupries <akupries@...>:
>> >> > While walking to the office I managed to come up with a number of
>> >> > additional ideas ...
>> >> >
>> >> > printf is different from the other functions in two respects.
>> >> >
>> >> >        It takes a variable number of arguments,
>> >> > and     it operates at a higher IO level than write and consorts.
>> >> >
>> >> >        Both might be a possible source of the problem I guess.
>> >> >        Although I have no idea of how that could be.
>> >> >
>> >> > Things to consider:
>> >> >
>> >> >        - Does using 'sprintf()' (+write) work ?
>> >> >                sprintf is also varargs, but doesn't do any I/O
>> >>
>> >> sprintf does not work
>> >>
>> >> >        - Does 'fprintf()' work ?
>> >> >                It is like printf, i.e. varargs and highlevel I/O
>> >> >                However it explicitly takes an output argument.
>> >> >                printf (...) is like fprintf (stdout, ...).
>> >>
>> >> fprintf does not work
>> >>

(Continue reading)

headers

Andreas Kupries | 7 Jul 05:26

Re: GSoC ShlibMemLoad


> > Some leading questions ...
> >
> > You are using 'printf' in rtld itself to print log/debug messages, and
> > it works there. What is the address of this 'printf' function ?
> >
> > Is the address of the 'printf' function seen by the loaded
> > test-library the same ? (**)
> >
> > If not, why?

> These addresses differ. I am not sure why, but I think that printf in
> log/debug messages uses printf from libc.so which was loaded at the
> beginning of rtld execution by the regular dynamic loader and printf
> in libprintf.so uses printf from libc.so loaded by
> load_needed_objects.

Yes, they differed for me as well, and I am drawing the same conclusion.

> > Do you remember my comments about missing symbols ?
> >
> >        I.e.: The list of libraries we have in our rtld is the list of
> >        memory-loaded libraries (ML/lib) and not of all libraries
> >        (ALL/lib).
> >
> >        And a symbol not found in ML/lib has to be looked for in
> >        ALL/lib too ?
> >
> > What does that same distinction mean for 'load_needed_objects' ?

(Continue reading)

headers

Tomasz Kosiak | 7 Jul 09:08

Re: GSoC ShlibMemLoad

On Mon, Jul 7, 2008 at 5:28 AM, Andreas Kupries <akupries@...> wrote:

[...]

>> Because I have already studied this whole code for quite a long time
>> and have not came up with any solution I am thinking of writing an
>> email to John Polstra who is an author of the rtld for
>> FreeBSD. Maybe he will be able to help out. Do you think it is a
>> good idea?
>
> Yes. Getting the help of (one of) the original authors of some code is
> (in my opinion) a good idea. He should know the intricacies of the
> code.

As Daniel intends to contact John Polstra from FreeBSD I think that it
would be easier to work with him if Daniel installs FreeBSD and will
convey his experiments (and work with John) in this enviroment. I
would recommend installing FreeBSD on separate machine or in some kind
of virtual enviroment like VMware or VirtualBox to make simultaneous
testing easy (such a FreeBSD installation could run in runlevel 3
without graphical console to limit resources requirements).

>From high level perspective what we need is support in OS dynamic
linker to link with some libraries loaded by our program into memory
from some virtual filesystem specific to application (ex. starkit for
Tcl). Such feature can benefit not only Tcl, but also others who think
about single file deployments. As far as I understand MacOSX already
has got this functionality and other OS-es lack it. We currently try
to emulate it in Linux using code from FreeBSD. But maybe first we
should try to make it work in FreeBSD. FreeBSD folks can even maybe

(Continue reading)

headers

Andreas Kupries | 7 Jul 21:37

Re: GSoC ShlibMemLoad

> On Mon, Jul 7, 2008 at 5:28 AM, Andreas Kupries <akupries@...> wrote:
>
> [...]
>
> >> Because I have already studied this whole code for quite a long time
> >> and have not came up with any solution I am thinking of writing an
> >> email to John Polstra who is an author of the rtld for
> >> FreeBSD. Maybe he will be able to help out. Do you think it is a
> >> good idea?
> >
> > Yes. Getting the help of (one of) the original authors of some code is
> > (in my opinion) a good idea. He should know the intricacies of the
> > code.
>
> As Daniel intends to contact John Polstra from FreeBSD I think that it
> would be easier to work with him if Daniel installs FreeBSD and will
> convey his experiments (and work with John) in this enviroment. I
> would recommend installing FreeBSD on separate machine or in some kind
> of virtual enviroment like VMware or VirtualBox to make simultaneous
> testing easy (such a FreeBSD installation could run in runlevel 3
> without graphical console to limit resources requirements).

I have no big opinion here. I have never worked with FreeBSD, so I have no
idea how much effort it will be to set it up.

An image for some virtual machine is likely the easiest way to go for that.

A quick google for 'freebsd vmware image' gave me

	http://people.freebsd.org/~matusita/VMware/

(Continue reading)

headers

Andreas Kupries | 7 Jul 22:06

Re: GSoC ShlibMemLoad


As a general note, for those interested in looking at things deeper, the
project repository is available at

	andreask <at> gila:~/workbench/shmemload/gsoc2008-tcl_dynamic_libraries> svn
info
	Path: .
	URL: http://svn.assembla.com/svn/gsoc2008-tcl_dynamic_libraries
	Repository UUID: d7ba3245-3e7b-42d0-9cd4-356c8b94b330
	Revision: 6
	Node Kind: directory
	Schedule: normal
	Last Changed Author: dani3l
	Last Changed Rev: 6
	Last Changed Date: 2008-07-06 13:04:21 -0700 (Sun, 06 Jul 2008)

--
	Andreas Kupries <andreask@...>
	Developer @ http://www.ActiveState.com
	Tel: +1 778-786-1122

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08

headers

Donal K. Fellows | 8 Jul 10:00

Re: GSoC ShlibMemLoad

Andreas Kupries wrote:
> Solaris and other unixes, except the oldest (*), are all based on ELF. We
> should be able to use the project results for them as well. windows uses PE
> and/or COFF if I remember correctly, that would need a separate branch, yes.
> And I agree that MS is likely not receptive to such a proposal.

To be clearer, even just doing ELF would be a good step forward as that
would still tackle many platforms that are currently in use. For the
others, falling back to the existing "copy out and load that" would do.
(Mind you, if we could handle Windows DLLs too, that'd be great! It
would cover just about everything that comes up in practice.)

Donal.

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08

headers

Daniel Hans | 7 Jul 20:08

Re: GSoC ShlibMemLoad

2008/7/7 Andreas Kupries <akupries@...>:
>
>> > Some leading questions ...
>> >
>> > You are using 'printf' in rtld itself to print log/debug messages, and
>> > it works there. What is the address of this 'printf' function ?
>> >
>> > Is the address of the 'printf' function seen by the loaded
>> > test-library the same ? (**)
>> >
>> > If not, why?
>
>> These addresses differ. I am not sure why, but I think that printf in
>> log/debug messages uses printf from libc.so which was loaded at the
>> beginning of rtld execution by the regular dynamic loader and printf
>> in libprintf.so uses printf from libc.so loaded by
>> load_needed_objects.
>
> Yes, they differed for me as well, and I am drawing the same conclusion.
>
>> > Do you remember my comments about missing symbols ?
>> >
>> >        I.e.: The list of libraries we have in our rtld is the list of
>> >        memory-loaded libraries (ML/lib) and not of all libraries
>> >        (ALL/lib).
>> >
>> >        And a symbol not found in ML/lib has to be looked for in
>> >        ALL/lib too ?
>> >
>> > What does that same distinction mean for 'load_needed_objects' ?

(Continue reading)

headers

Andreas Kupries | 7 Jul 21:24

Re: GSoC ShlibMemLoad

> >> > (**) In (one of) the other test libraries you are using a
> for-loop and
> >> >     write to print the pid. If you encapsulate that into a function
> >> >     FOO to generally print unsigned numbers and then put FOO into the
> >> >     libprintf code you can print any number you need, like function
> >> >     addresses.  Inside of rtld itself it is of course possible to use
> >> >     the working printf to print its address.
> >
> >> Because I have already studied this whole code for quite a long time
> >> and have not came up with any solution I am thinking of writing an
> >> email to John Polstra who is an author of the rtld for
> >> FreeBSD. Maybe he will be able to help out. Do you think it is a
> >> good idea?
> >
> > Yes. Getting the help of (one of) the original authors of some code is
> > (in my opinion) a good idea. He should know the intricacies of the
> > code.
>
> I wrote the email today in the morning.

> >
> >> Today I also wrote a simple program which generates defs.h file.
> >
> > That note I did not understand. Can you explain a bit more?
>
> I committed make_defs.c file which automatically generates file
> elf-defs.h (the same as defs.h).
> An argument for this program should be the program itself, like
> ./make_defs make_defs

(Continue reading)

headers

Andreas Kupries | 7 Jul 22:09

Re: GSoC ShlibMemLoad


> >> Because I have already studied this whole code for quite a long time
> >> and have not came up with any solution I am thinking of writing an
> >> email to John Polstra who is an author of the rtld for
> >> FreeBSD. Maybe he will be able to help out. Do you think it is a
> >> good idea?
> >
> > Yes. Getting the help of (one of) the original authors of some code is
> > (in my opinion) a good idea. He should know the intricacies of the
> > code.
>
> I wrote the email today in the morning.

Thank you. Please add us (*) to the CC: and/or To: list of your conversation
with John.

(*) DanielS, Tomasz, Tcl-core, and myself.

--
	Andreas Kupries <andreask@...>
	Developer @ http://www.ActiveState.com
	Tel: +1 778-786-1122

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08