Mehendran T | 5 Nov 11:41 2007
Picon

fileno support is not in jython. Reason?

Hi,

I need to access fileno(file descriptor number) of a file. But that function is 
not available in jython. 

Is there any specific reason behind that that is omitted currently? 

-Mehendran

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
Alan Kennedy | 5 Nov 13:28 2007

Re: fileno support is not in jython. Reason?

[Mehendran]
> I need to access fileno(file descriptor number) of a file. But that function is
> not available in jython.

Yes it is available; it's just not returning what you (perhaps
mistakenly) expect.

Jython 2.2.1 on java1.4.2_15
Type "copyright", "credits" or "license" for more information.
>>> import socket
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s.connect( ('localhost', 80) )
>>> s.fileno()
java.nio.channels.SocketChannel[connected local=/127.0.0.1:1622
remote=localhost/127.0.0.1:80]
>>> s.getchannel()
java.nio.channels.SocketChannel[connected local=/127.0.0.1:1622
remote=localhost/127.0.0.1:80]
>>>

Both the socket.fileno() method and the jython-specific
socket.getchannel() method return a java specific
java.nio.channels.SocketChannel object, which is the only thing that
it is possible to return on the java platform.

[Mehendran]
> Is there any specific reason behind that that is omitted currently?

1. It is not omitted
2. Jython cannot return a numeric file-descriptor because java has no
(Continue reading)

Mehendran T | 5 Nov 15:07 2007
Picon

Re: fileno support is not in jython. Reason?

Hi Alan,

Thanks for that you have given more info about select module.

Actually I am trying to implement mmap module. I found that 
the constructor for mmap uses file descriptor value to get the file handle.
Then I checked with jython. But it is not supporting that fileno. 
I don't want to change the signature of the constructor.
So please help me to solve this.

Will it be nice to implement that in PyFile to return 
FileDescriptor java object?  
If we do so, the return value will be a java object and it will violate
the fileno() and mmap() contract, Isn't it?

-Mehendran 

 
>>> "Alan Kennedy" <jython-dev <at> xhaus.com> 11/05/07 5:58 PM >>> 
[Mehendran]
> I need to access fileno(file descriptor number) of a file. But that function is
> not available in jython.

Yes it is available; it's just not returning what you (perhaps
mistakenly) expect.

Jython 2.2.1 on java1.4.2_15
Type "copyright", "credits" or "license" for more information.
>>> import socket
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
(Continue reading)

Alan Kennedy | 5 Nov 17:33 2007

Re: fileno support is not in jython. Reason?

[Mehendran]
> Actually I am trying to implement mmap module.

Sorry Mehendran, I see now that I jumped the gun by assuming that you
were asking about the socket.fileno() method; actually you are looking
at something entirely different.

[Mehendran]
> I found that
> the constructor for mmap uses file descriptor value to get the
> file handle. Then I checked with jython. But it is not supporting
> that fileno. I don't want to change the signature of the constructor.
> So please help me to solve this.

Well, there are a number of things to say here.

Number one is that, strictly speaking, in my opinion, using fileno()
in the mmap API is not portable, in the same way fileno() is not
portable if used on sockets. Integer file descriptors are an OS
specific thing. Far better for the mmap API to be changed so that it
accepts an actual file object as the first parameter; this is more
intuitively obvious, and more widely implementable and portable.

Given what Guido has already stated about the platform-specificity of
fileno(), in relation to sockets and select

http://www.mail-archive.com/python-dev <at> python.org/msg23202.html

I think we have a strong case to get the cpython folks to change the
cpython mmap API.
(Continue reading)

Mehendran T | 6 Nov 08:46 2007
Picon

Re: fileno support is not in jython. Reason?

Hi,

Thanks for your more valuable info and full support. It feels me great. 

Alan:
I am using fileobj in the constructor argument to mmap as you said. 
I think that will be fine and easier and we have strong reason.

Philip:
> fileno isn't implemented yet. I will make it return the underlying  
>'RawIO' object, which is pretty much a wrapper for a java nio Channel  
>(or some other underlying 'raw' io object).
Sorry, I couldn't get you clearly. Do you mean to say that fileno() will 
return the filechannel obj? This also sounds good. 

I got stuck again. I started with fileobj anyway. If need be, I can change 
it to. However, please tell me which would be nice, fileobj or nio-channel 
considering sockets and all?

And, I won't be available for next 2-3 days. I will catch you 
once I will be back. 

-Mehendran.

 
>>> "Alan Kennedy" <jython-dev <at> xhaus.com> 11/05/07 10:03 PM >>> 
[Mehendran]
> Actually I am trying to implement mmap module.

Sorry Mehendran, I see now that I jumped the gun by assuming that you
(Continue reading)

Philip Jenvey | 6 Nov 16:15 2007

Re: fileno support is not in jython. Reason?


On Nov 5, 2007, at 8:33 AM, Alan Kennedy wrote:
>
> Well, there are a number of things to say here.
>
> Number one is that, strictly speaking, in my opinion, using fileno()
> in the mmap API is not portable, in the same way fileno() is not
> portable if used on sockets. Integer file descriptors are an OS
> specific thing. Far better for the mmap API to be changed so that it
> accepts an actual file object as the first parameter; this is more
> intuitively obvious, and more widely implementable and portable.
>
> Given what Guido has already stated about the platform-specificity of
> fileno(), in relation to sockets and select
>
> http://www.mail-archive.com/python-dev <at> python.org/msg23202.html
>
> I think we have a strong case to get the cpython folks to change the
> cpython mmap API.

We can't really make the non-portable argument any longer now that  
we're supporting faux file descriptors, though. IronPython also  
supports fake file descriptors (as integers FYI).

>
> [Mehendran]
>> Will it be nice to implement that in PyFile to return
>> FileDescriptor java object?
>> If we do so, the return value will be a java object and it will
>> violate the fileno() and mmap() contract, Isn't it?
(Continue reading)

Alan Kennedy | 7 Nov 11:39 2007

Re: fileno support is not in jython. Reason?

Philip, many thanks for your hard work on java.nio PyFile's; it will
modernise jython I/O in a neat and elegant fashion, and enable jython
to capitalise on all the really excellent features in java.nio, with
performance being highest on the list (or my list at least :-)

[Alan]
>> I think we have a strong case to get the cpython folks to change the
>> cpython mmap API.

[Philip]
> We can't really make the non-portable argument any longer now that
> we're supporting faux file descriptors, though. IronPython also
> supports fake file descriptors (as integers FYI).

Hmm, do you mean "now that we CAN support faux FDs"? Note that we can
support faux FDs on files *only*, not on any other type of I/O stream,
e.g. sockets, pipes, etc, so we would be introducing an asymmetry into
jython by supporting them. If we do support them, I think we will then
find that users will expect to be able to pass these faux file
descriptors to e.g. select.select, etc, and expect them to be operated
upon, e.g. multiplexed. This cannot happen on jython;
java.nio.FileChannels do NOT implement SelectableChannel; FileChannels
can never be multiplexed.

I seem to remember Mehendran saying that he chose to make the first
parameter of mmap a file OBJECT, not a file descriptor. I think this
is the right way to go; mmap taking a file descriptor is an historical
accident, IMHO, because mmap was first implemented in a language
(cpython) that supported file descriptors. The API would have made
just as much sense (more sense, IMHO) if an actual file object were
(Continue reading)

Philip Jenvey | 7 Nov 19:59 2007

Re: fileno support is not in jython. Reason?


On Nov 7, 2007, at 2:39 AM, Alan Kennedy wrote:

> [Philip]
>> We can't really make the non-portable argument any longer now that
>> we're supporting faux file descriptors, though. IronPython also
>> supports fake file descriptors (as integers FYI).
>
> Hmm, do you mean "now that we CAN support faux FDs"? Note that we can
> support faux FDs on files *only*, not on any other type of I/O stream,
> e.g. sockets, pipes, etc, so we would be introducing an asymmetry into
> jython by supporting them. If we do support them, I think we will then
> find that users will expect to be able to pass these faux file
> descriptors to e.g. select.select, etc, and expect them to be operated
> upon, e.g. multiplexed. This cannot happen on jython;
> java.nio.FileChannels do NOT implement SelectableChannel; FileChannels
> can never be multiplexed.

Right, I meant 'CAN support' (for PyFile). I also meant that our  
sockets currently support faux FDs -- their fileno() returns a faux  
FD (currently a SocketChannel).

I don't understand -- you're saying only files can support faux FDs  
and sockets can't, when they already do. Granted socket's fileno()  
only works after the socket has been initialized to connect/serve,  
but this still counts IMO (this could even be worked around).

Yes the fact that FileChannels can't be multiplexed will be an issue.  
I would expect select to raise the 'Object is not watchable'  
exception when this is attempted.
(Continue reading)

Alan Kennedy | 8 Nov 12:47 2007

Re: fileno support is not in jython. Reason?

[Philip]
> The pyfile-newio branch doesn't support faux FDs yet, but my
> intention is to support them in the future (I was hoping sooner --
> but probably not, maybe after the branch is merged to trunk). So
> Mehendran's mmap will act on a file object instead of a fileno()
> because that's all that is currently available.
>
> When we do get faux FDs for PyFiles, ideally mmap would work on an FD
> as to be compatible with the CPython API.
>
> Which is the whole point of having faux FDs -- we then have a chance
> of supporting more CPython code that deals with them. E.g. os.fdopen/
> read/write and friends. IronPython supports these.
>
> fd, name = tempfile.mkstemp() is a commonly used one.
>
> I agree that many things in CPython shouldn't operate on file
> descriptors, but that's its current API.

This is the core our issue, I think. We're really talking about the
same thing here; opaque handles on I/O streams.

Every high-level python I/O object has an underlying I/O stream
through which the actual bytes are tx/rx'ed, e.g. files, sockets,
memory-mapped files, etc.

In the cpython world, because it is written in C, there are "File
Descriptors", aka "File Numbers". They are abbreviated to file
numbers, or filenos, because in C, they are integers which index into
a table of C structs which describe the underlying streams. I will use
(Continue reading)

Philip Jenvey | 8 Nov 21:47 2007

Re: fileno support is not in jython. Reason?


On Nov 8, 2007, at 3:47 AM, Alan Kennedy wrote:

>
> Jython also needs a "universal" operation that permits the user to get
> the opaque handle for any I/O object, by implementing a similar
> convention, i.e. all I/O objects should implement a method which
> returns the opaque handle. This would not return an integer index into
> a table, but an actual object; If jython I/O was fully implemented in
> java.nio, then the obvious choice would be to return the objects
> Channel. Note that providing this universal operation does not
> guarantee that the return object is suitable for the desired
> operation, e.g. selecting on files, memory-mapping sockets, etc.
>
> The existing PyFile implementation has no such "universal" operation.
> Only the socket implementation has such an operation, because
> implementing select required it. But even with the existing PyFile
> implementation, we can still return a FileChannel for files, because
> each of the FileInputStream, FileOutputStream, and RandomAccessFile
> classes used by PyFile has a getChannel() method that returns the
> corresponding FileChannel. (Note also that this solves the problem of
> implementing memory-mapped files; simply call the map() method of the
> returned FileChanel, and bingo).

PyFile (and PyFile nio) can also be backed by a Writer, which doesn't  
have a corresponding Channel.

>
> So with a simple addition to the existing PyFile implementation, we
> *can* always return a Channel (we could return a
(Continue reading)

Alan Kennedy | 12 Nov 12:20 2007

Re: fileno support is not in jython. Reason?

[Philip]
> PyFile (and PyFile nio) can also be backed by a Writer, which doesn't
> have a corresponding Channel.

I see.

I thought the character encoding problem was going to raise its head
later than sooner, but I can see now that it has to be addressed up
front.

As you're aware, java.io.Writer's are used for writing character
streams, rather than byte streams. Since a character can be
represented by multiple bytes, depending on character encoding, this
is a non-trivial issue, as you know. Better layering would mean that
PyFile would only be responsible for writing bytes, rather than
characters. But I can see that a constructor for PyFile that takes a
Writer has been included presumably for coding convenience, e.g. for
use with character terminals such as the command line. (It is
interesting to note that PEP 3116 attempts to layer I/O correctly,
with byte oriented Raw IO, Buffered I/O and then text oriented
Character I/O on top of all that, which is the right way, IMHO. But
more on that later).

What I am confused about is that there are no PyFile constructors that
take java.io.Readers? If those constructors are there for the purpose
of wrapping PyFiles around, e.g. character terminals, then there
should be a constructor that takes a Reader, which would supply
buffers full of chars to the interpreter, as the user typed them in?
But my expectation is obviously wrong, there is no such constructor.
So is user input decoded from the sys.defaultencoding?
(Continue reading)

Philip Jenvey | 13 Nov 08:35 2007

Re: fileno support is not in jython. Reason?


On Nov 12, 2007, at 3:20 AM, Alan Kennedy wrote:

> [Philip]
>> PyFile (and PyFile nio) can also be backed by a Writer, which doesn't
>> have a corresponding Channel.
>
> As you're aware, java.io.Writer's are used for writing character
> streams, rather than byte streams. Since a character can be
> represented by multiple bytes, depending on character encoding, this
> is a non-trivial issue, as you know. Better layering would mean that
> PyFile would only be responsible for writing bytes, rather than
> characters. But I can see that a constructor for PyFile that takes a
> Writer has been included presumably for coding convenience, e.g. for
> use with character terminals such as the command line. (It is
> interesting to note that PEP 3116 attempts to layer I/O correctly,
> with byte oriented Raw IO, Buffered I/O and then text oriented
> Character I/O on top of all that, which is the right way, IMHO. But
> more on that later).
>
> What I am confused about is that there are no PyFile constructors that
> take java.io.Readers? If those constructors are there for the purpose
> of wrapping PyFiles around, e.g. character terminals, then there
> should be a constructor that takes a Reader, which would supply
> buffers full of chars to the interpreter, as the user typed them in?
> But my expectation is obviously wrong, there is no such constructor.
> So is user input decoded from the sys.defaultencoding?

Yea, the lack of an accompanying PyFile(Reader) is odd. As I said in  
an earlier email, AFAICT the only reason for PyFile(Writer) is to  
(Continue reading)

Philip Jenvey | 13 Nov 09:20 2007

Re: fileno support is not in jython. Reason?


On Nov 12, 2007, at 11:35 PM, Philip Jenvey wrote:

>
> On Nov 12, 2007, at 3:20 AM, Alan Kennedy wrote:
>>
>>> o was it marked as a tty? for os.isatty
>>
>> I don't see how it is possible to implement this without getting into
>> JNI; it's too platform specific.
>
> What I have for isatty is really a hack that makes it return True for
> sys.stdin/out/err. It's totally fake, and we can drop the idea if
> it's going to be problematic. I can't think of a good example of
> where faking this would be big issue though (anyone correct if I'm
> wrong).
>
> Most of the use cases for isatty() I've seen are for determining
> whether we can use ANSI color codes or if we should/can print
> something (like a progress bar).
>

Well, on second thought we should hold off on isatty for now. I did a  
cursory look for uses of it on google code before, but it really  
warrants a closer look before going in.

http://www.google.com/codesearch?hl=en&lr=&q=isatty%5C%28+file%3A% 
5C.py%24&btnG=Search

--
(Continue reading)

Alan Kennedy | 13 Nov 11:31 2007

Re: fileno support is not in jython. Reason?

Philip,

This is all hugely encouraging; I have a strong feeling that your
implementation is definitely heading along the right track.

[Philip]
> I'm all for killing PythonInterpreter.setStdout/err(Writer) and the
> associated PyFile support if we can all agree on it. My take on this
> is that if you really want a Writer or a Reader for stdout, or any
> file for that matter, write your own file-like wrapper for them.

+1.

Having a PyFile wrapping a Writer is at the wrong level. Character I/O
should be happening at (in PEP 3116 terms) at the Text level.

Mixing up the layers like this is what prevents us from having a clean
PEP 263 implementation, for example. Implementing PEP 263 should be a
simple case of receiving a stream of bytes, from the PyFile
representing stdin, and decoding according to whatever encoding is
specified. The interpreter should *only* receive Unicode strings from
the input. Similarly, the interpreter should *only* ever write Unicode
strings to output, to a Textual output layer. The latter would take
care of the characters->bytes encoding, and the PyFile representing
stdout should only ever see a byte stream, given to it from the
Textual layer above.

> In
> fact now that I think about it PyFile started enforcing conversion to
> PyStrings in write() for 2.2.1 (if you feed it unicode, it does
(Continue reading)

Philip Jenvey | 5 Nov 17:41 2007

Re: fileno support is not in jython. Reason?


On Nov 5, 2007, at 6:07 AM, Mehendran T wrote:

> Hi Alan,
>
> Thanks for that you have given more info about select module.
>
> Actually I am trying to implement mmap module. I found that
> the constructor for mmap uses file descriptor value to get the file  
> handle.
> Then I checked with jython. But it is not supporting that fileno.
> I don't want to change the signature of the constructor.
> So please help me to solve this.
>
> Will it be nice to implement that in PyFile to return
> FileDescriptor java object?
> If we do so, the return value will be a java object and it will  
> violate
> the fileno() and mmap() contract, Isn't it?

I've got a rewrite of PyFile that uses nio Channels (were applicable)  
in my mercurial repository. It's almost finished -- I could throw it  
on a subversion branch if you'd like to work from it (since you're  
going to be dealing with the nio MappedByteBuffer anyway).

fileno isn't implemented yet. I will make it return the underlying  
'RawIO' object, which is pretty much a wrapper for a java nio Channel  
(or some other underlying 'raw' io object).

--
(Continue reading)

Philip Jenvey | 5 Nov 17:13 2007

Re: fileno support is not in jython. Reason?


On Nov 5, 2007, at 4:28 AM, Alan Kennedy wrote:

> [Mehendran]
>> Is there any specific reason behind that that is omitted currently?
>
> 1. It is not omitted
> 2. Jython cannot return a numeric file-descriptor because java has no
> concept of numeric file-descriptors; numeric file descriptors are a
> unix-specific thing (but also a paradigm that has been made available
> in most C implementations on other OSes); hence cpython, because it's
> written in C, supports them on most platforms.
>
> Java instead has its own concept, called Channels, which are used for
> several purposes, with the most obvious being multiplexing.
>
> I could have implemented a file-descriptor table which stored these
> java.nio.channels.*Channels in an array, which would then be indexed
> by an integer. (Which is exactly what C-language file-descriptors are:
> an index into an array of struct FILE*). But my opinion of this
> approach/hack was swayed by the following conversation with Guido on
> python-dev.

I've also been thinking about implementing this approach lately.  
Groves brought up the fact that doing this in a thread safe manner  
might incur a performance hit which could be noticeable when  
allocating/deallocating a large amount of sockets/files in multiple  
threads. It might be negligible, though, I'd like to benchmark this  
in the future.

(Continue reading)

Mehendran T | 6 Nov 08:48 2007
Picon

Re: fileno support is not in jython. Reason?

Hi,

Thanks for your more valuable info and full support. It feels me great. 

Alan:
I am using fileobj in the constructor argument to mmap as you said. 
I think that will be fine and easier and we have strong reason.

Philip:
> fileno isn't implemented yet. I will make it return the underlying  
>'RawIO' object, which is pretty much a wrapper for a java nio Channel  
>(or some other underlying 'raw' io object).
Sorry, I couldn't get you clearly. Do you mean to say that fileno() will 
return the filechannel obj? This also sounds good. 

I got stuck again. I started with fileobj anyway. If need be, I can change 
it to. However, please tell me which would be nice, fileobj or nio-channel 
considering sockets and all?

And, I won't be available for next 2-3 days. I will catch you 
once I will be back. 

-Mehendran.

 
>>> "Alan Kennedy" <jython-dev <at> xhaus.com> 11/05/07 10:03 PM >>> 
[Mehendran]
> Actually I am trying to implement mmap module.

Sorry Mehendran, I see now that I jumped the gun by assuming that you
(Continue reading)


Gmane