headers
Haynes, Tom | 6 Jun 2012 21:02
Picon

Meeting Agenda for 06/07/12

   Time: Thursdays 12:30-1:30 PM Central/1:30-2:30 PM Eastern/10:30-11:30 AM Pacific
Dial-in: 1-866-844-9418 (Toll Free)
       1-517-345-9208 (Caller paid)

     Id: 4082151519

Proposed Agenda
-------------------------

+ IETF Note Well Agreement

This is a reminder that our discussions are governed by the
IETF Note Well Agreement. See:

   http://www.ietf.org/NOTEWELL.html

We will start each week's meeting with this announcement.

+ We will be working from draft version 11

http://www.ietf.org/id/draft-ietf-nfsv4-minorversion2-11.txt
http://www.ietf.org/id/draft-ietf-nfsv4-minorversion2-dot-x-11.txt

+ Get a lined copy

1) Use cat:

$cat -n draft-ietf-nfsv4-minorversion2-11.txt > listing-11.txt

2) Peter suggested that you can use IE to see the numbers.
(Continue reading)

Permalink | Reply |
headers
Dean | 6 Jun 2012 22:23
Picon

NFSv4.2 Draft 11 IOADVISE Comments

Couple comments on IOADVISE in Draft 11

1)
   4148       If NFL42_UFLG_IO_ADVISE_THRU_MDS is set, then if the client
   4149       implements IO_ADVISE, then if it wants the DS to honor 
IO_ADVISE, the
   4150       client MUST send the operation to the MDS, and the server will
   4151       communicate the advice back each DS.
This is a little convoluted.  How about something like the following,

If NFL42_UFLG_IO_ADVISE_THRU_MDS is set, the client MUST send the 
IO_ADVISE operation to the MDS in order for it to be honored by the DS.  
Once the MDS receives the IO_ADVISE operation, it will communicate the 
advice to each DS.

2)
   4154       If NFL42_UFLG_IO_ADVISE_THRU_MDS is not set, then this 
indicates to
   4155       client that if wants to inform the server via IO_ADVISE of the
   4156       client's intended use of the file, then the client SHOULD 
send an
   4157       IO_ADVISE to each DS....
This has a small mistake in it (missing the word 'it').  But maybe it 
can be re-written thus,

If NFL42_UFLG_IO_ADVISE_THRU_MDS is not set, then the client SHOULD send 
an IO_ADVISE operation to the appropriate DS for the specified byte range.

3)
In section 13.8.6, I'm curious about the use of IO_ADVISE4_NOREUSE.
(Continue reading)

Permalink | Reply |
headers
Sebastian Moors | 13 Jun 2012 16:22
Picon
Gravatar

Re: NFSv4.2 Draft 11 IOADVISE Comments

Hi Dean,

Dean wrote:
> 3)
> In section 13.8.6, I'm curious about the use of IO_ADVISE4_NOREUSE.  
> Currently it is listed as an operation that applies to a specified 
> byte range and is treated as a one time instruction.  I don't think 
> this is correct.  For example, if we wish to use IO_ADVISE4_NOREUSE in 
> the same compound as OPEN (if the file is opened with O_DIRECT), then 
> it cannot be a 'one time' operation since the server will not remember 
> that all forthcoming writes should not be cached.
Just a remark: I don't find it suiting to implement O_DIRECT by sending 
IO_ADVISE4_NOREUSE to the server. There is a small difference that 
occurs when doing a read operation: A server that sees an operation with 
O_DIRECT should make shure that the read operation bypasses the cache 
completely.  But the definition of POSIX_FADV_NOREUSE is:  "Specifies 
that the application expects to access the specified data once and then 
not reuse it thereafter.". Therefore, IO_ADVISE4_NOREUSE allows that 
read requests are getting satisfied from the cache (which stands in 
contrast to O_DIRECT). Imho, that's a good reason to handle both advices 
differently.
Why not introducing sth. like IO_ADVISE4_DIRECTIO to IO_ADVISE_type4 to 
avoid these conflicts?
>
> A simple solution may be to move IO_ADVISE4_NOREUSE to the list of 
> operations that apply to the entire file.  This way, the server can 
> set a single flag when it receives IO_ADVISE4_NOREUSE that all 
> forthcoming writes should not be cached.
This would take the possibility to issue the advice on byte-ranges, 
which is especially interesting for HPC applications which work on large
(Continue reading)

Permalink | Reply |
headers
Myklebust, Trond | 13 Jun 2012 17:37
Picon

Re: NFSv4.2 Draft 11 IOADVISE Comments

On Wed, 2012-06-13 at 16:22 +0200, Sebastian Moors wrote:
> Hi Dean,
> 
> Dean wrote:
> > 3)
> > In section 13.8.6, I'm curious about the use of IO_ADVISE4_NOREUSE.  
> > Currently it is listed as an operation that applies to a specified 
> > byte range and is treated as a one time instruction.  I don't think 
> > this is correct.  For example, if we wish to use IO_ADVISE4_NOREUSE in 
> > the same compound as OPEN (if the file is opened with O_DIRECT), then 
> > it cannot be a 'one time' operation since the server will not remember 
> > that all forthcoming writes should not be cached.
> Just a remark: I don't find it suiting to implement O_DIRECT by sending 
> IO_ADVISE4_NOREUSE to the server. There is a small difference that 
> occurs when doing a read operation: A server that sees an operation with 
> O_DIRECT should make shure that the read operation bypasses the cache 
> completely.

That wouldn't fit the concept of a "hint". That would be forcing the
server to manage the data caching in a certain way.

>   But the definition of POSIX_FADV_NOREUSE is:  "Specifies 
> that the application expects to access the specified data once and then 
> not reuse it thereafter.". Therefore, IO_ADVISE4_NOREUSE allows that 
> read requests are getting satisfied from the cache (which stands in 
> contrast to O_DIRECT). Imho, that's a good reason to handle both advices 
> differently.
> Why not introducing sth. like IO_ADVISE4_DIRECTIO to IO_ADVISE_type4 to 
> avoid these conflicts?
(Continue reading)

Permalink | Reply |
headers
Dean | 13 Jun 2012 18:49
Picon

Re: NFSv4.2 Draft 11 IOADVISE Comments

Hi Sebastian,

>>> 3)
>>> In section 13.8.6, I'm curious about the use of IO_ADVISE4_NOREUSE.
>>> Currently it is listed as an operation that applies to a specified
>>> byte range and is treated as a one time instruction.  I don't think
>>> this is correct.  For example, if we wish to use IO_ADVISE4_NOREUSE in
>>> the same compound as OPEN (if the file is opened with O_DIRECT), then
>>> it cannot be a 'one time' operation since the server will not remember
>>> that all forthcoming writes should not be cached.
>> Just a remark: I don't find it suiting to implement O_DIRECT by sending
>> IO_ADVISE4_NOREUSE to the server. There is a small difference that
>> occurs when doing a read operation: A server that sees an operation with
>> O_DIRECT should make shure that the read operation bypasses the cache
>> completely.
> That wouldn't fit the concept of a "hint". That would be forcing the
> server to manage the data caching in a certain way.

Exactly.  Actually implementing O_DIRECT is not the goal, but rather to 
give the server a hint about the data access pattern.  The client cannot 
'rely' on the results of IO_ADVISE...

>
>>> A simple solution may be to move IO_ADVISE4_NOREUSE to the list of
>>> operations that apply to the entire file.  This way, the server can
>>> set a single flag when it receives IO_ADVISE4_NOREUSE that all
>>> forthcoming writes should not be cached.
>> This would take the possibility to issue the advice on byte-ranges,
>> which is especially interesting for HPC applications which work on large
>> files.
(Continue reading)

Permalink | Reply |
headers
Sebastian Moors | 15 Jun 2012 00:50
Picon
Gravatar

Re: NFSv4.2 Draft 11 IOADVISE Comments

Dean wrote:
> Hi Sebastian,
>>>
>> That wouldn't fit the concept of a "hint". That would be forcing the
>> server to manage the data caching in a certain way.
>
> Exactly.  Actually implementing O_DIRECT is not the goal, but rather 
> to give the server a hint about the data access pattern.  The client 
> cannot 'rely' on the results of IO_ADVISE...
>
Thanks for the comments! Sorry, my comment was not precise... What i 
wanted to suggest was that a server might choose different optimizations 
for direct i/o and NOREUSE. I understand that an implementation of 
direct i/o via hints makes no sense.

>>> which is especially interesting for HPC applications which work on 
>>> large
>>> files.
>>> If an application programmer uses this hint on the last half of a file,
>>> it would disable (write) caching for the complete file, which might be
>>> not his intention.
>> This would take the possibility to issue the advice on byte-ranges,
>
> The problem is that its unusable today for a byte-range since it is a 
> one time operation.  For example, if the client wants to tell the 
> server it doesn't need the byte-range any longer, it should use 
> DONTNEED.  If the client wants to tell the server it won't (in the 
> future) need the byte-range, using NOREUSE will have no affect because 
> the server will have immediately forgotten the information. 
I'm a little bit curious about the categorisation in 'one-time
(Continue reading)

Permalink | Reply |
headers
Dean | 15 Jun 2012 19:25
Picon

Re: NFSv4.2 Draft 11 IOADVISE Comments

Hi Sebastian,

>>>>
>>> That wouldn't fit the concept of a "hint". That would be forcing the
>>> server to manage the data caching in a certain way.
>>
>> Exactly.  Actually implementing O_DIRECT is not the goal, but rather 
>> to give the server a hint about the data access pattern.  The client 
>> cannot 'rely' on the results of IO_ADVISE...
>>
> Thanks for the comments! Sorry, my comment was not precise... What i 
> wanted to suggest was that a server might choose different 
> optimizations for direct i/o and NOREUSE. I understand that an 
> implementation of direct i/o via hints makes no sense.

I'd be fine with that, although as a hint, I'm not exactly sure what 
advantage it has.  Isn't it better to simply tell the server of your 
access pattern and let it decide on the best course of action?

>
>>>> which is especially interesting for HPC applications which work on 
>>>> large
>>>> files.
>>>> If an application programmer uses this hint on the last half of a 
>>>> file,
>>>> it would disable (write) caching for the complete file, which might be
>>>> not his intention.
>>> This would take the possibility to issue the advice on byte-ranges,
>>
>> The problem is that its unusable today for a byte-range since it is a
(Continue reading)

Permalink | Reply |
headers
Myklebust, Trond | 15 Jun 2012 22:59
Picon

Re: NFSv4.2 Draft 11 IOADVISE Comments

On Fri, 2012-06-15 at 10:25 -0700, Dean wrote:

> As I mentioned before, having the NFS server store state that contains 
> byte-ranges and their associated hints would be too onerous.  So we 
> decided that the amount of state that the server would be required to 
> store is limited to an entire file (single byte range) and its 
> associated hints.

Is it really all that onerous to have the server remember it for the
duration of the enclosing COMPOUND?

Cheers
  Trond
--

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust <at> netapp.com
www.netapp.com

_______________________________________________
nfsv4 mailing list
nfsv4 <at> ietf.org
https://www.ietf.org/mailman/listinfo/nfsv4
Permalink | Reply |
headers
Dean | 16 Jun 2012 00:01
Picon

Re: NFSv4.2 Draft 11 IOADVISE Comments


On 6/15/12 1:59 PM, Myklebust, Trond wrote:
> On Fri, 2012-06-15 at 10:25 -0700, Dean wrote:
>
>> As I mentioned before, having the NFS server store state that contains
>> byte-ranges and their associated hints would be too onerous.  So we
>> decided that the amount of state that the server would be required to
>> store is limited to an entire file (single byte range) and its
>> associated hints.
> Is it really all that onerous to have the server remember it for the
> duration of the enclosing COMPOUND?
I'm ok with it if it will work.  Are you suggesting something like a 
PUT_IO_ADVISE?
Dean

>
> Cheers
>    Trond
_______________________________________________
nfsv4 mailing list
nfsv4 <at> ietf.org
https://www.ietf.org/mailman/listinfo/nfsv4
Permalink | Reply |
headers
Dean | 27 Jun 2012 19:37
Picon

Re: NFSv4.2 Draft 11 IOADVISE Comments

On 6/15/12 1:59 PM, Myklebust, Trond wrote:
> On Fri, 2012-06-15 at 10:25 -0700, Dean wrote:
>
>> As I mentioned before, having the NFS server store state that contains
>> byte-ranges and their associated hints would be too onerous.  So we
>> decided that the amount of state that the server would be required to
>> store is limited to an entire file (single byte range) and its
>> associated hints.
> Is it really all that onerous to have the server remember it for the
> duration of the enclosing COMPOUND?

I've been thinking about IO_ADVISE4_NOREUSE a little more.

One way to resolve the O_DIRECT discussion (and completely remove it 
from the draft), may be to go through an example of how an application 
would instead use posix_fadvise POSIX_FADV_NOREUSE to issue 
IO_ADVISE4_NOREUSE to the server.  This would allow applications to 
combine the use of O_DIRECT and posix_fadvise POSIX_FADV_NOREUSE to a) 
eliminate caching on the client and b) inform the server that this data 
is only being accessed once (and thus allowing the server to optimize 
this, which is the whole point of all of this).

When an application issues posix_fadvise POSIX_FADV_NOREUSE for a 
byte-range, if the NFS client is going to include it in the enclosing 
COMPOUND for every READ/READ_PLUS request, then the application must be 
able to specify the hint for every read issued to the server (including 
reads driven by readahead).  I don't see this as possible, since the 
application knows only the ranges for which it accesses, not the ranges 
read by the NFS client.
(Continue reading)

Permalink | Reply |
headers
Haynes, Tom | 27 Jun 2012 21:10
Picon

Re: NFSv4.2 Draft 11 IOADVISE Comments


On 6/27/12 12:37 PM, "Dean" <seattleplus <at> gmail.com> wrote:

>On 6/15/12 1:59 PM, Myklebust, Trond wrote:
>> On Fri, 2012-06-15 at 10:25 -0700, Dean wrote:
>>
>>> As I mentioned before, having the NFS server store state that contains
>>> byte-ranges and their associated hints would be too onerous.  So we
>>> decided that the amount of state that the server would be required to
>>> store is limited to an entire file (single byte range) and its
>>> associated hints.
>> Is it really all that onerous to have the server remember it for the
>> duration of the enclosing COMPOUND?
>
>I've been thinking about IO_ADVISE4_NOREUSE a little more.
>
>One way to resolve the O_DIRECT discussion (and completely remove it
>from the draft), may be to go through an example of how an application
>would instead use posix_fadvise POSIX_FADV_NOREUSE to issue
>IO_ADVISE4_NOREUSE to the server.  This would allow applications to
>combine the use of O_DIRECT and posix_fadvise POSIX_FADV_NOREUSE to a)
>eliminate caching on the client and b) inform the server that this data
>is only being accessed once (and thus allowing the server to optimize
>this, which is the whole point of all of this).

Well, the example would still keep it in, no?

I don't see having requirements driven by O_DIRECT as making sense for
this document.
(Continue reading)

Permalink | Reply |
headers
Chuck Lever | 27 Jun 2012 21:41
Picon
Favicon

Re: NFSv4.2 Draft 11 IOADVISE Comments


On Jun 27, 2012, at 3:10 PM, Haynes, Tom wrote:

> 
> 
> On 6/27/12 12:37 PM, "Dean" <seattleplus <at> gmail.com> wrote:
> 
>> On 6/15/12 1:59 PM, Myklebust, Trond wrote:
>>> On Fri, 2012-06-15 at 10:25 -0700, Dean wrote:
>>> 
>>>> As I mentioned before, having the NFS server store state that contains
>>>> byte-ranges and their associated hints would be too onerous.  So we
>>>> decided that the amount of state that the server would be required to
>>>> store is limited to an entire file (single byte range) and its
>>>> associated hints.
>>> Is it really all that onerous to have the server remember it for the
>>> duration of the enclosing COMPOUND?
>> 
>> I've been thinking about IO_ADVISE4_NOREUSE a little more.
>> 
>> One way to resolve the O_DIRECT discussion (and completely remove it
>> from the draft), may be to go through an example of how an application
>> would instead use posix_fadvise POSIX_FADV_NOREUSE to issue
>> IO_ADVISE4_NOREUSE to the server.  This would allow applications to
>> combine the use of O_DIRECT and posix_fadvise POSIX_FADV_NOREUSE to a)
>> eliminate caching on the client and b) inform the server that this data
>> is only being accessed once (and thus allowing the server to optimize
>> this, which is the whole point of all of this).

I agree, an application should be entirely in control of whether a NOREUSE hint is issued when O_DIRECT is
(Continue reading)

Permalink | Reply |
headers
Dean | 28 Jun 2012 00:42
Picon

Re: NFSv4.2 Draft 11 IOADVISE Comments


> I agree, an application should be entirely in control of whether a NOREUSE hint is issued when O_DIRECT is
specified.  Having an NFS client automatically send a hint when O_DIRECT is requested could be
problematic in some cases.
>
> For example, four NFS clients running Oracle RAC will specify O_DIRECT to disable their local caches, but
will benefit from having a server-side data cache by keeping active database pages in a memory working set
shared by all the clients.
>
> Thus O_DIRECT should stand by itself, and applications are free to augment it any way they like.  But the
client shouldn't automatically do this for them.
>
>> Well, the example would still keep it in, no?
>>
>> I don't see having requirements driven by O_DIRECT as making sense for
>> this document.
>>
>> I do see a subsection of 13.8.4 that describes an example as above as
>> enhancing the document. What I'd also like to see is a discussion of both
>> what happens if an application uses O_DIRECT and does not use
>> posix_fadvise POSIX_FADV_NOREUSE and what happens if the server decides
>> not to use the hint.
> I prefer Dean's example of how both the client and server treat NOREUSE.  If one or the other decides to
ignore that hint, you have basically the same discussion, but without bringing up O_DIRECT.
>
In the end, I want to make sure that O_DIRECT can be used in conjunction 
with posix_fadvise/POSIX_FADV_NOREUSE so that applications can disable 
caching in both the client and server.  So while we can remove the 
discussion of O_DIRECT, I want to ensure the spec has a discussion of 
how applications can disable server caching for forth coming read and
(Continue reading)

Permalink | Reply |
headers
Haynes, Tom | 28 Jun 2012 02:19
Picon

Re: NFSv4.2 Draft 11 IOADVISE Comments


On 6/27/12 5:42 PM, "Dean" <seattleplus <at> gmail.com> wrote:

>
>> I agree, an application should be entirely in control of whether a
>>NOREUSE hint is issued when O_DIRECT is specified.  Having an NFS client
>>automatically send a hint when O_DIRECT is requested could be
>>problematic in some cases.
>>
>> For example, four NFS clients running Oracle RAC will specify O_DIRECT
>>to disable their local caches, but will benefit from having a
>>server-side data cache by keeping active database pages in a memory
>>working set shared by all the clients.
>>
>> Thus O_DIRECT should stand by itself, and applications are free to
>>augment it any way they like.  But the client shouldn't automatically do
>>this for them.
>>
>>> Well, the example would still keep it in, no?
>>>
>>> I don't see having requirements driven by O_DIRECT as making sense for
>>> this document.
>>>
>>> I do see a subsection of 13.8.4 that describes an example as above as
>>> enhancing the document. What I'd also like to see is a discussion of
>>>both
>>> what happens if an application uses O_DIRECT and does not use
>>> posix_fadvise POSIX_FADV_NOREUSE and what happens if the server decides
>>> not to use the hint.
>> I prefer Dean's example of how both the client and server treat
(Continue reading)

Permalink | Reply |
headers
Myklebust, Trond | 28 Jun 2012 02:40
Picon

Re: NFSv4.2 Draft 11 IOADVISE Comments

On Thu, 2012-06-28 at 00:19 +0000, Haynes, Tom wrote:
> 
> On 6/27/12 5:42 PM, "Dean" <seattleplus <at> gmail.com> wrote:
> 
> >
> >> I agree, an application should be entirely in control of whether a
> >>NOREUSE hint is issued when O_DIRECT is specified.  Having an NFS client
> >>automatically send a hint when O_DIRECT is requested could be
> >>problematic in some cases.
> >>
> >> For example, four NFS clients running Oracle RAC will specify O_DIRECT
> >>to disable their local caches, but will benefit from having a
> >>server-side data cache by keeping active database pages in a memory
> >>working set shared by all the clients.
> >>
> >> Thus O_DIRECT should stand by itself, and applications are free to
> >>augment it any way they like.  But the client shouldn't automatically do
> >>this for them.
> >>
> >>> Well, the example would still keep it in, no?
> >>>
> >>> I don't see having requirements driven by O_DIRECT as making sense for
> >>> this document.
> >>>
> >>> I do see a subsection of 13.8.4 that describes an example as above as
> >>> enhancing the document. What I'd also like to see is a discussion of
> >>>both
> >>> what happens if an application uses O_DIRECT and does not use
> >>> posix_fadvise POSIX_FADV_NOREUSE and what happens if the server decides
> >>> not to use the hint.
(Continue reading)

Permalink | Reply |
headers
J. Bruce Fields | 13 Jun 2012 19:44

Re: NFSv4.2 Draft 11 IOADVISE Comments

On Wed, Jun 06, 2012 at 01:23:26PM -0700, Dean wrote:
> Couple comments on IOADVISE in Draft 11
> 
> 
> 1)
>   4148       If NFL42_UFLG_IO_ADVISE_THRU_MDS is set, then if the client
>   4149       implements IO_ADVISE, then if it wants the DS to honor
> IO_ADVISE, the
>   4150       client MUST send the operation to the MDS, and the server will
>   4151       communicate the advice back each DS.
> This is a little convoluted.  How about something like the following,
> 
> If NFL42_UFLG_IO_ADVISE_THRU_MDS is set, the client MUST send the
> IO_ADVISE operation to the MDS in order for it to be honored by the
> DS.  Once the MDS receives the IO_ADVISE operation, it will
> communicate the advice to each DS.
> 
> 2)
>   4154       If NFL42_UFLG_IO_ADVISE_THRU_MDS is not set, then this
> indicates to
>   4155       client that if wants to inform the server via IO_ADVISE of the
>   4156       client's intended use of the file, then the client
> SHOULD send an
>   4157       IO_ADVISE to each DS....
> This has a small mistake in it (missing the word 'it').  But maybe
> it can be re-written thus,
> 
> If NFL42_UFLG_IO_ADVISE_THRU_MDS is not set, then the client SHOULD
> send an IO_ADVISE operation to the appropriate DS for the specified
> byte range.
(Continue reading)

Permalink | Reply |

Gmane