headers
Garth Goodson | 1 Dec 2005 22:33
Picon

Re: pNFS -layouts and byte ranges

Todd Pisek wrote:
> I have a couple of questions:
> 
> 1 - The pNFS draft refers to layouts and layout segments.
>     If I understand correctly, a layout segment is a byte
>     range within a file layout. Am I correct in assuming
>     that LAYOUTGET specifies a layout range?
> 
Correct.  LAYOUTGET specifies a byte range.

> 2 - Are layout segments recalled for range splitting?
> 
Not sure what you mean exactly by "range splitting", but recalls can be 
done for file ranges.  These file ranges need not match the ranges 
returned by LAYOUTGET.  They can either span or sub-divide layout 
segments held by clients.  Note: layout segment is just terminology for 
a byte range within a layout.  A client may hold a layout that is 
comprised of multiple sets of byte-ranges gotten through separate 
LAYOUTGETs.

> 3 - Does a layout segment carry an implicit byte range
>     delegation (as opposed to a byte range lock)?
> 
NO.  Delegations and locks are separate from layouts (at least to the 
extent possible as determined by the layout type).  E.g., for file 
layouts they are totally separate.  Also note, that byte range 
delegations don't currently exist.  Trond has proposed them, but they 
are not part of pNFS.

> 4 - How is byte range locking enforced by object/block
(Continue reading)

Permalink | Reply |
headers
Todd Pisek | 2 Dec 2005 00:35
Picon

Re: pNFS -layouts and byte ranges

Garth Goodson wrote On 12/01/05 03:33 PM,:
> Todd Pisek wrote:
> 
:
:
> 
>>4 - How is byte range locking enforced by object/block
>>    data servers?
>>
> 
> Check with the corresponding specs.  Objects will use capabilities to 
> enforce mandatory locks.  Blocks can tie byte-ranged locks to layouts. 
> Note: only mandatory locks need be enforced at the data server.
> 

When you refer to "mandatory locks", is this synonymous with LOCK?
I thought the only operations between a client and the data servers
was READ, WRITE, COMMIT, and NULL. Is there an implied operation between
the metadata server and the data servers for the LOCK operation?

For OSD data servers, how are the capabilities for the data servers
conveyed to the client?

--- Todd

_______________________________________________
nfsv4 mailing list
nfsv4 <at> ietf.org
https://www1.ietf.org/mailman/listinfo/nfsv4
(Continue reading)

Permalink | Reply |
headers
Garth Goodson | 2 Dec 2005 00:41
Picon

Re: pNFS -layouts and byte ranges


Todd Pisek wrote:
> Garth Goodson wrote On 12/01/05 03:33 PM,:
> 
>>Todd Pisek wrote:
>>
> 
> :
> :
> 
>>>4 - How is byte range locking enforced by object/block
>>>   data servers?
>>>
>>
>>Check with the corresponding specs.  Objects will use capabilities to 
>>enforce mandatory locks.  Blocks can tie byte-ranged locks to layouts. 
>>Note: only mandatory locks need be enforced at the data server.
>>
> 
> 
> When you refer to "mandatory locks", is this synonymous with LOCK?
> I thought the only operations between a client and the data servers
> was READ, WRITE, COMMIT, and NULL. Is there an implied operation between
> the metadata server and the data servers for the LOCK operation?
> 
All metadata ops go to the metadata server.  However, mandatory locks 
must be respected on the data server even though there is no client call 
to that data server.  Thus, there is a need for a back-end control 
protocol between metadata and data servers.  Data servers do not accept 
LOCK operations from the client.
(Continue reading)

Permalink | Reply |
headers
Todd Pisek | 3 Dec 2005 17:25
Picon

Re: pNFS -layouts and byte ranges


Garth Goodson wrote On 12/01/05 05:41 PM,:
> 
> Todd Pisek wrote:
> 
:
:
>>
>>When you refer to "mandatory locks", is this synonymous with LOCK?
>>I thought the only operations between a client and the data servers
>>was READ, WRITE, COMMIT, and NULL. Is there an implied operation between
>>the metadata server and the data servers for the LOCK operation?
>>
> 
> All metadata ops go to the metadata server.  However, mandatory locks 
> must be respected on the data server even though there is no client call 
> to that data server.  Thus, there is a need for a back-end control 
> protocol between metadata and data servers.  Data servers do not accept 
> LOCK operations from the client.
> 
> However, for some non-intelligent devices (e.g., blocks) a control 
> protocol is out of the question, thus locks are tied more closely with 
> layouts.  Object devices use capabilities to control access to data, 
> thus these capabilities can be tied in with locks as well.
> 

OK, I think I now understand this. If a client wants a mandatory lock, it
needs to issue LOCK, then issue LAYOUTGET (probably as a compound). Correct?

What's the current view regarding client side caching with many writers
(Continue reading)

Permalink | Reply |
headers
Garth Goodson | 5 Dec 2005 18:46
Picon

Re: pNFS -layouts and byte ranges

Todd Pisek wrote:
> 
> OK, I think I now understand this. If a client wants a mandatory lock, it
> needs to issue LOCK, then issue LAYOUTGET (probably as a compound). Correct?
> 
The goal was for pNFS layout ops to be separate from delegation and lock 
ops.  Thus, at least for file-based layouts, there is no correspondence 
between LOCK and LAYOUTGET.  It doesn't matter the order you do them in. 
  Whenever the LOCK completes you have the LOCK, whenever the LAYOUTGET 
completes you have the layout.  The client can pick how it makes sense 
to order them.

However, for other layouttypes, a LOCK may result in the recall of other 
layouts.

> What's the current view regarding client side caching with many writers
> (think parallel applications)?
>
pNFS does not add any client side caching features to NFSv4 (this was 
debated during pNFS meetings prior to it moving into the WG).  pNFS uses 
delegations as NFSv4 would use them.  The only difference is that dirty 
data can be flushed using the layout, or through the metadata server 
(and ideally would be flushed prior to returning a layout).

Trond has a separate byte-range delegation proposal that would allow 
byte-ranged caching.  Again, these are not (and will not) be tied to 
layouts (although could be used together).

-Garth
(Continue reading)

Permalink | Reply |
headers
Todd Pisek | 5 Dec 2005 19:32
Picon

Re: pNFS -layouts and byte ranges


Garth Goodson wrote On 12/05/05 11:46 AM,:
> Todd Pisek wrote:
> 
>>OK, I think I now understand this. If a client wants a mandatory lock, it
>>needs to issue LOCK, then issue LAYOUTGET (probably as a compound). Correct?
>>
> 
> The goal was for pNFS layout ops to be separate from delegation and lock 
> ops.  Thus, at least for file-based layouts, there is no correspondence 
> between LOCK and LAYOUTGET.  It doesn't matter the order you do them in. 
>   Whenever the LOCK completes you have the LOCK, whenever the LAYOUTGET 
> completes you have the layout.  The client can pick how it makes sense 
> to order them.
> 

I believe the only way an OSD can enforce byte range access is through
a capability. Version 1 of the OSD spec does not specify how this is done.
My understanding is that this ability is in OSD version 2, which is not
yet finalized.

It appears that the only mechanism by which OSD capabilities can be conveyed
to the client is via the LAYOUTGET response.

I don't understand how mandatory locking can be accomplished without using
LOCK in combination with LAYOUTGET in the OSD case.

I do understand that OSD pNFS can obtain an segment that is enforced by the
OSD without using LOCK. But if I remember correctly, the result of LAYOUTGET
does not imply a range delegation or a range lock.
(Continue reading)

Permalink | Reply |
headers
Brent Welch | 5 Dec 2005 20:19
X-Face
Favicon

Re: pNFS -layouts and byte ranges

OSDv1 does not have byte-range granularity of its capabilities,
but we can still use capabilities to enforce mandatory byte range
locks.  However, it obviously limits the file to one writer at
a time; you cannot use OSDv1 capabilities to enforce mandatory locks
where two different clients are simultaneously writing to two
different regions of the file.  The metadata server will have to
drive this via CB_LAYOUTRECALL.

I would just note this limitation and move on.  I don't expect a lot
of mandatory lock usage, and I am content to wait for OSDv2 and
its (proposed) byte range capabilities to optimize this scenario.

I think it is much more likely that clients will use advisory byte
range locks, and that they will self-enforce those locks without
any help from the data servers, just like they do today.  The metadata
server will coordinate the lock requests, but the clients do
the enforcement.
>>>Todd Pisek said:
 > 
 > 
 > 
 > Garth Goodson wrote On 12/05/05 11:46 AM,:
 > > Todd Pisek wrote:
 > > 
 > >>OK, I think I now understand this. If a client wants a mandatory lock, 
it
 > >>needs to issue LOCK, then issue LAYOUTGET (probably as a compound). 
Correc
     t?
 > >>
(Continue reading)

Permalink | Reply |
headers
Todd Pisek | 6 Dec 2005 16:04
Picon

Re: pNFS -layouts and byte ranges

I'm not so concerned about V1 versus V2 of OSD as I am about
how LAYOUTGET is being used. Let's assume V2 OSD. If a client
requests a segment using only LAYOUTGET, and the metadata
server grants the full range requested, will the data server(s)
reject client reads/writes outside this segment for both object
and file pNFS?

Brent Welch wrote On 12/05/05 01:19 PM,:
> OSDv1 does not have byte-range granularity of its capabilities,
> but we can still use capabilities to enforce mandatory byte range
> locks.  However, it obviously limits the file to one writer at
> a time; you cannot use OSDv1 capabilities to enforce mandatory locks
> where two different clients are simultaneously writing to two
> different regions of the file.  The metadata server will have to
> drive this via CB_LAYOUTRECALL.
> 
> I would just note this limitation and move on.  I don't expect a lot
> of mandatory lock usage, and I am content to wait for OSDv2 and
> its (proposed) byte range capabilities to optimize this scenario.
> 
> I think it is much more likely that clients will use advisory byte
> range locks, and that they will self-enforce those locks without
> any help from the data servers, just like they do today.  The metadata
> server will coordinate the lock requests, but the clients do
> the enforcement.
> 
>>>>Todd Pisek said:
> 
>  > 
>  >
(Continue reading)

Permalink | Reply |
headers
Brent Welch | 2 Dec 2005 02:39
X-Face
Favicon

Re: pNFS -layouts and byte ranges

The capability will be an opaque sent from the metadata server.
Ideally it will be encyrpted using the GSS context that the
client has established with the server, but I'm not sure the
best way to spec that yet.

>>>Garth Goodson said:
 > 
 > 
 > 
 > Todd Pisek wrote:
 > > Garth Goodson wrote On 12/01/05 03:33 PM,:
 > > 
 > >>Todd Pisek wrote:
 > >>
 > > 
 > > :
 > > :
 > > 
 > >>>4 - How is byte range locking enforced by object/block
 > >>>   data servers?
 > >>>
 > >>
 > >>Check with the corresponding specs.  Objects will use capabilities to 
 > >>enforce mandatory locks.  Blocks can tie byte-ranged locks to layouts. 
 > >>Note: only mandatory locks need be enforced at the data server.
 > >>
 > > 
 > > 
 > > When you refer to "mandatory locks", is this synonymous with LOCK?
 > > I thought the only operations between a client and the data servers
(Continue reading)

Permalink | Reply |

Gmane