Tomasz Chmielewski | 14 Sep 22:29 2010

strange owners issue? ownership reset by the other gluster node

I have a /shared/www/example.com directory, being set up on a mirrored 
glusterfs filesystem (two servers).

I have a /shared/www/example.com/images/ directory there, owned by 
root:root.

Now, I want the webserver to be able to write to this directory: simple, 
just make the webserver the owner:

server1# chown www-data:www-data images

server1# ls -ld images
drwxr-xr-x 2 www-data www-data 4096 2010-09-14 22:19 images

Let's see what server2 thinks about it:

server2# ls -ld images
drwxr-xr-x 2 root root 4096 2010-09-14 22:19 images

Oops, it's back to root!

Now, what does server1 show?

server1# ls -ld images
drwxr-xr-x 2 root root 4096 2010-09-14 22:19 images

As we can see, it's root again - ownership was reset, although I didn't 
use "chown root:root". As a result, the webserver can't upload files 
(or, any other process which would need a similar operation), although I 
initially did "chown www-data:www-data images" on one of the servers.
(Continue reading)

Douglas Stanley | 14 Sep 22:43 2010
Picon

Re: strange owners issue? ownership reset by the other gluster node

Did you do the chown operation to the mounted gluster filesystem, or
to the exported by gluster filesystem on one of your storage bricks?

What I mean is, is /shared/www what is exported in your
glusterfsd.vol, or did you do mount -t glusterfs volfile /shared/www ?

Doug

On Tue, Sep 14, 2010 at 4:29 PM, Tomasz Chmielewski <mangoo@...> wrote:
> I have a /shared/www/example.com directory, being set up on a mirrored
> glusterfs filesystem (two servers).
>
> I have a /shared/www/example.com/images/ directory there, owned by
> root:root.
>
>
> Now, I want the webserver to be able to write to this directory: simple,
> just make the webserver the owner:
>
> server1# chown www-data:www-data images
>
> server1# ls -ld images
> drwxr-xr-x 2 www-data www-data 4096 2010-09-14 22:19 images
>
>
> Let's see what server2 thinks about it:
>
> server2# ls -ld images
> drwxr-xr-x 2 root root 4096 2010-09-14 22:19 images
>
(Continue reading)

Tomasz Chmielewski | 14 Sep 22:50 2010

Re: strange owners issue? ownership reset by the other gluster node

On 14.09.2010 22:43, Douglas Stanley wrote:
> Did you do the chown operation to the mounted gluster filesystem, or
> to the exported by gluster filesystem on one of your storage bricks?
> 
> What I mean is, is /shared/www what is exported in your
> glusterfsd.vol, or did you do mount -t glusterfs volfile /shared/www ?

I made the chown operation on the glusterfs filesystem mounted on /shared:

/etc/glusterfs/glusterfs.vol on /shared type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

cat /etc/fstab:
(...)
/etc/glusterfs/glusterfs.vol  /shared  glusterfs  defaults  0  0

Also, permissions are reset when doing similar operations.

What's interesting, with exactly the same config files on Debian Lenny, it works as expected.

The difference from Ubuntu 10.04 are:
- kernel (2.6.26 on Debian, 2.6.32 on Ubuntu)
- glusterfs (3.0.4 on Debian, 3.0.5 on Ubuntu)
- fuse (2.7.4 on Debian, 2.8.1 on Ubuntu)
- loads of other userspace...

--

-- 
Tomasz Chmielewski
http://wpkg.org
Douglas Stanley | 14 Sep 22:56 2010
Picon

Re: strange owners issue? ownership reset by the other gluster node

On Tue, Sep 14, 2010 at 4:50 PM, Tomasz Chmielewski <mangoo@...> wrote:
> On 14.09.2010 22:43, Douglas Stanley wrote:
>> Did you do the chown operation to the mounted gluster filesystem, or
>> to the exported by gluster filesystem on one of your storage bricks?
>>
>> What I mean is, is /shared/www what is exported in your
>> glusterfsd.vol, or did you do mount -t glusterfs volfile /shared/www ?
>
> I made the chown operation on the glusterfs filesystem mounted on /shared:
>
> /etc/glusterfs/glusterfs.vol on /shared type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
>
>
> cat /etc/fstab:
> (...)
> /etc/glusterfs/glusterfs.vol  /shared  glusterfs  defaults  0  0
>
>
>
> Also, permissions are reset when doing similar operations.
>
>
>
> What's interesting, with exactly the same config files on Debian Lenny, it works as expected.
>
> The difference from Ubuntu 10.04 are:
> - kernel (2.6.26 on Debian, 2.6.32 on Ubuntu)
> - glusterfs (3.0.4 on Debian, 3.0.5 on Ubuntu)
> - fuse (2.7.4 on Debian, 2.8.1 on Ubuntu)
> - loads of other userspace...
(Continue reading)

Tomasz Chmielewski | 14 Sep 22:59 2010

Re: strange owners issue? ownership reset by the other gluster node

On 14.09.2010 22:56, Douglas Stanley wrote:

> Odd, on my 10.04 box, the version is only 3.0.2. Where'd you get a
> 3.0.5 version? I've seen that in debian testing, but not in ubuntu
> 10.04 yet.

http://ftp.gluster.com/pub/gluster/glusterfs/3.0/LATEST/Ubuntu/

--

-- 
Tomasz Chmielewski
http://wpkg.org
Vikas Gorur | 15 Sep 00:44 2010

Re: strange owners issue? ownership reset by the other gluster node


On Sep 14, 2010, at 1:59 PM, Tomasz Chmielewski wrote:

> On 14.09.2010 22:56, Douglas Stanley wrote:
> 
>> Odd, on my 10.04 box, the version is only 3.0.2. Where'd you get a
>> 3.0.5 version? I've seen that in debian testing, but not in ubuntu
>> 10.04 yet.
> 
> http://ftp.gluster.com/pub/gluster/glusterfs/3.0/LATEST/Ubuntu/

Thomas,

It looks like replicate's self-heal reverted the directory's permissions. This is because the default
value of the option that controls whether Gluster keeps track of changes to ownership/permissions is
off. This is a bug, and is being fixed in the next release. For now, please use this option in your client
volume file so that Gluster keeps track of attributes properly.

option metadata-change-log on
option metadata-lock-server-count 1

------------------------------
Vikas Gorur
Engineer - Gluster, Inc.
------------------------------
Tomasz Chmielewski | 15 Sep 08:29 2010

Re: strange owners issue? ownership reset by the other gluster node

On 15.09.2010 00:44, Vikas Gorur wrote:
>
> On Sep 14, 2010, at 1:59 PM, Tomasz Chmielewski wrote:
>
>> On 14.09.2010 22:56, Douglas Stanley wrote:
>>
>>> Odd, on my 10.04 box, the version is only 3.0.2. Where'd you get
>>> a 3.0.5 version? I've seen that in debian testing, but not in
>>> ubuntu 10.04 yet.
>>
>> http://ftp.gluster.com/pub/gluster/glusterfs/3.0/LATEST/Ubuntu/
>
> Thomas,
>
> It looks like replicate's self-heal reverted the directory's
> permissions. This is because the default value of the option that
> controls whether Gluster keeps track of changes to
> ownership/permissions is off. This is a bug, and is being fixed in
> the next release. For now, please use this option in your client
> volume file so that Gluster keeps track of attributes properly.
>
> option metadata-change-log on option metadata-lock-server-count 1

I unmounted gluster on both systems, then mounted it again - and it
looks like the issue is gone for now.

It would be interesting to know what triggers it though.

--

-- 
Tomasz Chmielewski
(Continue reading)

Aaron Porter | 20 Sep 22:53 2010
Picon

Re: strange owners issue? ownership reset by the other gluster node

On Tue, Sep 14, 2010 at 3:44 PM, Vikas Gorur <vikas@...> wrote:
> It looks like replicate's self-heal reverted the directory's permissions. This is because the default
value of the option that controls whether Gluster keeps track of changes to ownership/permissions is
off. This is a bug, and is being fixed in the next release. For now, please use this option in your client
volume file so that Gluster keeps track of attributes properly.
>
> option metadata-change-log on
> option metadata-lock-server-count 1

We're seeing this too, is there a bug ID for this?
Aaron Porter | 20 Sep 23:30 2010
Picon

Re: strange owners issue? ownership reset by the other gluster node

On Tue, Sep 14, 2010 at 3:44 PM, Vikas Gorur <vikas@...> wrote:
> option metadata-change-log on
> option metadata-lock-server-count 1

What are the performance implications of enabling these?
Tomasz Chmielewski | 14 Sep 22:58 2010

Re: strange owners issue? ownership reset by the other gluster node

On 14.09.2010 22:50, Tomasz Chmielewski wrote:
> On 14.09.2010 22:43, Douglas Stanley wrote:
>> Did you do the chown operation to the mounted gluster filesystem, or
>> to the exported by gluster filesystem on one of your storage bricks?
>>
>> What I mean is, is /shared/www what is exported in your
>> glusterfsd.vol, or did you do mount -t glusterfs volfile /shared/www ?
>
> I made the chown operation on the glusterfs filesystem mounted on /shared:
>
> /etc/glusterfs/glusterfs.vol on /shared type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
>
>
> cat /etc/fstab:
> (...)
> /etc/glusterfs/glusterfs.vol  /shared  glusterfs  defaults  0  0
>
>
>
> Also, permissions are reset when doing similar operations.

Really strange:

server1# cd /shared
server1# chmod 777 ldirectord.cf

server2# cd /shared
server2# ls -l
(...)
-rwxrwxrwx 1 root root   1941 2010-09-14 15:12 ldirectord.cf
(Continue reading)


Gmane