headers
Tommi Virtanen | 31 Jul 2012 20:48

Re: Puppet modules for Ceph

On Tue, Jul 24, 2012 at 6:15 AM,  <loic.dachary <at> enovance.com> wrote:
> Note that if puppet client was run on nodeB before it was run on nodeA, all
> three steps would have been run in sequence instead of being spread over two
> puppet client invocations.

Unfortunately, even that is not enough.

The relevant keys (client.admin, client.bootstrap-osd, later
bootstrap-mds radosgw etc also) can only be created once the mons have
reached quorum. This is some time after they have started, even in the
best case. Making the puppet/chef run wait for that sounds like a bad
idea; especially since I use further chef-client runs to feed ceph-mon
information about its peers, which may be necessary for it to ever
reach quorum.

While I can find ways of making the key generation happen as soon as
quorum is reached, communicating the keys to other nodes only happens
at the mercy of the configuration management system; both puppet and
chef seem to be in the mindset of "run every N minutes" option. So
even if we generate the keys best case 2 seconds after ceph-mon
startup, it needs a full configuration manager run on the source node,
and then a run on the destination node, before OSD bring-up etc can
succeed.

I have found no satisfying solution to this, so far.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo <at> vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
(Continue reading)

Permalink | Reply |
headers
Sage Weil | 31 Jul 2012 20:51

Re: Puppet modules for Ceph

On Tue, 31 Jul 2012, Tommi Virtanen wrote:
> On Tue, Jul 24, 2012 at 6:15 AM,  <loic.dachary <at> enovance.com> wrote:
> > Note that if puppet client was run on nodeB before it was run on nodeA, all
> > three steps would have been run in sequence instead of being spread over two
> > puppet client invocations.
> 
> Unfortunately, even that is not enough.
> 
> The relevant keys (client.admin, client.bootstrap-osd, later
> bootstrap-mds radosgw etc also) can only be created once the mons have
> reached quorum. This is some time after they have started, even in the
> best case. Making the puppet/chef run wait for that sounds like a bad
> idea; especially since I use further chef-client runs to feed ceph-mon
> information about its peers, which may be necessary for it to ever
> reach quorum.
> 
> While I can find ways of making the key generation happen as soon as
> quorum is reached, communicating the keys to other nodes only happens
> at the mercy of the configuration management system; both puppet and
> chef seem to be in the mindset of "run every N minutes" option. So
> even if we generate the keys best case 2 seconds after ceph-mon
> startup, it needs a full configuration manager run on the source node,
> and then a run on the destination node, before OSD bring-up etc can
> succeed.
> 
> I have found no satisfying solution to this, so far.

It is also possible to feed initial keys to the monitors during the 'mkfs' 
stage.  If the keys can be agreed on somehow beforehand, then they will 
already be in place when the initial quorum is reached.  Not sure if that
(Continue reading)

Permalink | Reply |
headers
Tommi Virtanen | 31 Jul 2012 21:09

Re: Puppet modules for Ceph

On Tue, Jul 31, 2012 at 11:51 AM, Sage Weil <sage <at> inktank.com> wrote:
> It is also possible to feed initial keys to the monitors during the 'mkfs'
> stage.  If the keys can be agreed on somehow beforehand, then they will
> already be in place when the initial quorum is reached.  Not sure if that
> helps in this situation or not...

Yeah, we're going that way for the "mon." key in the chef cookbooks
(to get the mons talking to each other at all, that *has* to be done
that way), but putting more and more stuff there is not very nice.

Your typical CM framework does not let the recipe run arbitrary code
at that sort of an instantiation time, and pushing this work on the
admin makes it laborous and brittle; what happens when we need a new
type of a bootstrap-foo key? Get all admins to cram an extra entry
into their environment json?
http://ceph.com/docs/master/config-cluster/chef/#configure-your-ceph-environment

That just does not seem like a good way.

Juju seems to provide a real-time notification mechanism between
peers, using it's <name>-relation-changed hook. Other CM frameworks
may need to step up their game, or be subject to the "keep re-running
chef-client until it works" limitation.

If the CM makes it safe to trigger a run manually (e.g. "sudo
chef-client" whenever you feel like it), we can trigger that locally
when we finally create the keys. This still doesn't help the receiving
side to notice any faster.

If the CM makes it safe for us to change node attributes outside of
(Continue reading)

Permalink | Reply |
headers
Jonathan D. Proulx | 7 Aug 2012 15:51
Picon
Picon
Favicon

Re: Puppet modules for Ceph

On Tue, Jul 31, 2012 at 12:09:13PM -0700, Tommi Virtanen wrote:

:Juju seems to provide a real-time notification mechanism between
:peers, using it's <name>-relation-changed hook. Other CM frameworks
:may need to step up their game, or be subject to the "keep re-running
:chef-client until it works" limitation.

Have you lloked at "puppet kick"

puppet-kick(8) -- Remotely control puppet agent
========

SYNOPSIS
--------
Trigger a puppet agent run on a set of hosts.

I'm not using it in my environment and it does require some specific
puppet config on the agent side to make it work so less than ideal,
but might atleast provide a path.

-Jon

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo <at> vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Permalink | Reply |
headers
Tommi Virtanen | 9 Aug 2012 19:04

Re: Puppet modules for Ceph

On Tue, Aug 7, 2012 at 6:51 AM, Jonathan D. Proulx <jon <at> csail.mit.edu> wrote:
> :Juju seems to provide a real-time notification mechanism between
> :peers, using it's <name>-relation-changed hook. Other CM frameworks
> :may need to step up their game, or be subject to the "keep re-running
> :chef-client until it works" limitation.
>
> Have you lloked at "puppet kick"
>
> puppet-kick(8) -- Remotely control puppet agent

I haven't really used puppet, but that doesn't look like it's peer to
peer, the manpage talks about reading the "master's" config file. I
may be wrong in that.

The other nice thing Juju is let you transfer data between the peers
in a relationship -- e.g. we could transport the crypto secret to a
new OSD that way.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo <at> vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Permalink | Reply |

Gmane