headers
John Jasen | 27 Apr 22:25

Puppet: a few questions and issues


As a brief summary, $work had a puppet module call the underlying OS
package manager. The package manager (apt) apparently failed, reporting
a non-zero exit status to puppet, which seems to have aborted the rest
of the puppet run. The systems were left in a half-provisioned state,
which escaped our monitoring system for a few days.

In looking at the matter, we were not quickly able to come up with a way
to detect failed puppet runs. Mining syslog may be an option, but see
the point about that further in. Is there an acceptable external way
(say through nagios) to determine if the last puppet run succeeded or
failed?

Second, it seems that puppetd -t and puppetd -t --noop both log the same
to syslog and also update the state.yaml file. Is there a way to
differentiate --noop or test runs from syslog? Is there a way to make it
not update the state file? If not, should there be? :)

Thanks in advance.

--

-- 
-- John E. Jasen (jjasen@...)
-- "Deserve Victory." -- Terry Goodkind, Naked Empire
Permalink | Reply |
headers
Justin Clayton | 27 Apr 22:31

Re: Puppet: a few questions and issues

I would recommend taking a look at the report functions of puppet as a start. There are many different
supported reporting methods such as email, puppet's dashboard app, or even additional local logging.

http://docs.puppetlabs.com/guides/reporting.html
http://docs.puppetlabs.com/references/latest/report.html

Hope this helps you in your situation.

Justin Clayton
justin@...

Not sent from my iPhone

On Apr 27, 2011, at 1:25 PM, John Jasen wrote:

> 
> As a brief summary, $work had a puppet module call the underlying OS
> package manager. The package manager (apt) apparently failed, reporting
> a non-zero exit status to puppet, which seems to have aborted the rest
> of the puppet run. The systems were left in a half-provisioned state,
> which escaped our monitoring system for a few days.
> 
> In looking at the matter, we were not quickly able to come up with a way
> to detect failed puppet runs. Mining syslog may be an option, but see
> the point about that further in. Is there an acceptable external way
> (say through nagios) to determine if the last puppet run succeeded or
> failed?
> 
> Second, it seems that puppetd -t and puppetd -t --noop both log the same
> to syslog and also update the state.yaml file. Is there a way to
(Continue reading)

Permalink | Reply |
headers
Narayan Desai | 27 Apr 22:36
Favicon

Re: Puppet: a few questions and issues

This isn't a direct solution to your problem, but we've addressed this sort of problem in bcfg2 by including
a full feedback loop in the configuration process, so that you can inspect what has gone on across your
clients from a single, centralized location. (or detect outright tool failure by the lack of feedback
data at predictable intervals)

I'd suggest that you want something like that, preferably able to tell you about underlying failures in
some detail, as opposed to a high level thumbs up/thumbs down.  
 -nld

On Apr 27, 2011, at 3:25 PM, John Jasen wrote:

> 
> As a brief summary, $work had a puppet module call the underlying OS
> package manager. The package manager (apt) apparently failed, reporting
> a non-zero exit status to puppet, which seems to have aborted the rest
> of the puppet run. The systems were left in a half-provisioned state,
> which escaped our monitoring system for a few days.
> 
> In looking at the matter, we were not quickly able to come up with a way
> to detect failed puppet runs. Mining syslog may be an option, but see
> the point about that further in. Is there an acceptable external way
> (say through nagios) to determine if the last puppet run succeeded or
> failed?
> 
> Second, it seems that puppetd -t and puppetd -t --noop both log the same
> to syslog and also update the state.yaml file. Is there a way to
> differentiate --noop or test runs from syslog? Is there a way to make it
> not update the state file? If not, should there be? :)
> 
> Thanks in advance.
(Continue reading)

Permalink | Reply |
headers
John Jasen | 27 Apr 22:45

Re: Puppet: a few questions and issues

On 04/27/2011 04:36 PM, Narayan Desai wrote:
> This isn't a direct solution to your problem, but we've addressed this sort of problem in bcfg2 by
including a full feedback loop in the configuration process, so that you can inspect what has gone on
across your clients from a single, centralized location. (or detect outright tool failure by the lack of
feedback data at predictable intervals)
> 
> I'd suggest that you want something like that, preferably able to tell you about underlying failures in
some detail, as opposed to a high level thumbs up/thumbs down.  

Indeed. One of my peers suggested writing a similar capability for
puppet, where the changes were committed to (syslog |& a state file)
with status, so it could be queried as needed, and to the level desired.

--

-- 
-- John E. Jasen (jjasen@...)
-- "Deserve Victory." -- Terry Goodkind, Naked Empire
Permalink | Reply |
headers
Luke Kanies | 29 Apr 14:40
Gravatar

Re: Puppet: a few questions and issues

On Apr 27, 2011, at 10:26 PM, John Jasen <jjasen@...> wrote:

>
> As a brief summary, $work had a puppet module call the underlying OS
> package manager. The package manager (apt) apparently failed, reporting
> a non-zero exit status to puppet, which seems to have aborted the rest
> of the puppet run. The systems were left in a half-provisioned state,
> which escaped our monitoring system for a few days.
>
> In looking at the matter, we were not quickly able to come up with a way
> to detect failed puppet runs. Mining syslog may be an option, but see
> the point about that further in. Is there an acceptable external way
> (say through nagios) to determine if the last puppet run succeeded or
> failed?
>
> Second, it seems that puppetd -t and puppetd -t --noop both log the same
> to syslog and also update the state.yaml file. Is there a way to
> differentiate --noop or test runs from syslog? Is there a way to make it
> not update the state file? If not, should there be? :)

As Justin says, Puppet's reports are the best fit for this, and if you
send your reports to the Dashboard, pretty much everything you're
asking for comes for free.

--

-- 
http://puppetlabs.com/ | +1-615-594-8199 | @puppetmasterd
Permalink | Reply |

Gmane