Jim Matysek | 9 Apr 16:54 2011

[BlueOnyx:06936] No web service after last night's yum and reboot

Two of my 3 VPS's running on Aventurin{e} had a lot of yum updates 
overnight. This morning neither of the updated ones would let me log in 
to the admin control panel (/login). The message received via the web is 
"Your login session has expired. To login again, please enter your user 
name and password", and an error is logged to /var/log/messages "client 
0:[48:21800]: AUTHKEY to user "" failed". This happens for any user - 
admin or a regular user. The virtual sites were all responding to http 
requests.

I rebooted one of the VPS's and now Apache won't respond at all to 
anything (regular websites are all offline). My /var/log httpd/error_log 
file is filled with the following messages, many per second:

  child pid 3862 exit signal Segmentation fault (11)
NSS_Initialize failed. Certificate database: /etc/httpd/alias.
SSL Library Error: -8038 SEC_ERROR_NOT_INITIALIZED

Any chance of getting what appears to be bad updates fixed so I can yum 
it again and get my server back? Customers are not happy.  Also, I 
forget where on the admin interface I can go to turn off automatic yum 
updates. I absolutely don't want my third VPS to install these updates 
until they are straightened out since that VPS houses my main site, 
which is running fine without these updates.

yum log:

Resolving Dependencies
-->  Running transaction check
--->  Package audit.i386 0:1.7.18-2.el5 set to be updated
--->  Package audit-libs.i386 0:1.7.18-2.el5 set to be updated
(Continue reading)

Rodrigo Ordoñez Licona | 9 Apr 18:25 2011
Picon

[BlueOnyx:06937] Re: No web service after last night's yum and reboot

NAILED IT,

Somehow the certificate database got corrupted on two of our blueonyx. Maybe
updates

...Fixed it by moving these files from a healthy blueonyx to the failed
ones.

...Exact same behavior apache would not serve pages and segmentation faults.

Files from /etc/httpd/alias

cert8.db
key3.db
secmod.db
install.log

HTH

Rodrigo O
Xnet

P.S. Used ftp to move them to the other server to a temp directory. And then
to /etc/httpd/alias on the failed one.

-----Mensaje original-----
De: blueonyx-bounces@...
[mailto:blueonyx-bounces@...] En
nombre de Jim Matysek
Enviado el: Sábado, 09 de Abril de 2011 08:54 a.m.
(Continue reading)

Rodrigo Ordoñez Licona | 9 Apr 19:06 2011
Picon

[BlueOnyx:06938] Re: No web service after last night's yum and reboot

Sorry my last email was not very good structured, 

Here are more detailed steps as we found a third server with the same
symptoms:

a)First ssh to the damaged server
b)Change to user root (su root)
c)issue a :
	/etc/init.d/cced.init restart
d)Point your web browser to the admin page
	http://yourip:444 (you might want to try your domain/admin too)
e)Finish the setup wizard (don’t forget to setup your password) (in some
cases you might just login regularly and everything would be alright if not
continue to step f)
f)SSH to a healthy server 
g)Change to user root (su root)
h)change to the certs/keys directory 
	cd /etc/https/alias
i)ftp to the damaged server (I recommend using the admin user - check
directories below)
	ftp ip_of_damaged_server
j)Create a directory on the damaged server to store the files
	mkdir alias
	cd alias	

k)copy healthy files to the damaged server
	send cert8.db
	send install.log
	send key3.db
	send secmod.db
(Continue reading)

Rodrigo Ordoñez Licona | 9 Apr 19:27 2011
Picon

[BlueOnyx:06939] Re: No web service after last night's yum and reboot

More inform about this,

The damage seems to be serious only on VPS,

The damage on KVM or Regular BO boxes, seems to repair itself
By issuing the /etc/init.d/cced.init restart
(maybe self signed certificate gets regenerated)

Regards

Rodrigo O
Xnet

p.s. Sorry for the typos and poor grammar, just wanted to get this info out
asap. Hope it helps others.

-----Mensaje original-----
De: blueonyx-bounces@...
[mailto:blueonyx-bounces@...] En
nombre de Rodrigo Ordoñez Licona
Enviado el: Sábado, 09 de Abril de 2011 11:06 a.m.
Para: 'BlueOnyx General Mailing List'
Asunto: [BlueOnyx:06938] Re: No web service after last night's yum and
reboot

Sorry my last email was not very good structured, 

Here are more detailed steps as we found a third server with the same
symptoms:

(Continue reading)

Michael Aronoff | 9 Apr 19:48 2011
Picon

[BlueOnyx:06940] Re: No web service after last night's yum and reboot

Rodrigo wrote:
> The damage on KVM or Regular BO boxes, seems to repair itself By issuing
the /etc/init.d/cced.init restart (maybe > self signed certificate gets
regenerated)

Well I did not find that to be the case. I have 4 dedicated servers, not
VPS. I had the problem on one BO box (my other 3 had not done yum yet)  and
even after restarting cced I had to copy the files over. However once I did
that I was able to restart httpd without error and then I restarted the box
and everything was good again.

So if any others are having this problem follow Rodrigo's great instructions
and you should be all good.

As for others I would disable auto YUM updates until this is fixed.

M Aronoff Out

Picon

[BlueOnyx:06943] KVM or Regular BO boxes,


----- Original Message ----- 
From: "Rodrigo Ordoñez Licona" <rodrigo@...>

Rodrigo

You mentioned "KVM or Regular BO boxes".

I am more used to OpenVZ / Aventurine.
How do you set up a BX/BO server on a KVM?

----
Ken M
Precision Web Hosting, Inc.
http://www.precisionweb.net

Rodrigo Ordoñez Licona | 9 Apr 21:35 2011
Picon

[BlueOnyx:06944] Re: KVM or Regular BO boxes,

You just load the iso on the virtualization software that you use,

And create a kvm machine with the iso image loaded on the cd rom, 

Then a regular install as if it were a physical server. The rest is the same

Hth

Rodrigo O
xnet

-----Mensaje original-----
De: blueonyx-bounces@...
[mailto:blueonyx-bounces@...] En
nombre de Ken - Precision Web Hosting, Inc
Enviado el: Sábado, 09 de Abril de 2011 01:20 p.m.
Para: BlueOnyx General Mailing List
Asunto: [BlueOnyx:06943] KVM or Regular BO boxes,

----- Original Message ----- 
From: "Rodrigo Ordoñez Licona" <rodrigo@...>

Rodrigo

You mentioned "KVM or Regular BO boxes".

I am more used to OpenVZ / Aventurine.
How do you set up a BX/BO server on a KVM?

----
(Continue reading)

Picon

[BlueOnyx:06946] Re: KVM or Regular BO boxes,


> -----Mensaje original-----
> De: blueonyx-bounces@...
[mailto:blueonyx-bounces@...] En
> nombre de Ken - Precision Web Hosting, Inc
> Enviado el: Sábado, 09 de Abril de 2011 01:20 p.m.
> Para: BlueOnyx General Mailing List
> Asunto: [BlueOnyx:06943] KVM or Regular BO boxes,
>
>
> ----- Original Message ----- 
> From: "Rodrigo Ordoñez Licona" <rodrigo@...>
>
> Rodrigo
>
> You mentioned "KVM or Regular BO boxes".
>
> I am more used to OpenVZ / Aventurine.
> How do you set up a BX/BO server on a KVM?
>
> ----
> Ken M
> Precision Web Hosting, Inc.
> http://www.precisionweb.net
>
>

----- Original Message ----- 
From: "Rodrigo Ordoñez Licona" <rodrigo@...>
To: "'BlueOnyx General Mailing List'" <blueonyx@...>
(Continue reading)

rodrigo | 10 Apr 00:39 2011
Picon

[BlueOnyx:06950] Re: KVM or Regular BO boxes,

Proxmox. Pve.proxmox.com

servicio email movil Blackberry
by xnet.mx

-----Original Message-----
From: "Ken  - Precision Web Hosting, Inc" <kenlists@...>
Sender: blueonyx-bounces@...
Date: Sat, 9 Apr 2011 12:59:15 
To: BlueOnyx General Mailing List<blueonyx@...>
Reply-To: BlueOnyx General Mailing List <blueonyx@...>
Subject: [BlueOnyx:06946] Re: KVM or Regular BO boxes,

> -----Mensaje original-----
> De: blueonyx-bounces@...
[mailto:blueonyx-bounces@...] En
> nombre de Ken - Precision Web Hosting, Inc
> Enviado el: Sábado, 09 de Abril de 2011 01:20 p.m.
> Para: BlueOnyx General Mailing List
> Asunto: [BlueOnyx:06943] KVM or Regular BO boxes,
>
>
> ----- Original Message ----- 
> From: "Rodrigo Ordoñez Licona" <rodrigo@...>
>
> Rodrigo
>
> You mentioned "KVM or Regular BO boxes".
>
> I am more used to OpenVZ / Aventurine.
(Continue reading)

Jim Matysek | 9 Apr 21:13 2011

[BlueOnyx:06942] Re: No web service after last night's yum and reboot

On 4/9/2011 1:06 PM, Rodrigo Ordoñez Licona wrote:
> Sorry my last email was not very good structured,
>
> Here are more detailed steps as we found a third server with the same
> symptoms:

Thanks for the details. Unfortunately, it's not working for me. I did 
notice that all 3 VPS's had files in the /etc/httpd/alias directory from 
the same date - April 3.  I'm thinking that the certificate database may 
be bad on all 3 of the VPS's and my "working" VPS has the same problem 
lurking, but it just hasn't shown up yet because it didn't get the 
latest yum updates. I don't have any other source of files to try.

-jim

Rodrigo Ordoñez Licona | 9 Apr 21:40 2011
Picon

[BlueOnyx:06945] Re: No web service after last night's yum and reboot

Hi jim , 

If you have a blank-pc available at office or home, download the regular iso
and install it there.

it 'll create a fresh BO so you can copy the files over.

I can post generic files, but they are ssl certificates that should be
private (specially the key file).

So it can lead to a security problem if many users have the same files
sitting around.

Regenerating the self signed certificate from the GUI might renew the files.

Post again if you need further help I can email you generic files. So you
can get it running 

HTH

Rodrigo O
Xnet

-----Mensaje original-----
De: blueonyx-bounces@...
[mailto:blueonyx-bounces@...] En
nombre de Jim Matysek
Enviado el: Sábado, 09 de Abril de 2011 01:14 p.m.
Para: BlueOnyx General Mailing List
Asunto: [BlueOnyx:06942] Re: No web service after last night's yum and
(Continue reading)

[BlueOnyx:06947] Re: No web service after last night's yum and reboot

Hi Everybody,
Wow... quite the cluster-bomb, eh?   We just had another customer 
experience the same issue on their 3 Aventurin{e}-based servers.

I believe you may find this a faster / easier fix that does not involve 
taking the contents of /etc/httpd/alias from another server:

Perform the following as root:

==============================
/etc/init.d/httpd stop

cd /etc/httpd/

mkdir alias_bork

mv alias/*  alias_bork/

cp/etc/pki/nssdb/* alias/

cp alias_bork/* alias/

/etc/init.d/httpd start

==============================

Interestingly, I checked one of our stand-alone BX boxes (as opposed to 
a virtualized system) and found there is no /etc/httpd/alias directory. 
  <shrug>.

(Continue reading)

Jim Matysek | 9 Apr 22:44 2011

[BlueOnyx:06948] Re: No web service after last night's yum and reboot

On 4/9/2011 4:04 PM, Chris Gebhardt - VIRTBIZ Internet wrote
> Perform the following as root:
>
> ==============================
> /etc/init.d/httpd stop
>
> cd /etc/httpd/
>
> mkdir alias_bork
>
> mv alias/*  alias_bork/
>
> cp/etc/pki/nssdb/* alias/
>
> cp alias_bork/* alias/
>
> /etc/init.d/httpd start
>
> ==============================
>

I assume that second-to-last line was a mistake because it just copies 
the bad files back. I ran everything except that line and now httpd 
won't start - generates an error "[error] Certificate not found: 
'Server-Cert'".  For completeness, I ran all commands including the one 
that probably shouldn't be there and yes, the same old problem was back.

--

-- 
Jim Matysek, Webmaster
U.S. Masters Swimming
(Continue reading)

[BlueOnyx:06949] Re: No web service after last night's yum and reboot

Hi Jim,
Jim Matysek wrote:
> On 4/9/2011 4:04 PM, Chris Gebhardt - VIRTBIZ Internet wrote
>> Perform the following as root:
>>
>> ==============================
>> /etc/init.d/httpd stop
>>
>> cd /etc/httpd/
>>
>> mkdir alias_bork
>>
>> mv alias/*  alias_bork/
>>
>> cp/etc/pki/nssdb/* alias/
>>
>> cp alias_bork/* alias/
>>
>> /etc/init.d/httpd start
>>
>> ==============================
>>
> 
> I assume that second-to-last line was a mistake because it just copies 
> the bad files back. I ran everything except that line and now httpd 
> won't start - generates an error "[error] Certificate not found: 
> 'Server-Cert'".  For completeness, I ran all commands including the one 
> that probably shouldn't be there and yes, the same old problem was back.

Nope, not a mistake.   Try the whole thing and see how it works out. 
(Continue reading)

Jim Matysek | 10 Apr 03:04 2011

[BlueOnyx:06953] Re: No web service after last night's yum and reboot

On 4/9/2011 4:54 PM, Chris Gebhardt - VIRTBIZ Internet wrote:
>
>> I assume that second-to-last line was a mistake because it just copies
>> the bad files back. I ran everything except that line and now httpd
>> won't start - generates an error "[error] Certificate not found:
>> 'Server-Cert'".  For completeness, I ran all commands including the one
>> that probably shouldn't be there and yes, the same old problem was back.
> Nope, not a mistake.   Try the whole thing and see how it works out.
> Have a bit of faith.  ;)

I finally got them working with your instructions. I had already tried 
them once without success, although I have to admit that on my first try 
I copied things out of /etc/httpd/alias rather than moving.  Later I 
tried them again exactly as written and it didn't work for me. Then I 
tried it again and it worked. I'm not asking any questions - just glad 
to get things back online.

Thanks to you and Rodrigo for your help. I also tried using Rodrigo's 
files without success earlier.

-jim

David Booth | 10 Apr 03:43 2011
Picon

[BlueOnyx:06954] Re: No web service after last night's yum and reboot

On Sun, 10 Apr 2011 11:04:10 +1000, Jim Matysek <matysekj@...> wrote:

> On 4/9/2011 4:54 PM, Chris Gebhardt - VIRTBIZ Internet wrote:
>>
<snip>
>> Have a bit of faith.  ;)
>
<snip>
> Then I
> tried it again and it worked. I'm not asking any questions - just glad
> to get things back online.
>
> -jim

Nasty.

I had the prob on a BO stand alone - all good after reboot without having  
changed anything.

I've turned off yum updates on 2 VPS under aventurine and will wait for  
news of when it is ok to turn it back on.

Thanks for the headsup
--

-- 
___________
David Booth
Picon

[BlueOnyx:06951] Re: No web service after last night's yum and reboot


----- Original Message ----- 
From: "Chris Gebhardt - VIRTBIZ Internet" <cobaltfacts@...>
To: "BlueOnyx General Mailing List" <blueonyx@...>
Sent: Saturday, April 09, 2011 1:04 PM
Subject: [BlueOnyx:06947] Re: No web service after last night's yum and 
reboot

> Hi Everybody,
> Wow... quite the cluster-bomb, eh?   We just had another customer
> experience the same issue on their 3 Aventurin{e}-based servers.
>
> I believe you may find this a faster / easier fix that does not involve
> taking the contents of /etc/httpd/alias from another server:
>
> Perform the following as root:
>
> ==============================
> /etc/init.d/httpd stop
>
> cd /etc/httpd/
>
> mkdir alias_bork
>
> mv alias/*  alias_bork/
>
> cp/etc/pki/nssdb/* alias/
>
> cp alias_bork/* alias/
>
(Continue reading)

[BlueOnyx:06952] Re: No web service after last night's yum and reboot

Ken - Precision Web Hosting, Inc wrote:
>> Hope this proves helpful if you've been bitten by this bug.
>>
>> -- 
>> Chris Gebhardt
>>
> 
> For some reason, this method did not work for me.
> 
> But Rodrigo's method did work.

<hangs head in shame>
Sorry it didn't work out for you.  I pulled it from the history on the 
boxes that I tried it on successfully.   Maybe if there's a next time 
I'll have a chance to try it out again and see what I did wrong.

But then, hopefully, I won't get that opportunity.

--

-- 
Chris Gebhardt
VIRTBIZ Internet Services
Access, Web Hosting, Colocation, Dedicated
www.virtbiz.com | toll-free (866) 4 VIRTBIZ
Picon

[BlueOnyx:06955] Re: No web service after last night's yum and reboot


----- Original Message ----- 
From: "Chris Gebhardt - VIRTBIZ Internet" <cobaltfacts@...>
To: "BlueOnyx General Mailing List" <blueonyx@...>
Sent: Saturday, April 09, 2011 5:04 PM
Subject: [BlueOnyx:06952] Re: No web service after last night's yum and 
reboot

> Ken - Precision Web Hosting, Inc wrote:
>>> Hope this proves helpful if you've been bitten by this bug.
>>>
>>> -- 
>>> Chris Gebhardt
>>>
>>
>> For some reason, this method did not work for me.
>>
>> But Rodrigo's method did work.
>
> <hangs head in shame>
> Sorry it didn't work out for you.  I pulled it from the history on the
> boxes that I tried it on successfully.   Maybe if there's a next time
> I'll have a chance to try it out again and see what I did wrong.
>
> But then, hopefully, I won't get that opportunity.
>
> -- 
> Chris Gebhardt

Maybe my copying and pasting was defective. (Going from coffee to green tea 
(Continue reading)


Gmane