Darren Bolding | 3 Jul 13:19 2009

Fire, Power loss at Fisher Plaza in Seattle

Fisher Plaza, a self-styled carrier hotel in Seattle, and home to multiple
datacenter and colocation providers, has had a major issue in one of its
buildings late last night, early this morning.
The best information I am aware of is that there was a failure in the
main/generator transfer switch which resulted in a fire.  The sprinkler
system activated.  From speaking to the fire battalion chief, I am under the
impression that Seattle Fire did use water on the fire as well, but I am
unsure of this.

Given the failure location, generator power was not available, and cooling
failed.  UPS power to systems continued, and I can personally vouch that
they held out for well over an hour.  When we were able to access our
equipment, ambient air temps were well over 100 degrees in the room our
equipment is located in.

At least some, if not many circuits were affected.  Several large
co-location providers and other datacenters are located in the facility,
these facilities have no power.

As this was the main/generator switch, and it is now highly damaged, the
circuits in the area are damaged, and the entire area is doused in water, a
rapid restoration of power does not seem likely.  Fisher Plaza's phone
numbers now result in fast-busy signals, so I have no recent update from
them directly.

Interestingly, this building is also the production studios for several
Seattle TV and radio stations.

There is no ETA for resolution.

(Continue reading)

Joe Richards | 3 Jul 13:40 2009
Picon

Re: Fire, Power loss at Fisher Plaza in Seattle

Multiple folks on Twitter who are in the area are reporting a 5-6 hour
ETA.

-Joe

--

-- 
Joe Richards <joe <at> disconformity.net>
--
ipv4: http://www.disconformity.net             [ 72.29.169.48/28 ]
ipv6: http://ipv6.disconformity.net      [ 2001:48c0:1001:1::/64 ]
blog: http://www.mainlined.org

Seth Mattinen | 3 Jul 19:00 2009
Picon

Re: Fire, Power loss at Fisher Plaza in Seattle

Darren Bolding wrote:
>
> Interestingly, this building is also the production studios for several
> Seattle TV and radio stations.
> 
> There is no ETA for resolution.
> 

Apparently it took authorize.net with it, too:
http://twitter.com/authorizenet

~Seth

David Hubbard | 3 Jul 19:05 2009

RE: Fire, Power loss at Fisher Plaza in Seattle

From: Seth Mattinen [mailto:sethm <at> rollernet.us] 
> 
> Apparently it took authorize.net with it, too:
> http://twitter.com/authorizenet
> 
> ~Seth

No technical explanation of course but it also took down
their 'backup facility' according to them on twitter;
I assume some bad routing/DNS if they do actually have
a backup facility.  Lots of online stores are
offline right now because of this, and the holiday is
unfortunately keeping those store owners from knowing
they are not making sales right now.  Life in ecommerce...

David

Tomas L. Byrnes | 3 Jul 19:20 2009
Picon

RE: Fire, Power loss at Fisher Plaza in Seattle

This begs the question of what basic parameters should be for a "carrier
hotel" or co-lo.

Given that we're getting designated "Critical Infrastructure", we'd
getter start coming up with some, or we'll have them defined for us.

The old NEBS standards were too much of a straightjacket, but the
current situation, where any buffoon who wants to can claim to be
something they aren't (redundant and reliable) undermines the business
of those who actually spend the money, and make the effort, to provide a
true "carrier grade" co-lo.

This is life in the current Internet: Overpromise, and Underdeliver.

>-----Original Message-----
>From: David Hubbard [mailto:dhubbard <at> dino.hostasaurus.com]
>Sent: Friday, July 03, 2009 10:05 AM
>To: NANOG list
>Subject: RE: Fire, Power loss at Fisher Plaza in Seattle
>
>From: Seth Mattinen [mailto:sethm <at> rollernet.us]
>>
>> Apparently it took authorize.net with it, too:
>> http://twitter.com/authorizenet
>>
>> ~Seth
>
>No technical explanation of course but it also took down
>their 'backup facility' according to them on twitter;
>I assume some bad routing/DNS if they do actually have
(Continue reading)

Seth Mattinen | 3 Jul 19:40 2009
Picon

Re: Fire, Power loss at Fisher Plaza in Seattle

Tomas L. Byrnes wrote:
> This begs the question of what basic parameters should be for a "carrier
> hotel" or co-lo.
> 
> Given that we're getting designated "Critical Infrastructure", we'd
> getter start coming up with some, or we'll have them defined for us.
> 
> The old NEBS standards were too much of a straightjacket, but the
> current situation, where any buffoon who wants to can claim to be
> something they aren't (redundant and reliable) undermines the business
> of those who actually spend the money, and make the effort, to provide a
> true "carrier grade" co-lo.

Absolutely. Then your pricing is so far out of whack with the apparent
competition that it's hard to get customers when it appears one can get
the same/better for far less. Me, personally, I just don't say things
like "100% uptime" or claim to be a carrier-grade facility. But I think
that scares people off when my competitors (and I've seen the insides of
some of the horrid trash heaps they call a NOC) claim they do.

>
> This is life in the current Internet: Overpromise, and Underdeliver.
> 

"Our flywheel systems are so failure-proof and thinking outside the box
that we don't need a silly battery UPS that can cold-start!"

I know outages and related discussion end up attracting the off-topic
hammer here on NANOG, but I do find them interesting and worthwhile.

(Continue reading)

William Herrin | 3 Jul 20:49 2009

Re: Fire, Power loss at Fisher Plaza in Seattle

On Fri, Jul 3, 2009 at 1:20 PM, Tomas L. Byrnes<tomb <at> byrneit.net> wrote:
> This begs the question of what basic parameters should be for a
> "carrier hotel" or co-lo. [...] The old NEBS standards were too much
> of a straightjacket.

Tomas,

There is a useful standard: ANSI/TIA-942. It offers specifications for
four tiers of data centers ranging from tier 1 (a basic data center
with no redundancy) to tier 4 (fully fault tolerant).

http://www.tiaonline.org/standards/catalog/search.cfm?standards_criteria=TIA-942
(the 2005 one)

Judging from http://www.techlinks.net/community/articles/article/1-article-submission-forms/14833-a-quick-primer-on-data-center-tier-classifications
there's even research that projects what sort of annual downtime you
can expect for each of the tiers described by the standard.

When I walk into a data center, I make a habit of asking which tier
they achieve, at least for the HVAC and electrical systems. And then I
ask to see the components which the tier claim says they should have.

Regards,
Bill Herrin

--

-- 
William D. Herrin ................ herrin <at> dirtside.com  bill <at> herrin.us
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

(Continue reading)

Sean Donelan | 3 Jul 21:22 2009

Re: Fire, Power loss at Fisher Plaza in Seattle

On Fri, 3 Jul 2009, William Herrin wrote:
> There is a useful standard: ANSI/TIA-942. It offers specifications for
> four tiers of data centers ranging from tier 1 (a basic data center
> with no redundancy) to tier 4 (fully fault tolerant).

Are you better off with a single "tier 4" data center, multiple
"tier 1" data centers, or something in between?

Distance and quantity versus complexity and scaling versus cost and risk. 
Sometimes no matter what you choose, you might be wrong.

Earth is a single point of failure, where is your backup site?

Stephen Stuart | 3 Jul 22:10 2009

Re: Fire, Power loss at Fisher Plaza in Seattle

> Earth is a single point of failure, where is your backup site?

This reminds me of the 1996 thread about how MAE-East still had no
generator. Same topic, roughly, some of the same people (hi, Sean).

Sure, the line about the Earth SPOF is catchy, but in terms of more
likely scenarios: how many people stand *outside* the "tier 4"
datacenter and imagine a fire marshal pointing at the building and
saying, "Turn *that* off, now." I've seen that happen a couple times
since the WilTel POP thing in 1996.

Stephen

Tomas L. Byrnes | 3 Jul 22:21 2009
Picon

RE: Fire, Power loss at Fisher Plaza in Seattle


>
>Earth is a single point of failure, where is your backup site?

[TLB:] Given that all my customers are on Earth, I don't need one if my
customers also are "down".

Jeffrey Lyon | 3 Jul 22:29 2009
Picon

Re: Fire, Power loss at Fisher Plaza in Seattle

Wasn't Authorize.net affected by this? We received a support ticket
about why Authorize.net is down today (I don't know either, I don't
ask too many questions).

Jeff

On Fri, Jul 3, 2009 at 4:21 PM, Tomas L. Byrnes<tomb <at> byrneit.net> wrote:
>
>
>>
>>Earth is a single point of failure, where is your backup site?
>
> [TLB:] Given that all my customers are on Earth, I don't need one if my
> customers also are "down".
>
>
>
>

--

-- 
Jeffrey Lyon, Leadership Team
jeffrey.lyon <at> blacklotus.net | http://www.blacklotus.net
Black Lotus Communications of The IRC Company, Inc.

Look for us at HostingCon 2009 in Washington, DC on August 10th - 12th
at Booth #401.

Ben Carleton | 3 Jul 22:29 2009
Picon

Re: Fire, Power loss at Fisher Plaza in Seattle

Yes it was.

On Jul 3, 2009, at 4:29 PM, Jeffrey Lyon wrote:

> Wasn't Authorize.net affected by this? We received a support ticket
> about why Authorize.net is down today (I don't know either, I don't
> ask too many questions).
>
> Jeff
>
> On Fri, Jul 3, 2009 at 4:21 PM, Tomas L. Byrnes<tomb <at> byrneit.net>  
> wrote:
>>
>>
>>>
>>> Earth is a single point of failure, where is your backup site?
>>
>> [TLB:] Given that all my customers are on Earth, I don't need one  
>> if my
>> customers also are "down".
>>
>>
>>
>>
>
>
>
> -- 
> Jeffrey Lyon, Leadership Team
> jeffrey.lyon <at> blacklotus.net | http://www.blacklotus.net
(Continue reading)

Marshall Eubanks | 3 Jul 22:59 2009
Picon

Re: Fire, Power loss at Fisher Plaza in Seattle


On Jul 3, 2009, at 4:29 PM, Ben Carleton wrote:

> Yes it was.
>
> On Jul 3, 2009, at 4:29 PM, Jeffrey Lyon wrote:
>
>> Wasn't Authorize.net affected by this? We received a support ticket
>> about why Authorize.net is down today (I don't know either, I don't
>> ask too many questions).
>>

Authorize.net was for a while completely off the air, and companies  
that relied upon them
were not getting credit card authorizations (and, thus, no ecommerce).  
I think it is still only
partially functional.

Authorize.net has been communicating with customers mostly  
(entirely ?) with twitter - they are

 <at> AuthorizeNet with a hash tab of #authorizenet

If you go there, you will see a lot of status messages like

#authorizenet (cont.) Do not manually submit ARB transactions b/c you  
run the risk of your merchants being double billed.
10 minutes ago from web

(i.e., 4:47 EDT).
(Continue reading)

Jeffrey Lyon | 3 Jul 23:11 2009
Picon

Re: Fire, Power loss at Fisher Plaza in Seattle

That's a great idea, use some lame Web 2.0 trend to communicate with
actual real life customers. </sarcasm>

Jeff

On Fri, Jul 3, 2009 at 4:59 PM, Marshall Eubanks<tme <at> americafree.tv> wrote:
>
> On Jul 3, 2009, at 4:29 PM, Ben Carleton wrote:
>
>> Yes it was.
>>
>> On Jul 3, 2009, at 4:29 PM, Jeffrey Lyon wrote:
>>
>>> Wasn't Authorize.net affected by this? We received a support ticket
>>> about why Authorize.net is down today (I don't know either, I don't
>>> ask too many questions).
>>>
>
> Authorize.net was for a while completely off the air, and companies that
> relied upon them
> were not getting credit card authorizations (and, thus, no ecommerce). I
> think it is still only
> partially functional.
>
> Authorize.net has been communicating with customers mostly (entirely ?) with
> twitter - they are
>
>  <at> AuthorizeNet with a hash tab of #authorizenet
>
> If you go there, you will see a lot of status messages like
(Continue reading)

Marshall Eubanks | 4 Jul 01:23 2009
Picon

Re: Fire, Power loss at Fisher Plaza in Seattle


On Jul 3, 2009, at 5:11 PM, Jeffrey Lyon wrote:

> That's a great idea, use some lame Web 2.0 trend to communicate with
> actual real life customers. </sarcasm>
>

I would assume they figured it was better than just remaining silent.

Regards
Marshall

> Jeff
>
>
> On Fri, Jul 3, 2009 at 4:59 PM, Marshall Eubanks<tme <at> americafree.tv>  
> wrote:
>>
>> On Jul 3, 2009, at 4:29 PM, Ben Carleton wrote:
>>
>>> Yes it was.
>>>
>>> On Jul 3, 2009, at 4:29 PM, Jeffrey Lyon wrote:
>>>
>>>> Wasn't Authorize.net affected by this? We received a support ticket
>>>> about why Authorize.net is down today (I don't know either, I don't
>>>> ask too many questions).
>>>>
>>
>> Authorize.net was for a while completely off the air, and companies  
(Continue reading)

Roland Perry | 4 Jul 12:17 2009

Using twitter as an outage notification (was: Fire, Power loss at Fisher Plaza in Seattle)

In article <786BA8C0-B534-40FF-9126-1E33BD11CB3C <at> americafree.tv>, 
Marshall Eubanks <tme <at> americafree.tv> writes
>> That's a great idea, use some lame Web 2.0 trend to communicate with
>> actual real life customers. </sarcasm>
>>
>I would assume they figured it was better than just remaining silent.

I'm about to recommend to an organisation that it [a twitter account] is 
better than posting news of an outage on their low-volume website, which 
will get swamped when too many people poll it for news.

What does the team think?

Paying a lot more to host the website with higher "burst" capacity 
during an emergency, isn't an option.

The only other idea I've had is to sign all the customers up to receive 
an SMS via some sort of broadcast service (the news will fit easily in 
one SMS).
--

-- 
Roland Perry

Brandon Butterworth | 4 Jul 14:22 2009
Picon
Picon

Re: Using twitter as an outage notification (was: Fire, Power loss at Fisher Plaza in Seattle)

> Paying a lot more to host the website with higher "burst" capacity 
> during an emergency, isn't an option.
> 
> The only other idea I've had is to sign all the customers up to receive 
> an SMS via some sort of broadcast service (the news will fit easily in 
> one SMS).

If the event is suitably calamitous we will do that for you -

http://news.bbc.co.uk/1/hi/help/5194672.stm

brandon

Roland Perry | 4 Jul 14:52 2009

Re: Using twitter as an outage notification (was: Fire, Power loss at Fisher Plaza in Seattle)

In article <200907041222.NAA23352 <at> sunf10.rd.bbc.co.uk>, Brandon 
Butterworth <brandon <at> rd.bbc.co.uk> writes
>> Paying a lot more to host the website with higher "burst" capacity
>> during an emergency, isn't an option.
>>
>> The only other idea I've had is to sign all the customers up to receive
>> an SMS via some sort of broadcast service (the news will fit easily in
>> one SMS).
>
>If the event is suitably calamitous we will do that for you -
>
>http://news.bbc.co.uk/1/hi/help/5194672.stm

The "event" (typically closing a High School because of snow, but we 
have swine flu these days too) is currently reported mainly by local 
radio stations. However it doesn't scale - there are perhaps two hundred 
of them trying to phone in to one radio station during the same 15 
minutes after they made the decision, half an hour before the school is 
supposed to open for the day.

Another problem with a literally "broadcast" system is that it takes 
them too long to read out the names of the schools which are closed, 
even if trying to cover just one county.

Nor does it matter to anyone except a particular closed group of perhaps 
1000 households whether any one school is closed - so telling everyone 
is a bit of a waste.

So it seemed to me that a Tweet from the school would be an ideal 
solution.
(Continue reading)

Michael Holstein | 6 Jul 17:00 2009
Picon

Re: Using twitter as an outage notification (was: Fire, Power loss at Fisher Plaza in Seattle)


> However it doesn't scale

Anyone who's seen the "fail whale" might argue the same about Twitter.

Cheers,

Michael Holstein
Cleveland State University

nevin | 6 Jul 17:08 2009

Re: Using twitter as an outage notification (was: Fire, Power loss at Fisher Plaza in Seattle)

On Monday, July 6, 2009 10:00am, "Michael Holstein" <michael.holstein <at> csuohio.edu> said:
> 
>> However it doesn't scale
> 
> Anyone who's seen the "fail whale" might argue the same about Twitter.
>  
> Cheers,
> 
> Michael Holstein
> Cleveland State University

With a past week of highly visible outages in the data center/provider industry, take a look at the section:
"Crisis Communications Moves Fast" in this story, for another view of using communications channels
like Twitter.  http://www.datacenterknowledge.com/archives/2009/07/06/the-day-after-a-brutal-week-for-uptime/

-- Nevin Lyne
-- CTO
-- EngineHosting.com

Marshall Eubanks | 7 Jul 21:50 2009
Picon

Re: Using twitter as an outage notification (was : Fire, Power loss at Fisher Plaza in Seattle)


On Jul 6, 2009, at 11:08 AM, nevin <at> enginehosting.com wrote:

> On Monday, July 6, 2009 10:00am, "Michael Holstein" <michael.holstein <at> csuohio.edu 
> > said:
>>
>>> However it doesn't scale
>>
>> Anyone who's seen the "fail whale" might argue the same about  
>> Twitter.
>>
>> Cheers,
>>
>> Michael Holstein
>> Cleveland State University
>
> With a past week of highly visible outages in the data center/ 
> provider industry, take a look at the section: "Crisis  
> Communications Moves Fast" in this story, for another view of using  
> communications channels like Twitter.  http://www.datacenterknowledge.com/archives/2009/07/06/the-day-after-a-brutal-week-for-uptime/
>
> -- Nevin Lyne
> -- CTO
> -- EngineHosting.com
>
>

Just to add something to this, twitter has been slow all afternoon and  
now I am getting the "fail whale"

(Continue reading)

Marc Manthey | 7 Jul 22:03 2009
Picon

Re: Using twitter as an outage notification (was : Fire, Power loss at Fisher Plaza in Seattle)


>>>> However it doesn't scale
>>>
>>> Anyone who's seen the "fail whale" might argue the same about  
>>> Twitter.
>
> Just to add something to this, twitter has been slow all afternoon  
> and now I am getting the "fail whale"
>
>
> I just thought I would point out in real time the obvious danger of  
> using a backup service that itself could fail under load,
> especially if your outage and the load could be correlated, say in a  
> disaster or public emergency situation.

yep, got that too several time, but everytime i "reload" the page it  
works , there were plenty of outages before twitter
got the 35 million cash injection , but your absolutly right , its  
centralisted , so there will be serverfarms over serverfarms to get over
these "event" peaks. Would a decentralised system like  http://laconi.ca/ 
   not a better choice ?

just my 50 cents

http://identi.ca/macbroadcast/
--  
Les enfants teribbles - research / deployment
Marc Manthey
Vogelsangerstrasse 97
D - 50823 Köln - Germany
(Continue reading)

Marshall Eubanks | 7 Jul 22:10 2009
Picon

Re: Using twitter as an outage notification (was : Fire, Power loss at Fisher Plaza in Seattle)


On Jul 7, 2009, at 4:03 PM, Marc Manthey wrote:

>
>>>>> However it doesn't scale
>>>>
>>>> Anyone who's seen the "fail whale" might argue the same about  
>>>> Twitter.
>>
>> Just to add something to this, twitter has been slow all afternoon  
>> and now I am getting the "fail whale"
>>
>>
>> I just thought I would point out in real time the obvious danger of  
>> using a backup service that itself could fail under load,
>> especially if your outage and the load could be correlated, say in  
>> a disaster or public emergency situation.
>
> yep, got that too several time, but everytime i "reload" the page it  
> works , there were plenty of outages before twitter
> got the 35 million cash injection , but your absolutly right , its  
> centralisted , so there will be serverfarms over serverfarms to get  
> over
> these "event" peaks. Would a decentralised system like  http://laconi.ca/ 
>   not a better choice ?
>

In a real crisis, redundancy rules.

Regards
(Continue reading)

Mikael Abrahamsson | 7 Jul 22:24 2009
Picon

Re: Using twitter as an outage notification (was : Fire, Power loss at Fisher Plaza in Seattle)

On Tue, 7 Jul 2009, Marshall Eubanks wrote:

> In a real crisis, redundancy rules.

... and simplicity.

It's always "fun" when those outages pages rely on sql backends etc, so 
they're capable of tens or hundreds of users, so they look fine normally. 
When an outage happens and people really need the information and want it, 
things stop working.

I've been advocating a distributed system with static HTML pages being 
generated and pushed out when things change. Huge load capability, you can 
put it anycasted at multiple IXes so it's geographically and ISP 
resiliant, larger ISPs can even request to get their own mirror. Keeping 
it simple.

No takers yet though, people seem to have too much confidence in 
complicated, centralized, nice looking solutions.

--

-- 
Mikael Abrahamsson    email: swmike <at> swm.pp.se

Brandon Galbraith | 7 Jul 22:28 2009
Picon

Re: Re: Using twitter as an outage notification (was : Fire, Power loss at Fisher Plaza in Seattle)

On Tue, Jul 7, 2009 at 3:24 PM, Mikael Abrahamsson<swmike <at> swm.pp.se> wrote:
> On Tue, 7 Jul 2009, Marshall Eubanks wrote:
>
>> In a real crisis, redundancy rules.
>
> ... and simplicity.
>
> It's always "fun" when those outages pages rely on sql backends etc, so
> they're capable of tens or hundreds of users, so they look fine normally.
> When an outage happens and people really need the information and want it,
> things stop working.
>
> I've been advocating a distributed system with static HTML pages being
> generated and pushed out when things change. Huge load capability, you can
> put it anycasted at multiple IXes so it's geographically and ISP
resiliant,
> larger ISPs can even request to get their own mirror. Keeping it simple.
>
> No takers yet though, people seem to have too much confidence in
> complicated, centralized, nice looking solutions.
>
> --
> Mikael Abrahamsson    email: swmike <at> swm.pp.se
>
>

http://www.coralcdn.org/

--

-- 
Brandon Galbraith
(Continue reading)

Mikael Abrahamsson | 7 Jul 22:35 2009
Picon

Re: Re: Using twitter as an outage notification (was : Fire, Power loss at Fisher Plaza in Seattle)

On Tue, 7 Jul 2009, Brandon Galbraith wrote:

> http://www.coralcdn.org/

Nice, looks very much like the thing I was advocating. Hard part is 
getting authorities et al interested in such an "ad hoc" solution. 
Preferrably they could do both and then we can see which one works best in 
an emergency :P

--

-- 
Mikael Abrahamsson    email: swmike <at> swm.pp.se

Marshall Eubanks | 7 Jul 22:29 2009
Picon

Re: Using twitter as an outage notification (was : Fire, Power loss at Fisher Plaza in Seattle)


On Jul 7, 2009, at 4:24 PM, Mikael Abrahamsson wrote:

> On Tue, 7 Jul 2009, Marshall Eubanks wrote:
>
>> In a real crisis, redundancy rules.
>
> ... and simplicity.
>
> It's always "fun" when those outages pages rely on sql backends etc,  
> so they're capable of tens or hundreds of users, so they look fine  
> normally. When an outage happens and people really need the  
> information and want it, things stop working.
>
> I've been advocating a distributed system with static HTML pages  
> being generated and pushed out when things change. Huge load  
> capability, you can put it anycasted at multiple IXes so it's  
> geographically and ISP resiliant, larger ISPs can even request to  
> get their own mirror. Keeping it simple.
>

This would seem to be ideal for P2P, which is decentralized and has  
proven quite resilient under attack.

> No takers yet though, people seem to have too much confidence in  
> complicated, centralized, nice looking solutions.
>

Have you talked to the guys at BitTorrent ? I could make introductions  
during the Stockholm IETF if you need them.
(Continue reading)

Marshall Eubanks | 4 Jul 15:47 2009
Picon

Re: Using twitter as an outage notification (was: Fire, Power loss at Fisher Plaza in Seattle)


On Jul 4, 2009, at 6:17 AM, Roland Perry wrote:

> In article <786BA8C0-B534-40FF-9126-1E33BD11CB3C <at> americafree.tv>,  
> Marshall Eubanks <tme <at> americafree.tv> writes
>>> That's a great idea, use some lame Web 2.0 trend to communicate with
>>> actual real life customers. </sarcasm>
>>>
>> I would assume they figured it was better than just remaining silent.
>
> I'm about to recommend to an organisation that it [a twitter  
> account] is better than posting news of an outage on their low- 
> volume website, which will get swamped when too many people poll it  
> for news.
>

What if the outage takes out their website too ?

I don't think that their website was up, and I would guess that they  
didn't have email either. That
is a bad situation to be in.

Note, BTW, that twitter itself is subject to frequent planned and  
unplanned outages.

Marshall

> What does the team think?
>
> Paying a lot more to host the website with higher "burst" capacity  
(Continue reading)

Roland Perry | 4 Jul 16:43 2009

Re: Using twitter as an outage notification (was: Fire, Power loss at Fisher Plaza in Seattle)

In article <F832A12A-0AED-4A01-955D-E24DCA6181C8 <at> americafree.tv>, 
Marshall Eubanks <tme <at> americafree.tv> writes

>>>> That's a great idea, use some lame Web 2.0 trend to communicate with
>>>> actual real life customers. </sarcasm>
>>>>
>>> I would assume they figured it was better than just remaining silent.
>>
>> I'm about to recommend to an organisation that it [a twitter account] 
>>is better than posting news of an outage on their low- volume website, 
>>which will get swamped when too many people poll it for news.
>
>What if the outage takes out their website too ?

The website is hosted elsewhere, however the entire message can be 
delivered in one Tweet, so there's no need to confirm by looking at a 
website.

>I don't think that their website was up, and I would guess that they 
>didn't have email either. That is a bad situation to be in.

They don't plan to respond to email in real time.

>Note, BTW, that twitter itself is subject to frequent planned and 
>unplanned outages.

The question being, how often will they co-incide with the events I'm 
trying to track?

fwiw, I've been using twitter for about three months now, and have never 
(Continue reading)

Jeffrey Lyon | 4 Jul 16:47 2009
Picon

Re: Using twitter as an outage notification (was: Fire, Power loss at Fisher Plaza in Seattle)

Personally, I find it difficult to take Twitter seriously. It seems
like more of a kids toy than a business tool. Something like a
blogspot account would make a lot more sense.

Jeff

On 7/4/09, Marshall Eubanks <tme <at> americafree.tv> wrote:
>
>  On Jul 4, 2009, at 6:17 AM, Roland Perry wrote:
>
>
> > In article
> <786BA8C0-B534-40FF-9126-1E33BD11CB3C <at> americafree.tv>,
> Marshall Eubanks <tme <at> americafree.tv> writes
> >
> > >
> > > > That's a great idea, use some lame Web 2.0 trend to communicate with
> > > > actual real life customers. </sarcasm>
> > > >
> > > >
> > > I would assume they figured it was better than just remaining silent.
> > >
> >
> > I'm about to recommend to an organisation that it [a twitter account] is
> better than posting news of an outage on their low-volume website, which
> will get swamped when too many people poll it for news.
> >
> >
>
>  What if the outage takes out their website too ?
(Continue reading)

Michael Hallgren | 4 Jul 16:58 2009
Picon

Re: Using twitter as an outage notification (was: Fire, Power loss at Fisher Plaza in Seattle)

Le samedi 04 juillet 2009 à 10:47 -0400, Jeffrey Lyon a écrit :
> Personally, I find it difficult to take Twitter seriously. It seems
> like more of a kids toy than a business tool. Something like a
> blogspot account would make a lot more sense.

Yes.

What about (continue to) use old email (inc lists), coupled with 
some roughly out-of-band like cell/pots/sms service? And in parallel 
old irc, et al. 

Any severe problem with, asking us to move over to "portal 
services"?

mh
> 
> Jeff
> 
> 
> 
> On 7/4/09, Marshall Eubanks <tme <at> americafree.tv> wrote:
> >
> >  On Jul 4, 2009, at 6:17 AM, Roland Perry wrote:
> >
> >
> > > In article
> > <786BA8C0-B534-40FF-9126-1E33BD11CB3C <at> americafree.tv>,
> > Marshall Eubanks <tme <at> americafree.tv> writes
> > >
> > > >
(Continue reading)

Michael Hallgren | 4 Jul 17:26 2009
Picon

Re: Using twitter as an outage notification (was: Fire, Power loss at Fisher Plaza in Seattle)

Le samedi 04 juillet 2009 à 16:58 +0200, Michael Hallgren a écrit :
> Le samedi 04 juillet 2009 à 10:47 -0400, Jeffrey Lyon a écrit :
> > Personally, I find it difficult to take Twitter seriously. It seems
> > like more of a kids toy than a business tool. Something like a
> > blogspot account would make a lot more sense.
> 
> Yes.
> 
> What about (continue to) use old email (inc lists), coupled with 
> some roughly out-of-band like cell/pots/sms service? And in parallel 
> old irc, et al. 
> 
> Any severe problem with, asking us to move over to "portal 
> services"?

Of course not negative with respect to new innovative means... But if
we didn't have pidgin: msn, yahoo!, gtalk, icq, facebook,... ... 
would be hard to manage... and remember who's message to track via
what channel...

So, the channel I think is much dependent on the audience. The crowd
small enough, most any means will be fine. The crowd more universal,
well-known, stable communication protocols should be a natural
choice. No?

mh

> 
> mh
> > 
(Continue reading)

Roland Perry | 4 Jul 17:07 2009

Re: Using twitter as an outage notification (was: Fire, Power loss at Fisher Plaza in Seattle)

In article 
<16720fe00907040747k67ca1206kb871420deb5e8163 <at> mail.gmail.com>, Jeffrey 
Lyon <jeffrey.lyon <at> blacklotus.net> writes
>Personally, I find it difficult to take Twitter seriously. It seems
>like more of a kids toy than a business tool. Something like a
>blogspot account would make a lot more sense.

That's the kind of "marketing-led" response I was hoping to hear.

But the UK National Rail system now uses Tweets to tell customers about 
disruptions on the trains, and several major UK government departments 
and news organisations use it for announcements and "Breaking News".

So has it become "respectable" yet?
--

-- 
Roland Perry

Chris Hills | 4 Jul 17:22 2009

Re: Using twitter as an outage notification

On 04/07/09 17:07, Roland Perry wrote:
> That's the kind of "marketing-led" response I was hoping to hear.
>
> But the UK National Rail system now uses Tweets to tell customers about
> disruptions on the trains, and several major UK government departments
> and news organisations use it for announcements and "Breaking News".
>
> So has it become "respectable" yet?

When there are open-source equivalents available (e.g. Laconica, 
OpenMicroBlogger - both of which incidentally are compatible since they 
are based upon the OMB spec), I do wonder why a commercial or government 
entity would use a closed-source, non-domestic service.

Roland Perry | 4 Jul 17:37 2009

Re: Using twitter as an outage notification

In article <h2ns2s$kcv$1 <at> ger.gmane.org>, Chris Hills <chaz <at> chaz6.com> 
writes
>> That's the kind of "marketing-led" response I was hoping to hear.
>>
>> But the UK National Rail system now uses Tweets to tell customers about
>> disruptions on the trains, and several major UK government departments
>> and news organisations use it for announcements and "Breaking News".
>>
>> So has it become "respectable" yet?
>
>When there are open-source equivalents available (e.g. Laconica, 
>OpenMicroBlogger - both of which incidentally are compatible since they 
>are based upon the OMB spec), I do wonder why a commercial or 
>government entity would use a closed-source, non-domestic service.

That's fair comment, but how do you get your customers to install quirky 
niche solutions to what's a once-a-year problem?

They all seem pretty happy using a multitude of other "non-domestic" 
solutions, which probably accounts for 99% of the stuff they have on 
their PCs.

So "not sufficiently mature" we can get away with as an excuse, but 
"Made in America" isn't going to put many people off :)
--

-- 
Roland Perry

Frank Bulk | 4 Jul 22:59 2009
Picon

RE: Using twitter as an outage notification

When the local power companies uses twitter, then maybe I'll consider using
twitter for our customers.

There's the temptation by some of companies to leverage the latest
technology to appear "cool" and "in tune" with customers, but by far and
large, when something goes down customers either do no nothing, wait, or
call in.  I think the best use of everyone's time is to make sure their call
center/support desk has the capability to post an announcement to those that
call in.  And then make sure something gets posted to the website.  SMS,
Facebook, and Twitter fall in line after all that.

Frank

-----Original Message-----
From: Roland Perry [mailto:lists <at> internetpolicyagency.com] 
Sent: Saturday, July 04, 2009 10:38 AM
To: nanog <at> merit.edu
Subject: Re: Using twitter as an outage notification

In article <h2ns2s$kcv$1 <at> ger.gmane.org>, Chris Hills <chaz <at> chaz6.com> 
writes
>> That's the kind of "marketing-led" response I was hoping to hear.
>>
>> But the UK National Rail system now uses Tweets to tell customers about
>> disruptions on the trains, and several major UK government departments
>> and news organisations use it for announcements and "Breaking News".
>>
>> So has it become "respectable" yet?
>
>When there are open-source equivalents available (e.g. Laconica, 
(Continue reading)

Marc Manthey | 4 Jul 23:07 2009
Picon

Re: Using twitter as an outage notification


Am 04.07.2009 um 22:59 schrieb Frank Bulk:

> When the local power companies uses twitter, then maybe I'll  
> consider using
> twitter for our customers.

well it seems popular

http://www.dell.com/twitter

dell made some money with it too

http://en.community.dell.com/blogs/direct2dell/archive/2009/06/11/delloutlet-surpasses-2-million-on-twitter.aspx

:-))

>
> There's the temptation by some of companies to leverage the latest
> technology to appear "cool" and "in tune" with customers, but by far  
> and
> large, when something goes down customers either do no nothing,  
> wait, or
> call in.  I think the best use of everyone's time is to make sure  
> their call
> center/support desk has the capability to post an announcement to  
> those that
> call in.  And then make sure something gets posted to the website.   
> SMS,
> Facebook, and Twitter fall in line after all that.
(Continue reading)

Seth Mattinen | 4 Jul 23:09 2009
Picon

Re: Using twitter as an outage notification

Frank Bulk wrote:
> When the local power companies uses twitter, then maybe I'll consider using
> twitter for our customers.
> 
> There's the temptation by some of companies to leverage the latest
> technology to appear "cool" and "in tune" with customers, but by far and
> large, when something goes down customers either do no nothing, wait, or
> call in.  I think the best use of everyone's time is to make sure their call
> center/support desk has the capability to post an announcement to those that
> call in.  And then make sure something gets posted to the website.  SMS,
> Facebook, and Twitter fall in line after all that.
> 

A plain-text status website and recorded message before the phone tree 
scale quite well in the event of a major problem.

But yeah, Twitter will attract the "what's cool right now" demographic.

~Seth

Roland Perry | 5 Jul 12:10 2009

Re: Using twitter as an outage notification

In article <4A4FC4F3.2010301 <at> rollernet.us>, Seth Mattinen 
<sethm <at> rollernet.us> writes
>Twitter will attract the "what's cool right now" demographic.

But has it gone from "cool" to "useful" (for this kind of application), 
in a way that Facebook and other such sites haven't?

I remember an employer of mine when I was trying to persuade him to 
build a modem into a PC so people could exchange what we'd now call 
emails, and he said "Roland, come back and ask me again, when I can pay 
your wages through that modem thing".

Unfortunately I didn't get the opportunity, as I left there twenty years 
ago, and Paypal wasn't invented until about ten years ago.
--

-- 
Roland Perry

Mark E. Mallett | 4 Jul 23:12 2009

Re: Using twitter as an outage notification

On Sat, Jul 04, 2009 at 03:59:48PM -0500, Frank Bulk wrote:
> When the local power companies uses twitter, then maybe I'll consider using
> twitter for our customers.

During the ice storm we had here last winter, the local power
company did just that.  "psnh" "ice storm" "twitter" etc are all
good search terms.

I haven't been following this thread closely, so I dunno if that had
already been mentioned and your remark was a sardonic one. Nevertheless..

-mm-  (probably due for an annual nanog posting)

Michael Painter | 5 Jul 06:29 2009

Re: Using twitter as an outage notification

----- Original Message ----- 
From: "Frank Bulk"
Sent: Saturday, July 04, 2009 10:59 AM
Subject: RE: Using twitter as an outage notification

> When the local power companies uses twitter, then maybe I'll consider using
> twitter for our customers.
>
> There's the temptation by some of companies to leverage the latest
> technology to appear "cool" and "in tune" with customers, but by far and
> large, when something goes down customers either do no nothing, wait, or
> call in.  I think the best use of everyone's time is to make sure their call
> center/support desk has the capability to post an announcement to those that
> call in.  And then make sure something gets posted to the website.  SMS,
> Facebook, and Twitter fall in line after all that.
>
> Frank

I thought this was interesting:

"Bonnie Smalley has Internet bragging rights: She has been blocked by Twitter for hand-typing too many
tweets in an hour. 
They thought she was a computer program made to spew spam.
Ms. Smalley, it turns out, is a 100 percent human customer service representative for Comcast. She is one of
10 
representatives who reach out to customers through social networks, rather than waiting for them to find
Comcast's support

site."

(Continue reading)

Valdis.Kletnieks | 7 Jul 03:28 2009
Picon

Re: Using twitter as an outage notification

Seen on NANOG:

On Sat, 04 Jul 2009 18:29:22 -1000, Michael Painter said:

> "Bonnie Smalley has Internet bragging rights: She has been blocked by Twitter
> for hand-typing too many tweets in an hour. They thought she was a computer
> program made to spew spam.

> Ms. Smalley, it turns out, is a 100 percent human customer service
> representative for Comcast. She is one of 10 representatives who reach out to
> customers through social networks, rather than waiting for them to find
> Comcast's support site.

>
http://www.nytimes.com/2009/07/02/technology/personaltech/02basics.html?partner=rss&emc=rss 

There's a Dilbert strip in there, fighting to get out.. ;)

Seen on NANOG:

On Sat, 04 Jul 2009 18:29:22 -1000, Michael Painter said:

> "Bonnie Smalley has Internet bragging rights: She has been blocked by Twitter
> for hand-typing too many tweets in an hour. They thought she was a computer
> program made to spew spam.

> Ms. Smalley, it turns out, is a 100 percent human customer service
> representative for Comcast. She is one of 10 representatives who reach out to
(Continue reading)

Tomas L. Byrnes | 7 Jul 05:25 2009
Picon

Re: Using twitter as an outage notification

>-----Original Message-----
>From: funsec-bounces <at> linuxbox.org [mailto:funsec-bounces <at> linuxbox.org]
>On Behalf Of Valdis.Kletnieks <at> vt.edu
>Sent: Monday, July 06, 2009 6:29 PM
>To: funsec <at> linuxbox.org
>Subject: Re: [funsec] Using twitter as an outage notification
>
>Seen on NANOG:
>
>On Sat, 04 Jul 2009 18:29:22 -1000, Michael Painter said:
>
>> "Bonnie Smalley has Internet bragging rights: She has been blocked by
>> Twitter for hand-typing too many tweets in an hour. They thought she
>> was a computer program made to spew spam.
>
>> Ms. Smalley, it turns out, is a 100 percent human customer service
>> representative for Comcast. She is one of 10 representatives who
reach
>> out to customers through social networks, rather than waiting for
them
>> to find Comcast's support site.
>
>>
http://www.nytimes.com/2009/07/02/technology/personaltech/02basics.htm
>> l?partner=rss&emc=rss
>
>There's a Dilbert strip in there, fighting to get out.. ;)
>

Hmm, so what's the opposite of a Turing test?
(Continue reading)

Roland Perry | 5 Jul 12:01 2009

Re: Using twitter as an outage notification

In article 
<!&!AAAAAAAAAAAuAAAAAAAAAKTyXRN5/+lGvU59a+P7CFMBAN6gY+ZG84BMpVQcAbDh1IQAA
AATbSgAABAAAAAuldg0EWkrSZ9BD0db8+e2AQAAAAA= <at> iname.com>, Frank Bulk 
<frnkblk <at> iname.com> writes
>When the local power companies uses twitter, then maybe I'll consider using
>twitter for our customers.

That's a poor example as far as the UK's concerned. You can't get 
information from the power company for days if you are a domestic 
customer.

>There's the temptation by some of companies to leverage the latest
>technology to appear "cool" and "in tune" with customers, but by far and
>large, when something goes down customers either do no nothing, wait, or
>call in.  I think the best use of everyone's time is to make sure their call
>center/support desk has the capability to post an announcement to those that
>call in.

It's a High School. They don't have a "support desk" (or more than 
handful of phone lines [1]). Even the local radio station can't cope 
with one call per school asking them to broadcast the news that they 
have closed due to bad weather.

>And then make sure something gets posted to the website.

Unfortunately, the number of students polling the website for news means 
it can't cope with the traffic. I don't believe they can justify paying 
more for better web hosting, just to manage this once-a-year half hour 
event.

(Continue reading)

Adrian Chadd | 5 Jul 12:12 2009
Picon

Re: Using twitter as an outage notification

On Sun, Jul 05, 2009, Roland Perry wrote:

> Unfortunately, the number of students polling the website for news means 
> it can't cope with the traffic. I don't believe they can justify paying 
> more for better web hosting, just to manage this once-a-year half hour 
> event.

Is Twitter making a profit or not?

This discussion about (ab)using a publicly available message system which
isn't currently being charged for would makes me worried^Wamused as hell.

(Not that I can't see possible ways of making money off twitter - especially
if you offer reliable message dissemination services to companies for a fee,
but AFAIK they aren't doing this at the moment.)

Adrian

Roland Dobbins | 5 Jul 12:31 2009
Picon

Re: Using twitter as an outage notification


On Jul 5, 2009, at 5:12 PM, Adrian Chadd wrote:

> Is Twitter making a profit or not?

The other consideration is scalability and reliability.  Twitter has  
been subject to numerous feature disablements due to capacity issues,  
as well as complete outages.  Furthermore, Twitter does not appear to  
be deployed in a distributed, highly-available architecture.

The Twitter *aggregation/attention model* is what is of great  
interest, any merits of the specific service aside.

-----------------------------------------------------------------------
Roland Dobbins <rdobbins <at> arbor.net> // <http://www.arbornetworks.com>

         Unfortunately, inefficiency scales really well.

		   -- Kevin Lawton

Marc Manthey | 5 Jul 12:40 2009
Picon

Re: Using twitter as an outage notification

>
>> Is Twitter making a profit or not?
>
> The other consideration is scalability and reliability.  Twitter has  
> been subject to numerous feature disablements due to capacity  
> issues, as well as complete outages.  Furthermore, Twitter does not  
> appear to be deployed in a distributed, highly-available architecture.
>
> The Twitter *aggregation/attention model* is what is of great  
> interest, any merits of the specific service aside.

exactly, its like VHS versus BETAMAX , not the better system wins,its  
just better maketing and popularity.

just my 2 cents

http://identi.ca/macbroadcast/      < the opensource distributed  
alternative  > http://laconi.ca/

--  
Les enfants teribbles - research / deployment
Marc Manthey
Vogelsangerstrasse 97
D - 50823 Köln - Germany
Vogelsangerstrasse 97
Geo: 50.945554, 6.920293
PGP/GnuPG: 0x1ac02f3296b12b4d
Tel.:0049-221-29891489
Mobil:0049-1577-3329231
web : http://www.let.de
(Continue reading)

Marshall Eubanks | 5 Jul 12:39 2009
Picon

Re: Using twitter as an outage notification


On Jul 5, 2009, at 6:12 AM, Adrian Chadd wrote:

> On Sun, Jul 05, 2009, Roland Perry wrote:
>
>> Unfortunately, the number of students polling the website for news  
>> means
>> it can't cope with the traffic. I don't believe they can justify  
>> paying
>> more for better web hosting, just to manage this once-a-year half  
>> hour
>> event.
>
> Is Twitter making a profit or not?
>

The word on the street is that they have not yet "found a revenue  
model". In other words,
they make no money. They seem very dot com 1.0 unconcerned with this.  
That obviously cannot last.

The speculation on how to fix that tend to be either focused advertising

http://www.readwriteweb.com/archives/the_ultimate_twitter_revenue_model.php

ecommerce

http://blogs.imediaconnection.com/2009/6/22/A-real-Twitter-revenue-model---gasp-_731.aspx

or Google type data mining models (you can detect trends very
(Continue reading)

Roland Perry | 5 Jul 15:10 2009

Re: Using twitter as an outage notification

In article <9589B202-ED92-4C49-98EE-EEBAA43C87AA <at> americafree.tv>, 
Marshall Eubanks <tme <at> americafree.tv> writes
>as I said  before, this is a service that goes down. I would not rely 
>on it as the only way to  communicate.

I'd be proposing it as an additional way to communicate[1], but people 
could come to rely upon it.

[1] the present system seems to be those few students who can get 
through to the school then SMS the news to their friends.
--

-- 
Roland Perry

Roland Perry | 5 Jul 15:06 2009

Re: Using twitter as an outage notification

In article <20090705101237.GC14222 <at> skywalker.creative.net.au>, Adrian 
Chadd <adrian <at> creative.net.au> writes
>Is Twitter making a profit or not?
>
>This discussion about (ab)using a publicly available message system which
>isn't currently being charged for would makes me worried^Wamused as hell.

I've seen debates about whether it's possible to monetise Twitter. 
Operationally, it's an issue if they fail financially, but I don't think 
the investment in setting up an account is large enough to worry about.

Counter-intuitively, I've probably seen more subscription-based services 
fail, than free ones.
--

-- 
Roland Perry

Steve Pirk | 5 Jul 12:43 2009

Re: Using twitter as an outage notification

On Sun, 5 Jul 2009, Roland Perry wrote:
>> There's the temptation by some of companies to leverage the latest
>> technology to appear "cool" and "in tune" with customers, but by far and
>> large, when something goes down customers either do no nothing, wait, or
>> call in.  I think the best use of everyone's time is to make sure their 
>> call
>> center/support desk has the capability to post an announcement to those 
>> that
>> call in.
>
> It's a High School. They don't have a "support desk" (or more than handful of 
> phone lines [1]). Even the local radio station can't cope with one call per 
> school asking them to broadcast the news that they have closed due to bad 
> weather.
>
If your resources are that tight, do what our local school district 
did, mandate that all bus schedules will only be available on the web 
site.

>> And then make sure something gets posted to the website.
>
> Unfortunately, the number of students polling the website for news means it 
> can't cope with the traffic. I don't believe they can justify paying more for 
> better web hosting, just to manage this once-a-year half hour event.
>

Roland, sounds like you should have a few "public service" 
announcements saying that school closures will be delivered via a 
certain twitter username. Also send a flyer home with the students.

(Continue reading)

Joe Blanchard | 5 Jul 13:08 2009
Picon

RE: Using twitter as an outage notification


My gosh, <sarcasm> Ok, how about we use Facebook, myspace and the other
assorted community websites/services, no better
yet lets use AOL! </sarcasm>

Can we kill this thread please (for those that are still on AOL that's PLZ) 

This list is for professional content, not for boasting about high school
websties/services that will die
out in the next year or so.  

> -----Original Message-----
> From: Steve Pirk [mailto:orion <at> pirk.com] 
> Sent: Sunday, July 05, 2009 6:43 AM
> To: Roland Perry
> Cc: nanog <at> merit.edu
> Subject: Re: Using twitter as an outage notification
> 
> On Sun, 5 Jul 2009, Roland Perry wrote:
> >> There's the temptation by some of companies to leverage the latest 
> >> technology to appear "cool" and "in tune" with customers, 
bunch of $h1t, bunch of $h1t...
bunch of $h1t, bunch of $h1t...
bunch of $h1t, bunch of $h1t...
bunch of $h1t, bunch of $h1t...
bunch of $h1t, bunch of $h1t...
bunch of $h1t, bunch of $h1t...
bunch of $h1t, bunch of $h1t...
bunch of $h1t, bunch of $h1t...
bunch of $h1t, bunch of $h1t...
(Continue reading)

Roland Perry | 5 Jul 15:16 2009

Re: Using twitter as an outage notification

In article <Pine.LNX.4.64.0907050334130.23511 <at> mail.pirk.com>, Steve Pirk 
<orion <at> pirk.com> writes

>> It's a High School. They don't have a "support desk" (or more than 
>>handful of  phone lines [1]). Even the local radio station can't cope 
>>with one call per  school asking them to broadcast the news that they 
>>have closed due to bad  weather.
>>
>If your resources are that tight, do what our local school district 
>did, mandate that all bus schedules will only be available on the web 
>site.

The school doesn't have any buses. About 80% of the students walk (the 
average distance maybe a little over a mile) and most of the rest get 
taken in their parents car.

>Roland, sounds like you should have a few "public service" 
>announcements saying that school closures will be delivered via a 
>certain twitter username.

That's what my objective is - to build a sturdy enough case for the 
school to have a twitter account to use during these events.

>Also send a flyer home with the students.

Only about half of those ever reach home (no-one knows where they end 
up, but it's probably the same place as all those lost Biros).

But if the school had a twitter account I'm sure the news would spread 
rapidly. Most of the students spend hours online every day, even if the 
(Continue reading)

Matthew Palmer | 5 Jul 13:32 2009

Re: Using twitter as an outage notification

On Sun, Jul 05, 2009 at 11:01:43AM +0100, Roland Perry wrote:
[snow day notifications]
> Unfortunately, the number of students polling the website for news means  
> it can't cope with the traffic. I don't believe they can justify paying  
> more for better web hosting, just to manage this once-a-year half hour  
> event.

There are web hosting providers whose 18c/year hosting plans can't handle a
few thousand requests to a static page over a period of maybe 15 minutes
without falling over?  The mind boggles.

- Matt

Roland Perry | 5 Jul 15:19 2009

Re: Using twitter as an outage notification

In article <20090705113248.GP1499 <at> hezmatt.org>, Matthew Palmer 
<mpalmer <at> hezmatt.org> writes
>There are web hosting providers whose 18c/year hosting plans can't handle a
>few thousand requests to a static page over a period of maybe 15 minutes
>without falling over?  The mind boggles.

Apparently so. Of course, they could be deliberately throttled, rather 
than run on inherently low-bandwidth kit. Which raises the issue of 
whether such throttling schemes should take account of short bursts.
--

-- 
Roland Perry

JC Dill | 5 Jul 17:17 2009
Picon

Re: Using twitter as an outage notification

Roland Perry wrote:
>> There's the temptation by some of companies to leverage the latest
>> technology to appear "cool" and "in tune" with customers, but by far and
>> large, when something goes down customers either do no nothing, wait, or
>> call in.  I think the best use of everyone's time is to make sure 
>> their call
>> center/support desk has the capability to post an announcement to 
>> those that
>> call in.
>
> It's a High School. They don't have a "support desk" (or more than 
> handful of phone lines [1]). Even the local radio station can't cope 
> with one call per school asking them to broadcast the news that they 
> have closed due to bad weather.
>
>> And then make sure something gets posted to the website.
>
> Unfortunately, the number of students polling the website for news 
> means it can't cope with the traffic. 
Really?  Um, wow.  How big is this school?  Is the webserver on an ISDN 
line?
> I don't believe they can justify paying more for better web hosting, 
> just to manage this once-a-year half hour event.
This is a case where it makes *perfect* sense to offload emergency 
notifications to another, larger system such as twitter, *as well as* 
post to the school website (ideally via a blog, so you can use posterus 
to do both actions in one email).  There's no fee, the cost to "set 
up"[1] is your time to securely configure a posterus account and a list 
to send to posterus (see below) and then to send an announcement post to 
posterus (and thus post on the school blog and on twitter) and to send 
(Continue reading)

Roland Perry | 5 Jul 19:04 2009

Re: Using twitter as an outage notification

In article <4A50C401.9070307 <at> gmail.com>, JC Dill 
<jcdill.lists <at> gmail.com> writes

>> Unfortunately, the number of students polling the website for news 
>>means it can't cope with the traffic.

>Really?  Um, wow.  How big is this school?  Is the webserver on an ISDN 
>line?

It appears to be at a co-location centre in a distant city. I expect 
it's provided as part of a package by one of the $5 domain hosting 
companies. The bandwidth limiting is more likely a quota than a lack of 
connectivity.

>> I don't believe they can justify paying more for better web hosting, 
>>just to manage this once-a-year half hour event.

>This is a case where it makes *perfect* sense to offload emergency 
>notifications to another, larger system such as twitter,

That's my current view, too.

> you can use posterus

It's going to be hard enough getting them to be comfortable with 
Twitter.
--

-- 
Roland Perry

(Continue reading)

mike | 4 Jul 17:33 2009

Re: Using twitter as an outage notification

Roland Perry wrote:
> In article
> <16720fe00907040747k67ca1206kb871420deb5e8163 <at> mail.gmail.com>, Jeffrey
> Lyon <jeffrey.lyon <at> blacklotus.net> writes
>> Personally, I find it difficult to take Twitter seriously. It seems
>> like more of a kids toy than a business tool. Something like a
>> blogspot account would make a lot more sense.
>
> That's the kind of "marketing-led" response I was hoping to hear.
>
> But the UK National Rail system now uses Tweets to tell customers
> about disruptions on the trains, and several major UK government
> departments and news organisations use it for announcements and
> "Breaking News".
>
> So has it become "respectable" yet?
there are plenty of examples where twitter is being used for useful
notifications. in the sf bay area, there's a user maintained version of
what you describe for our commuter rail. a large example of this is
comcast's customer service (see http://twitter.com/comcastcares) 
personally, i like the twitter idea. i can follow/unfollow at will, i
can set up sms alerts for specific "users" i follow, etc. sure, twitter
had some stability issues, but i think if we're being fair, they've been
very stable of late. sure, twitter might be down at the same time, but
it seems more likely that the website for the provider in question would
be affected and twitter can be updated very quickly using a cell phone
either with a twitter app or simply via sms.

just my $0.02 worth (perhaps $0.03 and perhaps not worth over $0.01)

(Continue reading)

JC Dill | 4 Jul 15:50 2009
Picon

Re: Using twitter as an outage notification

Roland Perry wrote:
> In article <786BA8C0-B534-40FF-9126-1E33BD11CB3C <at> americafree.tv>, 
> Marshall Eubanks <tme <at> americafree.tv> writes
>>> That's a great idea, use some lame Web 2.0 trend to communicate with
>>> actual real life customers. </sarcasm>
>>>
>> I would assume they figured it was better than just remaining silent.
>
> I'm about to recommend to an organisation that it [a twitter account] 
> is better than posting news of an outage on their low-volume website, 
> which will get swamped when too many people poll it for news.
>
> What does the team think? 

I don't understand why this is an either/or question.  Why not post to both?

Twitter is a great method of communication, for those that use twitter.  
But some people don't use twitter.  So use every avenue you have.  If 
you have a customer mailing list for announcements, send email.  If you 
have a blog, post to the blog.

I'm in the process of setting up posterous to post to a blog, myspace, 
facebook, and twitter all with one email.  Can't get much simpler than 
that for getting the word out via all the channels (assuming that 
outbound email is working).  (Obviously some businesses don't want/need 
myspace or facebook, but if you use them, post there too.)

http://posterous.com/autopost

Quoting:
(Continue reading)

Roland Perry | 4 Jul 16:50 2009

Re: Using twitter as an outage notification

In article <4A4F5E3C.5040301 <at> gmail.com>, JC Dill 
<jcdill.lists <at> gmail.com> writes

>>>> That's a great idea, use some lame Web 2.0 trend to communicate with
>>>> actual real life customers. </sarcasm>
>>>>
>>> I would assume they figured it was better than just remaining silent.
>>
>> I'm about to recommend to an organisation that it [a twitter account] 
>>is better than posting news of an outage on their low-volume website, 
>>which will get swamped when too many people poll it for news.
>>
>> What does the team think?
>
>I don't understand why this is an either/or question.  Why not post to both?

Yes, that can be done.

What I'm trying to anticipate is the objection to *also* posting to 
Twitter, which might be raised on the grounds that it's too 
"unofficial", or "unsupported" or something like that.

>You control where we post.
>
>Just email the right address and we'll do the right thing.
>Post Everywhere?    post <at> posterous.com as usual
>Twitter?    twitter <at> posterous.com
>Flickr?    flickr <at> posterous.com
>Facebook?    facebook <at> posterous.com
>Tumblr?    tumblr <at> posterous.com
(Continue reading)

JC Dill | 4 Jul 17:02 2009
Picon

Re: Using twitter as an outage notification

Roland Perry wrote:
>
> What I'm trying to anticipate is the objection to *also* posting to 
> Twitter, which might be raised on the grounds that it's too 
> "unofficial", or "unsupported" or something like that.
Anyone who makes that argument is just showing how little they know 
about Twitter.  It would be like complaining you shouldn't use email 
because "not everyone has email".

>> You control where we post.
>>
>> Just email the right address and we'll do the right thing.
>> Post Everywhere?    post <at> posterous.com as usual
>> Twitter?    twitter <at> posterous.com
>> Flickr?    flickr <at> posterous.com
>> Facebook?    facebook <at> posterous.com
>> Tumblr?    tumblr <at> posterous.com
>> Any other blog?    blog <at> posterous.com
>> Posterous only?    posterous <at> posterous.com
>> Combine them!    flickr+twitter <at> posterous.com
>
> It's this richness which confuses the ordinary person. 
That's a lot like saying Perl is too complicated for the ordinary person 
so never use Perl.  :-)
> How are they to know which bit of the scattergun approach is the right 
> one to use? Or whether "posting everywhere" has some hidden disadvantage.

You can configure it and use it however YOU want.  If it's too confusing 
to use it selectively - sometimes posting in all places, sometimes 
posting in just one place, then configure it for your preferred use and 
(Continue reading)

Roland Perry | 4 Jul 17:16 2009

Re: Using twitter as an outage notification

In article <4A4F6EF5.9030502 <at> gmail.com>, JC Dill 
<jcdill.lists <at> gmail.com> writes

>> What I'm trying to anticipate is the objection to *also* posting to 
>>Twitter, which might be raised on the grounds that it's too 
>>"unofficial", or "unsupported" or something like that.

>Anyone who makes that argument is just showing how little they know 
>about Twitter.

So that's 98% of the population then...

>It would be like complaining you shouldn't use email because "not 
>everyone has email".

But email has become respectable, although I still see "people who know 
better" starting speeches with 'of course, ten years ago none of us used 
email, but now....' which shows they are very late adopters themselves.

>>> You control where we post.
>>>
>>> Just email the right address and we'll do the right thing.
>>> Post Everywhere?    post <at> posterous.com as usual
>>> Twitter?    twitter <at> posterous.com
>>> Flickr?    flickr <at> posterous.com
>>> Facebook?    facebook <at> posterous.com
>>> Tumblr?    tumblr <at> posterous.com
>>> Any other blog?    blog <at> posterous.com
>>> Posterous only?    posterous <at> posterous.com
>>> Combine them!    flickr+twitter <at> posterous.com
(Continue reading)

JC Dill | 5 Jul 00:19 2009
Picon

Re: Using twitter as an outage notification

Roland Perry wrote:
> In article <4A4F6EF5.9030502 <at> gmail.com>, JC Dill 
> <jcdill.lists <at> gmail.com> writes
>
>>> What I'm trying to anticipate is the objection to *also* posting to 
>>> Twitter, which might be raised on the grounds that it's too 
>>> "unofficial", or "unsupported" or something like that.
>
>> Anyone who makes that argument is just showing how little they know 
>> about Twitter.
>
> So that's 98% of the population then...

We aren't talking about the general population.  IMHO anyone in Network 
Operations or NOC management who doesn't know about emerging and cutting 
edge communications is in the wrong job or the wrong industry.
>
>> It would be like complaining you shouldn't use email because "not 
>> everyone has email".
>
> But email has become respectable, although I still see "people who 
> know better" starting speeches with 'of course, ten years ago none of 
> us used email, but now....' which shows they are very late adopters 
> themselves.
How many of them are running Internet Networks?

>>>
>>> It's this richness which confuses the ordinary person.
>
>> That's a lot like saying Perl is too complicated for the ordinary 
(Continue reading)

Stefan | 5 Jul 01:34 2009
Picon

Re: Using twitter as an outage notification

For DR issues (among many others, of course) think of Twitter as a paging
system of global proportions: not a lot to be said, but if you get the
message right its broadcast and amplification capabilities are unmatched.

-- 
***Stefan
http://twitter.com/netfortius

On Sat, Jul 4, 2009 at 5:19 PM, JC Dill <jcdill.lists <at> gmail.com> wrote:

> Roland Perry wrote:
>
>> In article <4A4F6EF5.9030502 <at> gmail.com>, JC Dill <jcdill.lists <at> gmail.com>
>> writes
>>
>>  What I'm trying to anticipate is the objection to *also* posting to
>>>> Twitter, which might be raised on the grounds that it's too "unofficial", or
>>>> "unsupported" or something like that.
>>>>
>>>
>>  Anyone who makes that argument is just showing how little they know about
>>> Twitter.
>>>
>>
>> So that's 98% of the population then...
>>
>
> We aren't talking about the general population.  IMHO anyone in Network
> Operations or NOC management who doesn't know about emerging and cutting
> edge communications is in the wrong job or the wrong industry.
(Continue reading)

Frank Bulk | 5 Jul 01:51 2009
Picon

RE: Using twitter as an outage notification

So does twitter address the mass public, or those who are Web 2.0 literate
or techies?  I'm glad to see that Dell reached two million people, but how
many more people call in or visit its web page every day?

My point in another fork of this thread is that for most people, the
traditional forms of communication are *it*.  I'm not saying that twitter
hasn't been used and found a way to reach the some portion of the population
-- the traditional methods (announcement at top of phone tree & note on
homepage) should be maintained and as one more additional way to
communication.  I think you mentioned that yourself a few posts ago. =)

Frank

-----Original Message-----
From: JC Dill [mailto:jcdill.lists <at> gmail.com] 
Sent: Saturday, July 04, 2009 5:20 PM
Cc: nanog <at> merit.edu
Subject: Re: Using twitter as an outage notification

Roland Perry wrote:
> In article <4A4F6EF5.9030502 <at> gmail.com>, JC Dill 
> <jcdill.lists <at> gmail.com> writes
>
>>> What I'm trying to anticipate is the objection to *also* posting to 
>>> Twitter, which might be raised on the grounds that it's too 
>>> "unofficial", or "unsupported" or something like that.
>
>> Anyone who makes that argument is just showing how little they know 
>> about Twitter.
>
(Continue reading)

Tomas L. Byrnes | 5 Jul 06:01 2009
Picon

RE: Using twitter as an outage notification


>-----Original Message-----
>From: Frank Bulk [mailto:frnkblk <at> iname.com]
>Sent: Saturday, July 04, 2009 4:51 PM
>To: 'JC Dill'
>Cc: nanog <at> merit.edu
>Subject: RE: Using twitter as an outage notification
>
>So does twitter address the mass public, 

[TLB:] 
The whole point of Twitter is that it works with SMS.

>
>My point in another fork of this thread is that for most people, the
>traditional forms of communication are *it*.  I'm not saying that
>twitter
>hasn't been used and found a way to reach the some portion of the
>population
>-- the traditional methods (announcement at top of phone tree & note on
>homepage) should be maintained and as one more additional way to
>communication.  I think you mentioned that yourself a few posts ago. =)
>
>Frank
>
[TLB:] I think everyone agrees it's not an "Either/Or". 

The argument that Twitter is a good, inexpensive, way to mass
communicate operational issues with those that are able to use it, is
kind of axiomatic.
(Continue reading)

Roland Perry | 5 Jul 12:23 2009

Re: Using twitter as an outage notification

In article <4A4FD58B.2000703 <at> gmail.com>, JC Dill 
<jcdill.lists <at> gmail.com> writes
>Even easier, you make an email address on your system that is an alias 
>to posterous.   So they send to "post <at> schoolsystem.edu" which .forwards 
>out to posterous, which posts to the school blog, myspace, facebook, 
>twitter,

It doesn't have any of those, that's the point really.

Is twitter the one I should get them started with first?

>Show them how a radio station can retweet the info

It's have to be automated as there are hundreds to do over a periods of 
a few tens of minutes (the schools don't generally announce they are 
closing until they see how many teachers made it to work, and that's not 
long before they have to open - students get marked down for being late, 
even in bad weather, so can't delay setting out from home; it's an 
interesting operational model.)

>and then announce "to get info on school closings, follow us on twitter 
>at...."

http://twitter.com/trentfmnews (but it's not exactly high traffic)

>and everyone can send the info TO the radio station and get the info 
>FROM the radio station quickly and easily.

The radio station would probably be overwhelmed if they got much more 
than one tweet per school.
(Continue reading)

Marshall Eubanks | 5 Jul 13:01 2009
Picon

Re: Using twitter as an outage notification


On Jul 5, 2009, at 6:23 AM, Roland Perry wrote:

> In article <4A4FD58B.2000703 <at> gmail.com>, JC Dill <jcdill.lists <at> gmail.com 
> > writes
>> Even easier, you make an email address on your system that is an  
>> alias to posterous.   So they send to "post <at> schoolsystem.edu"  
>> which .forwards out to posterous, which posts to the school blog,  
>> myspace, facebook, twitter,
>
> It doesn't have any of those, that's the point really.
>
> Is twitter the one I should get them started with first?

I would say this partially would depend on how and what you want to  
communicate. If there is just going to be
one pronouncement per day (the school is up / down / delayed), then  
facebook and / or myspace would suggest themselves.
They are to date free, and the students will know what they are. I  
would start with facebook.

If you look at the #AuthorizeNet situation, there was a lot of back  
and forth. Will the schools have a need for
back and forth ? If they do, then, yes, twitter might be part of the  
solution and you might start with it. It's free, cross-platform, and  
you can also assume that the students (if not their parents) know what  
it is. This might also be a good for teachers and
the school to communicate, say by DM (direct messages).

Note that this will take people answering questions / dealing with  
(Continue reading)

Roland Perry | 5 Jul 15:03 2009

Re: Using twitter as an outage notification

In article <0D357934-85DE-4935-8F58-02F5FCC1DC8A <at> americafree.tv>, 
Marshall Eubanks <tme <at> americafree.tv> writes

>I would say this partially would depend on how and what you want to 
>communicate. If there is just going to be
>one pronouncement per day (the school is up / down / delayed), then 
>facebook and / or myspace would suggest themselves.

There's going to be a handful a year. Such as "school closed today due 
to snow". or "remember - school closed today for staff training" [a 
curious British phenomenon].

>They are to date free, and the students will know what they are. I 
>would start with facebook.
>
>If you look at the #AuthorizeNet situation, there was a lot of back and 
>forth. Will the schools have a need for
>back and forth ?

No, if the school's closed, it's closed. No debate allowed.

>Note that this will take people answering questions / dealing with 
>issues on twitter. Specifically, someone would have to  pay attention 
>to it during any quasi-emergency period - do the schools have such a 
>person ?

Such a person could be designated.

>Also, if the school looses power in a storm,

(Continue reading)

Aleksandr Milewski | 4 Jul 23:10 2009

Re: Using twitter as an outage notification

On 7/4/09 7:50 AM, Roland Perry wrote:

> What I'm trying to anticipate is the objection to *also* posting to
> Twitter, which might be raised on the grounds that it's too
> "unofficial", or "unsupported" or something like that.

Anecdotal, of course, but I found twitter to be very useful during the 
SF Bay Area fiber cuts a few months back. I was able to fairly quickly 
get reports of who was down (UnitedLayer) and who wasn't (everyone 
else), and made some good contacts, some of whom I've done business with 
since (Cernio).

Set up a twitter account for outage/event notifications, and don't 
*ever* use it for marketing.

Martin List-Petersen | 5 Jul 14:59 2009
Picon

Re: Using twitter as an outage notification

Aleksandr Milewski wrote:
> On 7/4/09 7:50 AM, Roland Perry wrote:
> 
>> What I'm trying to anticipate is the objection to *also* posting to
>> Twitter, which might be raised on the grounds that it's too
>> "unofficial", or "unsupported" or something like that.
> 
> Anecdotal, of course, but I found twitter to be very useful during the
> SF Bay Area fiber cuts a few months back. I was able to fairly quickly
> get reports of who was down (UnitedLayer) and who wasn't (everyone
> else), and made some good contacts, some of whom I've done business with
> since (Cernio).
> 
> Set up a twitter account for outage/event notifications, and don't
> *ever* use it for marketing.
> 

I'd agree on this one.

We use it for outage/event/coverage expansion notifications.

Originally, we thought a blog style website somewhere outside our
network was the way to go, but twitter has so many more angles, like RSS
feed capability, an API to integrate it somewhere on your website and
mobile clients.
On top of that, you can update it via SMS if needed.

The hype some people are pushing twitter on, I can't follow, but for
those type of notifications, it's perfect, also because it's not part of
your own infrastructure.
(Continue reading)

Roland Perry | 5 Jul 15:21 2009

Re: Using twitter as an outage notification

In article <4A50A3C9.3080009 <at> airwire.ie>, Martin List-Petersen 
<martin <at> airwire.ie> writes

>for those type of notifications, it's perfect, also because it's not 
>part of your own infrastructure.

 From an operational resilience point of view, that's a very important 
feature.
--

-- 
Roland Perry

Martin List-Petersen | 5 Jul 15:37 2009
Picon

Re: Using twitter as an outage notification

Roland Perry wrote:
> In article <4A50A3C9.3080009 <at> airwire.ie>, Martin List-Petersen
> <martin <at> airwire.ie> writes
> 
>> for those type of notifications, it's perfect, also because it's not
>> part of your own infrastructure.
> 
> From an operational resilience point of view, that's a very important
> feature.

It's the main reason for choosing something like twitter, blogspot etc.
If you want to communicate an outage, it might be as bad as your
infrastructure is gone, even though that you'd might hope, that you've
designed your network in a way, that it never happens.

But let's just take the scenario, where some event basically whipes your
ASN of the face of global BGP :) . It doesn't have to be a physical
outage, that causes it.

Talking about monetizing twitter, there's a very simple approach, just
based on this type of service:

Service Providers, Carriers etc., that use Twitter can pay a monthly fee
for the service and twitter sends them responses, private messages etc.
by more organized means.

Just my 2c on another approach, but I can see that happening and I
wouldn't mind paying a few bob for the service.

As for some responses on this tread and also some reactions from a few
(Continue reading)

Roland Perry | 5 Jul 16:15 2009

Re: Using twitter as an outage notification

In article <4A50ACB7.6070901 <at> airwire.ie>, Martin List-Petersen 
<martin <at> airwire.ie> writes
>Calling it a lame web 2.0 is pretty much off, when it's actually used
>for something sensible.

I seem to be trying to find the middle ground between members of the 
public who think "The Internet isn't appropriate because they didn't 
teach it to me in college 20 years ago" and those who say "Web 2.0 isn't 
appropriate because they didn't teach it to me in college 5 years ago".

Shouldn't we at least be giving it the benefit of the doubt?
--

-- 
Roland Perry

Martin List-Petersen | 5 Jul 16:41 2009
Picon

Re: Using twitter as an outage notification

Roland Perry wrote:
> In article <4A50ACB7.6070901 <at> airwire.ie>, Martin List-Petersen
> <martin <at> airwire.ie> writes
>> Calling it a lame web 2.0 is pretty much off, when it's actually used
>> for something sensible.
> 
> I seem to be trying to find the middle ground between members of the
> public who think "The Internet isn't appropriate because they didn't
> teach it to me in college 20 years ago" and those who say "Web 2.0 isn't
> appropriate because they didn't teach it to me in college 5 years ago".
> 
> Shouldn't we at least be giving it the benefit of the doubt?

Since when has, what has been teached in college ever been a defining
standard for what is happening on the internet or what the trend in
computing is ?

A lot of people never touch Linux during studies, and don't get any of
it in college, however are faced with it in the corporate or public world.

Kind regards,
Martin List-Petersen
--

-- 
Airwire - Ag Nascadh Pobail an Iarthair
http://www.airwire.ie
Phone: 091-865 968

Roland Perry | 5 Jul 18:57 2009

Re: Using twitter as an outage notification

In article <4A50BB87.8000301 <at> airwire.ie>, Martin List-Petersen 
<martin <at> airwire.ie> writes

>>> Calling it a lame web 2.0 is pretty much off, when it's actually used
>>> for something sensible.
>>
>> I seem to be trying to find the middle ground between members of the
>> public who think "The Internet isn't appropriate because they didn't
>> teach it to me in college 20 years ago" and those who say "Web 2.0 isn't
>> appropriate because they didn't teach it to me in college 5 years ago".
>>
>> Shouldn't we at least be giving it the benefit of the doubt?
>
>Since when has, what has been teached in college ever been a defining
>standard for what is happening on the internet or what the trend in
>computing is ?

It shouldn't be, but I'm guessing this is where much of the conservatism 
is coming from.

--

-- 
Roland Perry

Neil | 5 Jul 22:50 2009
Picon

Re: Using twitter as an outage notification

On Sun, Jul 5, 2009 at 7:15 AM, Roland Perry <lists <at> internetpolicyagency.com
> wrote:

> In article <4A50ACB7.6070901 <at> airwire.ie>, Martin List-Petersen <
> martin <at> airwire.ie> writes
>
>> Calling it a lame web 2.0 is pretty much off, when it's actually used
>> for something sensible.
>>
>
> I seem to be trying to find the middle ground between members of the public
> who think "The Internet isn't appropriate because they didn't teach it to me
> in college 20 years ago" and those who say "Web 2.0 isn't appropriate
> because they didn't teach it to me in college 5 years ago".
>
> Shouldn't we at least be giving it the benefit of the doubt?

Well, I'm no social media expert, and I don't spend a whole lot of time on
any of the social networking sites (I particularly dislike Facebook,
actually). (And yet, I'm probably about as qualified for the SME title as
90% of those who claim to be...)

However, I was a student fairly recently, and so maybe my perspective will
hold some value.

I really like the Posterous+Twitter+Facebook+etc. combo. To manage the Fb
side, you could probably tap a trusted student to make the School an Fb
page. A lot of the students will check there. Parents will probably check
the Posterous or Twitter pages. Some of the more tech-savvy students and
parents will sign up for Twitter and get SMS notifications.
(Continue reading)

Warren Bailey | 4 Jul 23:16 2009

Re: Using twitter as an outage notification

Why aren't you all out getting drunk like me?? ;)

----- Original Message -----
From: Mark E. Mallett <mem <at> mv.mv.com>
To: Frank Bulk <frnkblk <at> iname.com>
Cc: nanog <at> merit.edu <nanog <at> merit.edu>
Sent: Sat Jul 04 13:12:14 2009
Subject: Re: Using twitter as an outage notification

On Sat, Jul 04, 2009 at 03:59:48PM -0500, Frank Bulk wrote:
> When the local power companies uses twitter, then maybe I'll consider using
> twitter for our customers.

During the ice storm we had here last winter, the local power
company did just that.  "psnh" "ice storm" "twitter" etc are all
good search terms.

I haven't been following this thread closely, so I dunno if that had
already been mentioned and your remark was a sardonic one. Nevertheless..

-mm-  (probably due for an annual nanog posting)

Skywing | 5 Jul 18:20 2009

RE: Using twitter as an outage notification

Hmm... doesn't that kind of defeat the point of using Twitter instead of your own infrastructure to begin
with, aside from adding another (Posterous) single point of failure for all your communication mechanisms?

Perhaps it is not so important for snow days vs. outage situations, but it seems to me like it would be simpler
and more reliable to go directly to the source and not use Posterous.

(Besides, I suspect the chances are reasonable that between mail/www/Twitter, you're going to have a low
set of users in the other social networking sites crowd who don't have any overlap to begin with. 
Diminishing returns?)

- S

-----Original Message-----
From: JC Dill <jcdill.lists <at> gmail.com>
Sent: Sunday, July 05, 2009 08:18
Cc: nanog <at> merit.edu <nanog <at> merit.edu>
Subject: Re: Using twitter as an outage notification

Roland Perry wrote:
>> There's the temptation by some of companies to leverage the latest
>> technology to appear "cool" and "in tune" with customers, but by far and
>> large, when something goes down customers either do no nothing, wait, or
>> call in.  I think the best use of everyone's time is to make sure
>> their call
>> center/support desk has the capability to post an announcement to
>> those that
>> call in.
>
> It's a High School. They don't have a "support desk" (or more than
> handful of phone lines [1]). Even the local radio station can't cope
(Continue reading)

Benjamin Billon | 5 Jul 18:59 2009
Picon

Re: Using twitter as an outage notification

I agree.
It seems (I didn't look for solid proofs of that) twitter went down when 
MJ's die was revealed. I don't want to know why (not enough cloud 
computing stuff?), but I still believe there is maybe not always an 
ultimate solution to all problems.

Twitter and its friends may sometimes help, that's for sure. But at an 
higher level, we don't need the info right here right now, we only want 
the issues to be solved. Still meaning the DC/ISP/hosting company has to 
keep a straight and up-to-date list of customers in order to contact 
them directly if necessary (but this is not part of the problems' 
resolution, this is commercial/relational matter), which I never saw in 
real life.

Furthermore, I personnaly don't use Twitter and as few "social 
networking whatever" websites as I can.

Ben

Skywing a écrit :
> Hmm... doesn't that kind of defeat the point of using Twitter instead of your own infrastructure to begin
with, aside from adding another (Posterous) single point of failure for all your communication mechanisms?
>
> Perhaps it is not so important for snow days vs. outage situations, but it seems to me like it would be
simpler and more reliable to go directly to the source and not use Posterous.
>
> (Besides, I suspect the chances are reasonable that between mail/www/Twitter, you're going to have a low
set of users in the other social networking sites crowd who don't have any overlap to begin with. 
Diminishing returns?)
>
(Continue reading)

Michael J McCafferty | 3 Jul 23:22 2009

RE: Fire, Power loss at Fisher Plaza in Seattle

On Fri, 2009-07-03 at 13:21 -0700, Tomas L. Byrnes wrote:
> 
> >
> >Earth is a single point of failure, where is your backup site?
> 
> [TLB:] Given that all my customers are on Earth, I don't need one if my
> customers also are "down".

Bad Day !

Leo Bicknell | 3 Jul 22:39 2009

Re: Fire, Power loss at Fisher Plaza in Seattle

In a message written on Fri, Jul 03, 2009 at 03:22:14PM -0400, Sean Donelan wrote:
> Are you better off with a single "tier 4" data center, multiple
> "tier 1" data centers, or something in between?

It depends entirely on your dependency on connectivity.

One extreme is something like a Central Office.  Lots of cables
from end-sites terminate in the building.  Having a duplicate of
the head end termination equipment on the opposite coast is darn
near useless.  If the building goes down, the users going through
it go down.  "Tier 4" is probably a good idea.

The other extreme is a pure content play (YouTube, Google Search).
Users don't care which data center they hit (within reason), and
indeed often don't know.  You're better off having data centers
spread out all over, both so you're more likely to only loose one
at a time, but also so that the loss of one is relatively unimportant.
Once you're already in this architecture, Tier 1 is generally
cheaper.

There are two problems though.  First, most folks don't fit neatly
in one of these buckets.  They have some ties to local infrastructure,
and some items which are not tied.  Latency as a performance penality
is very subjective.  A backup 1000 miles away is fine for many
things, and very bad for some things.

Second, most folks don't have choices.  It would be nice if most
cities had three each Tier 1, 2, 3 and 4 data centers available so
there was choice and competition but that's rare.

(Continue reading)

Darren Bolding | 3 Jul 22:57 2009

Re: Fire, Power loss at Fisher Plaza in Seattle

Power to some of the affected sections of the building has been restored via
existing onsite generators.  The central power risers cannot be connected to
current generators in a timely manner due to excessive damage to the
electrical switching equipment (and those generators may still be in
standing water).  These provide power to a number of colocated systems.
 Temporary generators are on order to be connected to the central risers,
and the site expects that to be complete sometime late this evening.  As
best I can tell, there is still no utility power connected to any of the
systems.
The AC systems (chiller and crac) are currently not working.  It is not
clear to me whether these will be brought back on line when the temporary
generators are available, but I am assuming so.

It was pleasant to see the general positive attitude, sharing of information
and offers of assistance that were made by representatives of the various
tenants, customers and carriers that were on the scene.  The usual suspects
(companies and individuals) stepped up and took care of things, as they
always seem to.

--D

On Fri, Jul 3, 2009 at 1:39 PM, Leo Bicknell <bicknell <at> ufp.org> wrote:

> In a message written on Fri, Jul 03, 2009 at 03:22:14PM -0400, Sean Donelan
> wrote:
> > Are you better off with a single "tier 4" data center, multiple
> > "tier 1" data centers, or something in between?
>
> It depends entirely on your dependency on connectivity.
>
(Continue reading)

Frank Bulk | 4 Jul 22:43 2009
Picon

RE: Fire, Power loss at Fisher Plaza in Seattle

Leo:

That presumes the people at the bottom care about the big picture.  I'd
hazard to guess that the people who participate in this listserv *do* care,
but if Dilbert has even any correlation to real life, then just as often
there are people who build DNS/DHCP/etc redundantly because that's what they
were told to do, but don't have the initiative or weren't directed to look
at the bigger picture.  At the end of the day it depends on these grunts'
managers to be sure that things are being designed and implemented properly.
And even then, we know how personalities, resources, and internal inertia
more of than not develop and grow networks that aren't BCP-friendly.

Frank

-----Original Message-----
From: Leo Bicknell [mailto:bicknell <at> ufp.org] 
Sent: Friday, July 03, 2009 3:40 PM
To: NANOG list
Subject: Re: Fire, Power loss at Fisher Plaza in Seattle

<snip>

Very few companies consider these choices rationally; often because
choices are made by different groups.  I am amazed how many times
inside of an ISP the folks deploying the DNS and mail servers are
firewalled from the folks deploying the network, to the point where
you have to get to the President to reach common management.  This
leads to them making choices in opposite directions that end up
costing extra money the company, and often resulting in a much lower
uptimes than expected.  Having the network group deploy a single point
(Continue reading)

Michael K. Smith - Adhost | 3 Jul 19:57 2009

RE: Fire, Power loss at Fisher Plaza in Seattle


-----Original Message-----
From: Tomas L. Byrnes [mailto:tomb <at> byrneit.net]
Sent: Fri 7/3/2009 10:20 AM
To: David Hubbard; NANOG list
Subject: RE: Fire, Power loss at Fisher Plaza in Seattle

This begs the question of what basic parameters should be for a "carrier
hotel" or co-lo.

Given that we're getting designated "Critical Infrastructure", we'd
getter start coming up with some, or we'll have them defined for us.

----

I think the more important question is, "what do you consider redundancy?"  We have facilities in Plaza East
(no down) and Plaza West (unaffected).  If you are critical infrastructure there is no amount of
redundancy that you should offload onto a colo provider.  Instead, you build your redundancy across
different data centers, different providers, different everything.  If you rely on a single provider for
any of the aforementioned then you have built in at least one single point of failure, regardless of the
resiliency of the underlying provider.

My .02, worth almost every penny.

Mike

Erik Soosalu | 3 Jul 20:13 2009

RE: Fire, Power loss at Fisher Plaza in Seattle

http://seattletimes.nwsource.com/html/localnews/2009415571_apwafisherpla
zafire1stldwritethru.html

-----Original Message-----
From: David Hubbard [mailto:dhubbard <at> dino.hostasaurus.com] 
Sent: Friday, July 03, 2009 1:05 PM
To: NANOG list
Subject: RE: Fire, Power loss at Fisher Plaza in Seattle

From: Seth Mattinen [mailto:sethm <at> rollernet.us] 
> 
> Apparently it took authorize.net with it, too:
> http://twitter.com/authorizenet
> 
> ~Seth

No technical explanation of course but it also took down
their 'backup facility' according to them on twitter;
I assume some bad routing/DNS if they do actually have
a backup facility.  Lots of online stores are
offline right now because of this, and the holiday is
unfortunately keeping those store owners from knowing
they are not making sales right now.  Life in ecommerce...

David


Gmane