Chris Adams | 17 Dec 20:02 2012
Picon

DNS ANY requests from Amazon?

I'm seeing a bunch of DNS ANY requests to my authoritative servers with
Amazon EC2 source IPs.  I guess somebody is now trying to run an
amplification attack against Amazon?

This is the first time I've seen Amazon targeted this way; are others
seeing this (am I just late to the party)?

--

-- 
Chris Adams <cmadams@...>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
Colm MacCárthaigh | 17 Dec 20:08 2012
Picon

Re: DNS ANY requests from Amazon?

Hi Chris,

Can you forward me any specific IPs, logs, traffic captures, or any
other data you have off list? I'll get it to the right Amazon people
to investigate.

On Mon, Dec 17, 2012 at 11:02 AM, Chris Adams <cmadams@...> wrote:
> I'm seeing a bunch of DNS ANY requests to my authoritative servers with
> Amazon EC2 source IPs.  I guess somebody is now trying to run an
> amplification attack against Amazon?
>
> This is the first time I've seen Amazon targeted this way; are others
> seeing this (am I just late to the party)?
>
> --
> Chris Adams <cmadams@...>
> Systems and Network Administrator - HiWAAY Internet Services
> I don't speak for anybody but myself - that's enough trouble.
> _______________________________________________
> dns-operations mailing list
> dns-operations@...
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
> dns-jobs mailing list
> https://lists.dns-oarc.net/mailman/listinfo/dns-jobs

--

-- 
Colm
Paul Vixie | 17 Dec 20:09 2012

Re: DNS ANY requests from Amazon?

On 2012-12-17 7:02 PM, Chris Adams wrote:
> I'm seeing a bunch of DNS ANY requests to my authoritative servers with
> Amazon EC2 source IPs.  I guess somebody is now trying to run an
> amplification attack against Amazon?
>
> This is the first time I've seen Amazon targeted this way; are others
> seeing this (am I just late to the party)?

it's happened before. can you share the addresses? and are you using
some form of ratelimiting?

paul

--

-- 
"It seems like the rules for automagic completion of incomplete names typed into browsers are going to
start to look like those for the game of fizbin." --rick jones

sthaug | 17 Dec 20:27 2012
Picon

Re: DNS ANY requests from Amazon?

> I'm seeing a bunch of DNS ANY requests to my authoritative servers with
> Amazon EC2 source IPs.  I guess somebody is now trying to run an
> amplification attack against Amazon?

Highly likely.

> This is the first time I've seen Amazon targeted this way; are others
> seeing this (am I just late to the party)?

You're just late to the party. This has been going on for months.

Steinar Haug, AS 2116
Patrick, Robert (CONTR | 17 Dec 20:57 2012

Re: DNS ANY requests from Amazon?

Chris,

Yes, many sites are seeing increasing "background noise" from Internet hosts repetitively submitting
DNS queries, especially for ANY.  Amplification attacks, or simply burning CPU cycles.

It's starting to look like per-client-IP rate-limiting features are necessary, with intelligent
defaults, to ensure applications facing the Internet are protected out-of-the-box, while service
providers and others with IT staff can adjust the settings where necessary.  The current default settings
for most applications to provide unlimited response to any IP address, especially for non-stateful
protocols (e.g. UDP), is proving to be noisy.

Where some customers haven't implemented rate-limiting within BIND, mitigation is available at the O/S
and network layer.  As an example, there are connection limits that can be enforced with iptables on Linux. 
Per-source-IP connection limits can also be restricted on Cisco ASA firewalls (and likely other vendor products).

There is a patch available for rate-limiting inside BIND.

-----Original Message-----
From: dns-operations-bounces@...
[mailto:dns-operations-bounces@...] On Behalf
Of sthaug@...
Sent: Monday, December 17, 2012 2:27 PM
To: cmadams@...
Cc: dns-operations@...
Subject: Re: [dns-operations] DNS ANY requests from Amazon?

> I'm seeing a bunch of DNS ANY requests to my authoritative servers 
> with Amazon EC2 source IPs.  I guess somebody is now trying to run an 
> amplification attack against Amazon?

(Continue reading)

Paul Vixie | 17 Dec 21:17 2012

Re: DNS ANY requests from Amazon?

On 2012-12-17 7:57 PM, Patrick, Robert (CONTR) wrote:
> ...
>
> Where some customers haven't implemented rate-limiting within BIND, mitigation is available at the O/S
and network layer.  As an example, there are connection limits that can be enforced with iptables on Linux. 
Per-source-IP connection limits can also be restricted on Cisco ASA firewalls (and likely other vendor products).

such rate limits are too coarse-grained for dns authority service. if
you limit your request flows rather than your response flows, then your
only choice is: too low, where a legitimate client asking a legitimately
diverse set of questions, does not get reliable service; or, too high,
where an attacker can get enough of your bandwidth directed at a victim
to be damaging.

OS-level rate limiting also lacks the ability to insert TC=1 responses
on a statistical basis, thus transforming rate limiting into transaction
delay rather than transaction loss.

to make this work without breaking things, the rate limiting logic has
to be within the server itself, and it has to be applied to responses
not requests.

> There is a patch available for rate-limiting inside BIND.

see http://www.redbarn.org/dns/ratelimits for background, including
patches (which are not currently supported by ISC) and a technical note
(which looks a bit like an RFC that some day i hope RRL will deserve.)

paul
(Continue reading)

Patrick, Robert (CONTR | 17 Dec 22:33 2012

Re: DNS ANY requests from Amazon?

I don't disagree that limiting responses is a smarter tact than limiting requests, with respect to making
an informed decision prior to discarding traffic.  Having zone and query-type plus response data to
evaluate the client hash is more information than looking only at source and destination IP address, as
may be implemented at a firewall or within the O/S.  Some of these data elements could also be tracked by an
application-aware firewall.

For authoritative DNS servers, there's an approach to rate-limiting per-client
request-count-over-time (or response-count-over-time) above the current legitimate client volume
and below infinite that is useful to reduce the volume of large amplification attacks.  The specific value
can be argued as site -specific, but odds are good there's a number that remains somewhat high which should
prove "good enough" as a default for the much of the installed base.

Allow administrators the freedom to set the limit to any value and/or disable the feature, but shipping the
product with a "smart" default may be viewed as a pragmatic step forward in noise reduction.

Continuing to deploy into the wild without any rate-limiting isn't the best approach long term.

Administrators need to be armed with options to implement countermeasures.  The more accurate the
countermeasure, the better.

-----Original Message-----
From: Paul Vixie [mailto:paul@...] 
Sent: Monday, December 17, 2012 3:17 PM
To: Patrick, Robert (CONTR)
Cc: dns-operations@...
Subject: Re: [dns-operations] DNS ANY requests from Amazon?

On 2012-12-17 7:57 PM, Patrick, Robert (CONTR) wrote:
> ...
>
(Continue reading)

Vernon Schryver | 17 Dec 23:44 2012

Re: DNS ANY requests from Amazon?

> From: "Patrick, Robert (CONTR)" <Robert.Patrick@...>

> I don't disagree that limiting responses is a smarter tact than
> limiting requests, with respect to making an informed decision prior
> to discarding traffic.  Having zone and query-type plus response
> data to evaluate the client hash is more information than looking
> only at source and destination IP address, as may be implemented
> at a firewall or within the O/S.  Some of these data elements could
> also be tracked by an application-aware firewall.

Yes, you could do response rate limiting (RRL) within an application
aware firewall by have the firewall do almost of all of the work
of your DNS server.  For example, your RRL mechanism (whether in a
firewall or DNS server) must count all NXDOMAIN responses to a given
IP address as identical to avoid spewing GBytes/sec of big signed NXDOMAIN
responses about distinct random, invalid domains.
`dig +dnssec asdf1234asdf.com  <at> a.gtld-servers.net` gives a 1K NXDOMAIN.
Referrals have a similar issue.

A firewall that is as DNS aware as that should not waste the computing
it has done to know whether to discard the response it computed to
count.  If things are ok, it should go ahead and send the response.

Never mind that consistency, maintenance, and other problems that
always come with duplicating things, whether definitions of constants
in code or the big chunks of code and data that are a modern DNS server.

> ...
> Allow administrators the freedom to set the limit to any value and/or
> disable the feature, but shipping the product with a "smart" default
(Continue reading)

Dobbins, Roland | 18 Dec 10:12 2012
Picon

Re: DNS ANY requests from Amazon?


On Dec 18, 2012, at 5:44 AM, Vernon Schryver wrote:

> Yes, you could do response rate limiting (RRL) within an application aware firewall by have the firewall
do almost of all of the work of your DNS server. 

The 'application-aware firewall' will collapse from state-table exhaustion, however, so this likely
isn't a very good idea.

;>

-----------------------------------------------------------------------
Roland Dobbins <rdobbins@...> // <http://www.arbornetworks.com>

	  Luck is the residue of opportunity and design.

		       -- John Milton

Paul Vixie | 18 Dec 16:30 2012

Re: DNS ANY requests from Amazon?

On 12/18/2012 9:12 AM, Dobbins, Roland wrote:
> On Dec 18, 2012, at 5:44 AM, Vernon Schryver wrote:
>
>> Yes, you could do response rate limiting (RRL) within an application aware firewall by have the firewall
do almost of all of the work of your DNS server. 
> The 'application-aware firewall' will collapse from state-table exhaustion, however, so this likely
isn't a very good idea.

i don't think that follows. RRL is designed in a way that keeps state
manageably finite. in speaking to the cloudshield folks and learning
more about "packetC" i think RRL can be done as part of a really smart
front end firewall.

paul
Vernon Schryver | 18 Dec 17:22 2012

Re: DNS ANY requests from Amazon?

> From: Paul Vixie <paul@...>

> On 12/18/2012 9:12 AM, Dobbins, Roland wrote:

> > The 'application-aware firewall' will collapse from state-table
> > exhaustion, however, so this likely isn't a very good idea.
>
> i don't think that follows. RRL is designed in a way that keeps state
> manageably finite.

State tables for some application-aware filtering can be smaller than
for filtering ICMP Echo-requests, TCP SYNs, or other trivial "applications".
If your application falls over at X requests/second, then it might not
matter that your firewall state table is exhaustd at 2X requests/second.

The BIND RRL state table design limit is supposed to be above the
limit for the DNS server itself, but that's an untested regime.
I think the recent NSD RRL code answered that issue by using a
large, fixed size hash table and tolerating false positives.

>                    in speaking to the cloudshield folks and learning
> more about "packetC" i think RRL can be done as part of a really smart
> front end firewall.

How would it filter requests for <random>.example.com without resolving
<random>.example.com to know if the response would be an NXDOMAIN
or wildcard answer the same as too many others?
You could assume that example.com has at most N subdomains, and that
all <random>.example.com after the first (or recent) N would get
NXDOMAIN, wildcard, or referral answers from the server behind the
(Continue reading)

Dobbins, Roland | 18 Dec 21:32 2012
Picon

Re: DNS ANY requests from Amazon?


On Dec 18, 2012, at 10:30 PM, Paul Vixie wrote:

> RRL is designed in a way that keeps state manageably finite. 

Sure, but RRL isn't the issue; it's all the rest of what 'application firewalls' do which causes them to
choke.  I've yet to see one which doesn't choke under even moderate DDoS, and have never seen one which
implements any form of classification in a stateless or minimized-state manner.

-----------------------------------------------------------------------
Roland Dobbins <rdobbins@...> // <http://www.arbornetworks.com>

	  Luck is the residue of opportunity and design.

		       -- John Milton

Vernon Schryver | 18 Dec 22:37 2012

Re: DNS ANY requests from Amazon?

> From: "Dobbins, Roland" <rdobbins@...>

> Sure, but RRL isn't the issue; it's all the rest of what 'application
> firewalls' do which causes them to choke.  I've yet to see one which
> doesn't choke under even moderate DDoS, and have never seen one which
> implements any form of classification in a stateless or minimized-state
> manner.

It's well known that Roland Dobbins doesn't think much of application
firewalls or stateful firewalls in general.  I also don't think much
of application firewalls, and not only because the FUD that is much
of their brochures, the never ending broken vendor promises, or the
exaggerated performace.  I've been grumbling since tcp wrappers first
appeared that application firewalls are usually poor bandaids for
stupid application security holes that could (and should) be more
securely and cheaply fixed in the applications.

But all of those criticisms are irrelevant to what hypothetical firewalls
might do for current and foreseeable DNS security issues.  That currently
popular firewalls can't cope or do only stupid stuff like ANY filtering
doesn't justify rejecting firewalls for reflection attacks on principle.

Besides, DoS attacks on DNS servers themselves (as opposed to using
DNS servers to attack others) are best handled outside in smart (e.g.
sane state table management) application firewalls.  It's not good for
a DNS server to discard excessive (relative to the server's own
resources) requests.  By the time a request can be discarded by the
server, too many local resources have been burned.

Vernon Schryver    vjs@...
(Continue reading)

Dobbins, Roland | 18 Dec 23:12 2012
Picon

Re: DNS ANY requests from Amazon?


On Dec 19, 2012, at 4:37 AM, Vernon Schryver wrote:

> Besides, DoS attacks on DNS servers themselves (as opposed to using DNS servers to attack others) are best
handled outside in smart (e.g. sane state table management) application firewalls.  

This seem to be an issue of semantics - I certainly agree that there are better solutions for dealing with DNS
DDoS attacks than every-server-for-itself.

[Full disclosure: I work for a vendor of such solutions.]

-----------------------------------------------------------------------
Roland Dobbins <rdobbins@...> // <http://www.arbornetworks.com>

	  Luck is the residue of opportunity and design.

		       -- John Milton

Stephane Bortzmeyer | 18 Dec 08:55 2012
Picon

Re: DNS ANY requests from Amazon?

On Mon, Dec 17, 2012 at 08:17:18PM +0000,
 Paul Vixie <paul@...> wrote 
 a message of 33 lines which said:

> if you limit your request flows rather than your response flows,
> then your only choice is: too low, where a legitimate client asking
> a legitimately diverse set of questions, does not get reliable
> service;

In theory, you're right. In practice, the attacks of *today* are quite
simple and quite separate from normal DNS traffic (nobody asks "ANY
isc.org" in the real world, except the attackers).

I appreciate the BIND RRL patch and it is obvious to me that we must
continue the research in dDoS mitigation, but let's not drop the
mitigations techniques that work *today*. (The attackers are not
superhuman, they use imperfect techniques.)

> OS-level rate limiting also lacks the ability to insert TC=1
> responses on a statistical basis, thus transforming rate limiting
> into transaction delay rather than transaction loss.

Yes.

> see http://www.redbarn.org/dns/ratelimits for background, including
> patches (which are not currently supported by ISC)

In actual deployments, some people may be unwilling or unauthorized
(corporate policy) to install "unofficial" patches on a production
server. That's why we should not reject blindly the OS-level rate
(Continue reading)

Paul Vixie | 18 Dec 16:25 2012

Re: DNS ANY requests from Amazon?

On 12/18/2012 7:55 AM, Stephane Bortzmeyer wrote:
> On Mon, Dec 17, 2012 at 08:17:18PM +0000,
>  Paul Vixie <paul@...> wrote 
>  a message of 33 lines which said:
>
>> if you limit your request flows rather than your response flows,
>> then your only choice is: too low, where a legitimate client asking
>> a legitimately diverse set of questions, does not get reliable
>> service;
> In theory, you're right. In practice, the attacks of *today* are quite
> simple and quite separate from normal DNS traffic (nobody asks "ANY
> isc.org" in the real world, except the attackers).

any time spent matching on things like bufsize=9000 is worse than
wasted. even the lowest quality attacker can change it to 9001 at the
start of a long holiday weekend. my rule of thumb is, don't install
stuff that's not worth significant lab time up front. your attackers can
adapt; so must your defense.

> I appreciate the BIND RRL patch and it is obvious to me that we must
> continue the research in dDoS mitigation, but let's not drop the
> mitigations techniques that work *today*. (The attackers are not
> superhuman, they use imperfect techniques.)

when i said that setting the per-requestor quota high enough to avoid
false positives would give attackers enough capacity to cause real
injury, i'm speaking from direct experience with f-root. believe me when
i tell you, if we could solve this in the kernel, without a process
context switch, without a user mode data copy... we would. that is,
*today* we have attackers who can adapt to per-requestor quotas who have
(Continue reading)

Matt Rowley | 18 Dec 19:59 2012
Picon

Re: DNS ANY requests from Amazon?


On Dec 17, 2012, at 3:17 PM, Paul Vixie wrote:

>> There is a patch available for rate-limiting inside BIND.
> 
> see http://www.redbarn.org/dns/ratelimits for background, including
> patches (which are not currently supported by ISC) and a technical note
> (which looks a bit like an RFC that some day i hope RRL will deserve.)

For what it's worth, ARIN also came under an amplification attack recently.  This was early last month.  They
were querying the heck out of ripe.net for which we provide secondary service.  It's a nice, signed zone
that's chunky on the outbound.
We were able to completely mitigate the attack using Schryver & Vixie's ratelimiter BIND patch.  It's
working quite well for us.

cheers,
Matt
Vernon Schryver | 17 Dec 22:08 2012

Re: DNS ANY requests from Amazon?

> It's starting to look like per-client-IP rate-limiting features
> are necessary...

> There is a patch available for rate-limiting inside BIND.

There is also RRL code for NSD.

Please note that the main thrust of the BIND and NSD rate limiting
code is response rate limiting (RRL) and *NOT* per-client IP address
rate limiting.  Per-client rate limiting is generally the best that
can be done with simple firewall rules or access control lists, but
has limitations and can cause harm.  While rate limiting by client IP
address stops a reflection attack, it also turns off almost all DNS
service to the client from the server.  Temporarily denying name service
to a target has long been a major part of more serious security problems
than denials of service.  For example, if you need to fool your target
about the IP address of www.example.com, it's handy to have the several
seconds of a full set of DNS client timeouts to try many DNS transaction
IDs instead only the milliseconds before the real answer arrives.

With RRL (especially with the "slip" feature), the victim of a reflection
attack often sees no change in DNS services from the rate limiting
server during a reflection attack.  With client IP address rate limiting,
the server stops answering practically all requests from the victim.

The current version the BIND RRL patch does have support for
per-client rate limiting, but it exists only to satisfy popular
demand.  Its use is a bad idea in most cases.

I've said something like this before but I keep seeing claims that
(Continue reading)

Stephane Bortzmeyer | 18 Dec 08:57 2012
Picon

Re: DNS ANY requests from Amazon?

On Mon, Dec 17, 2012 at 09:08:09PM +0000,
 Vernon Schryver <vjs@...> wrote 
 a message of 47 lines which said:

> Per-client rate limiting is generally the best that can be done with
> simple firewall rules or access control lists, but has limitations
> and can cause harm.  While rate limiting by client IP address stops
> a reflection attack, it also turns off almost all DNS service to the
> client from the server.

No one in his right mind limits simply by the client's IP
address. People typically also use the type of the request (today,
typically ANY). See my mini-HOWTO for Linux Netfilter in this thread.

Lutz Donnerhacke | 18 Dec 16:29 2012
Picon

Re: DNS ANY requests from Amazon?

* Stephane Bortzmeyer wrote:
> No one in his right mind limits simply by the client's IP address.

Interesting point. Nice to read.
Vernon Schryver | 18 Dec 17:02 2012

Re: DNS ANY requests from Amazon?

> From: Stephane Bortzmeyer <bortzmeyer@...>

> >                      While rate limiting by client IP address stops
> > a reflection attack, it also turns off almost all DNS service to the
> > client from the server.
>
> No one in his right mind limits simply by the client's IP
> address. People typically also use the type of the request (today,
> typically ANY). See my mini-HOWTO for Linux Netfilter in this thread.

That tactic makes limited sense if you assume that the bad guys are
too stupid to see that they can bypass ANY filters with almost as
much amplification with other query types such as A.

`dig +dnssec www.nic.fr  <at> ns1.nic.fr` offers amplification of more than 25X.

 +++++

] but my point is that it works *today*, with *actual* attacks. So, it
] definitely helps but keep your eyes open, have alternative solutions
] in place and do not put all your eggs in one basket

Yes, automated response rate limiting (RRL) is too small a basket
to hold all of anyone's eggs.  However, a static ANY filter amounts
to trying to keep all of your eggs in a handkerchief.  Manually
maintained iptable rules are akin to hiring jugglers keep all of
your eggs in the air.

>                                                   (nobody asks "ANY
> isc.org" in the real world, except the attackers).
(Continue reading)

Stephane Bortzmeyer | 18 Dec 08:51 2012
Picon

Re: DNS ANY requests from Amazon?

On Mon, Dec 17, 2012 at 02:57:28PM -0500,
 Patrick, Robert (CONTR) <Robert.Patrick@...> wrote 
 a message of 36 lines which said:

> mitigation is available at the O/S and network layer.  As an
> example, there are connection limits that can be enforced with
> iptables on Linux.  

The attached mini-HOWTO may be interesting to some.

[Not public]

If you have a DNS server used for reflection+amplification attack
*and* it is a Linux machine *and* you have Netfilter >= 1.4 *and* you
cannot or does not want to install the patches for BIND or NSD to do
rate-limiting (they may provide a better result) *and* the attack is
over IPv4 *and* the attacker uses only a few domain names, you could
be interested in this technique. Disclaimer: it works for us, it will
not work for ever, it works now.

The idea is to use the Netfilter u32 module to recognize the attack,
then to rate-limit it with the Netfilter hashlimit module.

First, get the iptables rules generation script
<http://www.bortzmeyer.org/files/generate-netfilter-u32-dns-rule.py>.

Then, look at the traffic so see the pattern: what query type
(typically ANY), what query domain name, etc. In the examples, we'll
(Continue reading)

Stephane Bortzmeyer | 18 Dec 09:09 2012
Picon

Re: DNS ANY requests from Amazon?

On Tue, Dec 18, 2012 at 08:51:18AM +0100,
 Stephane Bortzmeyer <bortzmeyer@...> wrote 
 a message of 80 lines which said:

> [Not public]

Actually, it was the censored version, the not-public one has more
details, useful for the attacker.

Gmane