SYN FLOOD ATTACKS - Please help

[ppaik]

Need some assistance if anyone can offer any. One of our servers is currently under a syn flood dos attack. I installed network monitor to view the traffic and also checked netstat -a -n. I tried configuring my windows to protect against syn flood attacks link. This caused my system to hang so I blocked port 80 on the firewall and was able to undo the changes. I looked for more information on the net and found an article on tcp intercept on the router link which froze up the router.

We've been down for about 18hrs now, is there any information anyone can share with me to get this resolved?

I would greatly appreciate it.

-Phil

 

[spidey07]

In all honesty, I'd call your provider. Without sophisticated security gear it is hard to prevent. The TCP intercept method should work pretty well and not crash your router - unless the router is underpowered. What model and the size of the pipe?

Are all the sources good IP addresses or are they private (which should NEVER happen on the Internet)? Are they coming from inside or from the Internet If you can collect this info your provider should be able to stop it. But if you are suffering from a DDoS - the first step is to get them involved, they've got better tools.

If it is from a single IP or a known network block/range then just block them at the router/firewall and service everybody else. If it is a true DDoS/bot attack you'll have to work with your ISP.

What are your symptoms? Server running out of resources/connections or bandwidth/traffic overload? We need to find out #1 - is this internal or external.

 

[ppaik]

The IPs are public IPs however they are all spoofs. I've called my provider and they tell me only thing they can do is blackhole my ip address which basically means its no longer usable. The router is a catalyst 6500 and it freezes when i do the tcp intercept. My firewall seems to be overloaded at times too where i lose connection.

 

[spidey07]

what version of code and what switching operation/cards are in the 6509. That thing should not buckle for this. If you have old sups and no fabric module, I can see everything being process switched with ip intercept and causing you problems. I can't recall if that feature is in the ASICs or not. Probably processor.

shame on your provider for allowing spoofed IPs. Time to shop around. Tell them "I have a DDoS going on and I expect you to resolve it." If they can't, then switch. Seriously.

hook a console to the 6500 when you enable ip intercept, slow down/rate limit any logging you may be doing. maybe no logging console. see if you can get a "show proc cpu" from it.

Otherwise, call Cisco and declare a "network down emergency" They WILL work you through this.

 

[ppaik]

ok pretty exhausted tonight...been up over 24hrs...will try it tomorrow and let you know


thanks again for the help spidey

-Phil

 

[RebateMonger]

If you haven't seen it, here's how Steve Gibson (of SpinRite fame) handled a DDOS attack on his Servers.

GRC.COM DDOS attack story.

 

[Nothinman]

If you haven't seen it, here's how Steve Gibson (of SpinRite fame) handled a DDOS attack on his Servers.

He's also the one of the biggest handwavers and exaggerationists on the Internet today.

 

[InlineFive]

Yeah, the summary of Gibsons "article" is that he called Verio and they filtered the attack.

 

[spike spiegal]

shame on your provider for allowing spoofed IPs. Time to shop around. Tell them "I have a DDoS going on and I expect you to resolve it." If they can't, then switch. Seriously.

Yep.

shame on your provider for allowing spoofed IPs. Time to shop around. Tell them "I have a DDoS going on and I expect you to resolve it." If they can't, then switch. Seriously.

Double yep

 

[spidey07]

not to sound like a know it all jerk...

But DDoS can only truly be stopped in the ISP. The customer can't truly stop it.

So you put ACLs and firewalls in the customer premise?

So I just flood you with too much traffic and overload your router/firewall.

oh - and that GRC site is a joke. Any decent firewall would have stopped what he encountered. He's the laughing stock of the network/security community.

 

[ppaik]

hey guys, the attack finally stopped last night >.< been about 48hrs that we've been down. Our ISP said they cant do anything but blackhole our ip...which really doesn't help us...

I was wondering if something like this would actually help out with syn attacks perhaps in the future. Also, can I just call the local police or get the fbi involved in something like this?

thanks again for your help guys

-Phil

 

[nightowl]

The problem with trying to stop a DDoS attack at your "front door" is that the data is still being sent down your pipe from the provider. So, your performance can still suffer. Like spidey said the only real way to stop an attack is at the ISP. What SUP do you have in your 6k and how many SYNs were you seeing per second?

 

[cmetz]

ppaik, the first thing that you need to do is to take all of your public-facing servers and get them colocated at a professional facility. Those servers are the obvious and public targets people will attack. I doubt you have the kind of bandwidth and equipment to your site to handle a serious attack, so moving them to an infrastructure better suited for that is a good idea. This allows your company to have outgoing Internet connectivity (web, email out) even if the incoming stuff is under heavy attack. You might also consider getting a new address block and renumbering, so that the same DoSer doesn't just throw the same attack at the same IP address even though the server's not there.

You need four things to resist a serious DoS attack: serious equipment, serious bandwidth, serious clues, and helpful ISPs. It does not sound like your ISP is particiularly helpful, that you have experience with this problem, that you have the bandwidth to your site, or the gear to handle it. So outsource. Put them on somebody else's network who will fix the problem for you if it happens. There are a bunch of value-added colocation providers who will do that for you.

BTW, I'd never ever put a cat 6500 on the public Internet (e.g., route public Internet traffic with it - bridging is ok though).

 

[cross6]

Depending on the size of the botnet and the ISP, even they can't stop it. The problem is it's hard to tell there is a botflood until the pipes start getting smaller and smaller toward your connection (To borrow from sen stevens analogy

If it's 200,000 cable modem bots your isp might even get choked depending on what tier they are. Even if you knew every single ip and blocked it, you would still only get about .0001% of your bandwidth because before the firewall the bandwidth is so used up that legit packets just aren't there to receive.


The SAVVIS datacenter in dallas had a flood attack from a HUGE botnet a few weeks ago, even their tier 1 providers were having trouble filtering it. Took ALOT of hosting/business down for a few hours.

 

[ppaik]

we were seeing over 999,000 connections per second. As far as the bandwidth goes, we have a 500Mbps connection going into the Cisco 6500 and bandwidth utilization never came close to 500Mbps. It actually dropped when i blocked port 80 or did tcp intercept because the router/firewall was getting overloaded and valid traffic wasn't coming through.

also not sure what SUP is. is there a way i can get this info?

thanks again

-phil

 

[ppaik]

our isp is cogent =/

 

[spidey07]

Originally posted by: ppaik
we were seeing over 999,000 connections per second. As far as the bandwidth goes, we have a 500Mbps connection going into the Cisco 6500 and bandwidth utilization never came close to 500Mbps. It actually dropped when i blocked port 80 or did tcp intercept because the router/firewall was getting overloaded and valid traffic wasn't coming through.

also not sure what SUP is. is there a way i can get this info?

thanks again

-phil

holy crap. 999,000 per second?

You got hit hard. No wonder your 6500 sup crapped. I'm still pretty sure that inspect feature is in processor. If you want to know what supervisor is in it "show module" Also "show fabric" to make sure it's running in the highest performing switching mode.

Research IPS. These devices can "shun" these kinds of attacks. I don't know who the top players are at the moment because it is changing so quickly. Basically send a RST to the source or just drop it all together. What port were the syn attacks coming in on? 80?

And again, shame on your provider for not working with you on this.

 

[p0lar]

An ISP cannot successfully stop a spoofed-source attack from a non-bogon source.

1) How do you prove it's a spoofed IP from a non-bogon source?
2) How do you filter such a thing without blocking other legitimate traffic?

It is extremely difficult. The ISP must contact the upstream peer from which the hostile packet originated. Then, the 'investigation' gets handed over to them essentially because they have to see which peer it originated from. etc.. etc.. etc.. MAYBE, if you're lucky (ha!), you'll be able to reach ONE of the offending originating networks and they can adjust their outbound access controls such that only traffic from their network is permitted to leave. If everyone on the Internet did this, it would spell the end of DDoS attacks.

To see someone of such experience suggest that it is the ISP's responsibility to stop this really shocks me. They can help you mitigate it, but stopping it is really everyone's responsibility. Please ensure that the networks you are responsible for never permit any egress traffic that doesn't originate from it! Distribute the ACL processing load out to the edges, it's not that difficult!

 

[p0lar]

Originally posted by: ppaik
hey guys, the attack finally stopped last night >.< been about 48hrs that we've been down. Our ISP said they cant do anything but blackhole our ip...which really doesn't help us...

Actually, with that kind of pps, they're right -- they really only can null route the IP. Our company endured (wrong word -- open ice hit at full speed?) a 755mbit/s attack at a major co-lo (can't recall the pps load but it was unholy, deep into the Mpps range).. the solution was one of brute force and extremely expensive so I'd have a hard time recommending it at this stage.

Regardless, they should be trying to help you out in some way. Do you have any kind of distributed redundancy? How about a fist-full of round-robin low-ttl DNS entries for your site? (this will more than likely mandate additional DNS servers)

 

[p0lar]

Originally posted by: spidey07
oh - and that GRC site is a joke. Any decent firewall would have stopped what he encountered. He's the laughing stock of the network/security community.

"Stealth!!! I'm gonna live foreeeeeeverrrr...."

Yeah.. Gibson, lol.. :disgust:

 

[spidey07]

polar,

uRPF (unicast reverse path forwarding check - basically don't accept an IP on your ingress that you don't have an egress path for) should be implemented on each and every Internet router. this stops spoofed IPs. It's not the end all be all, but it makes it that much harder to spoof an IP address.

If an ISP is not doing this they should be crucified for not doing so. Heck if every ISP followed RFC 2832 (i think that's the one - the one that details basic measures to be taken on all Internet routers) we'd be in much better shape.

 

[p0lar]

Originally posted by: spidey07
polar,

uRPF (unicast reverse path forwarding check - basically don't accept an IP on your ingress that you don't have an egress path for) should be implemented on each and every Internet router. this stops spoofed IPs. It's not the end all be all, but it makes it that much harder to spoof an IP address.

If an ISP is not doing this they should be crucified for not doing so. Heck if every ISP followed RFC 2832 (i think that's the one - the one that details basic measures to be taken on all Internet routers) we'd be in much better shape.

I'm aware of the benefits of uRPF, but in a major ISP -- one peered dozens of times over, this is impossible to implement on centralized, redundant routers. It must be done at the edges of every network for traffic that those edge routers are ultimately responsible for. In his case, his upstream router should ensure that nothing sourced outside his network makes it past that simple ingress ACL.

The real problem is that network admins, regardless of technical advances of their country of origin, do not do these simple things. Some countries actually encourage such wicked behaviour.

So, in short, I agree with you, but uRPF is worthless in heavily-peered networks because there may be dozens of 0/0 routes based on load, expense, delay and a multitude of other qualifiers.

Edit:

A simple Cisco example for those reading this thread with internet routers they control:

---8<---
hostname routerA
ip domain-name domainA.com
!
int fa0/0
description --{ WAN transport }--
ip address 1.1.1.2 255.255.255.252
ip access-group WAN_IN in
!
int fa0/1
description --{ DMZ }--
ip address 1.1.2.1 255.255.255.0
ip access-group DMZ_INGRESS in
!
int fa0/2
description --{ LAN }--
ip address 192.168.0.1 255.255.255.0
ip access-group LAN_INGRESS in
!
ip route 0.0.0.0 0.0.0.0 1.1.1.1
!
ip access-list extended WAN_IN
remark --{ No RFC1918 entrance }--
deny ip 10.0.0.0 0.255.255.255 any log-input
deny ip 172.16.0.0 0.15.255.255 any log-input
deny ip 192.168.0.0 0.0.255.255 any log-input
remark --{ Anti-Spoofing }--
deny ip 1.1.2.0 0.0.0.255 any log-input
deny ip host 1.1.1.2 any log-input
permit any host 1.1.2.x <etc other decent traffic etc>
!
ip access-list extended DMZ_IN
remark --{ Protect LAN (simple) }--
deny ip any 192.168.0.0 0.0.0.255 log-input
remark --{ Anti-Spoofing }--
permit ip 1.1.2.0 0.0.0.255 any
!
ip access-list extended LAN_IN
remark --{ Anti-Spoofing }--
permit ip 192.168.0.0 0.0.0.255 any
---8<---