You probably already know that a site hosted by Net Logistics was attacked with a major DDoS attack recently. The attack was publicised in newspapers and several discussion forums.

Prior to the attack, the core network infrastructure was upgraded to a pair of Cisco 7206VXR NPE-G2 routers and a pair of Cisco Catalyst 4507R switches. We believe if this network upgrade had not taken place before the attack, the attack would have been much more catastrophic.

We have dealt with DDoS attacks before, but we have never experienced such a large attack. The attack was bigger than 500 Mbps at certain times (from what we could see). At other times the attack was big enough to saturate several of our upstream providers’ links (probably bigger than 1 Gbps).

Apart from the size of the attack, what made this attack difficult to deal with was the frequency at which the attacker could change the target IP. This, however, allowed us to change website IPs on that particular server to quickly pinpoint which site was being attacked. Another aspect that made this attack particularly difficult to deal with was the number of sites being attacked. More than one site was being attacked, but never simultaneously. As we continued to null route the target IPs, the attack moved to devices higher up in the chain, such as our routers, and our upstream provider’s routers. It was clear that the attacker did not want us to resume hosting the sites that were now off the air.

Improvements

We are looking into solutions which provide us protection from distributed attacks. The costs of such solutions may not make it viable. The solutions typically involve installing an additional connection from a specialist company that is able to handle distributed attacks. Once we are under attack, we would be able to switch to the provider. The traffic we receive during the attack would be filtered by this provider.

Due to the attacker being able to switch the target IP so quickly, we were under pressure to change IPs even quicker to determine which site was being attacked. In future, we will simply send out an email to all customers on the affected server that the IP has changed, and instructions on how to retrieve the new IP from their control panel. This means we would not need to publicly post any IP addresses, which may be within view of the attacker.

Downtime Summary

The attack did not affect our network or servers in Equinix. The downtimes mentioned apply to our infrastructure in Global Switch only.

The total network downtime caused by the attack was 17 minutes.

The total cabinet downtime where the server is located was 1.5 hours.

The total server downtime of the server being attacked was roughly 3-4 hours excluding propagation times.

International traffic was patchy throughout the attack. We believe this is because the attack was mainly coming from international hosts.

This entry was posted on Wednesday, November 25th, 2009 at 12:33 pm and is filed under Uncategorized . You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.