We’ve finally taken delivery of the all new EqualLogic PS6010 SAN array, which provides a higher bandwidth of 10 Gbps connectivity (20 Gbps total) over fibre, and a separate management network. Previous models only provided 4 Gbps via 4 x 1 Gbps copper, which includes the management network.

The new EqualLogic SAN will allow us to provide new high availability hosting services at unparalleled speeds and reliability.

A few pictures below provide a sneak peek into some new infrastructure for 2010:

We decided to try go karting for our team day out. Unfortunately, three of our team members could not make it, so 12 of us had the entire track to ourselves for an hour and a half. The track (Kartatak Short Track) has plenty of bends and a few straights to gain some speed. Most of the karts tend to oversteer, and as a result lose much speed. It took a very long time to get back up to speed, so a bit more power would have significantly improved the karts. Some of the karts even had positive camber which didn’t really help in the corners. The best lap time on the day was a 22.1. A few snaps below:

Lecture

...

...

Kart with positive camber

...

...

...

You probably already know that a site hosted by Net Logistics was attacked with a major DDoS attack recently. The attack was publicised in newspapers and several discussion forums.

Prior to the attack, the core network infrastructure was upgraded to a pair of Cisco 7206VXR NPE-G2 routers and a pair of Cisco Catalyst 4507R switches. We believe if this network upgrade had not taken place before the attack, the attack would have been much more catastrophic.

Network Infrastructure

We have dealt with DDoS attacks before, but we have never experienced such a large attack. The attack was bigger than 500 Mbps at certain times (from what we could see). At other times the attack was big enough to saturate several of our upstream providers’ links (probably bigger than 1 Gbps).

Apart from the size of the attack, what made this attack difficult to deal with was the frequency at which the attacker could change the target IP. This, however, allowed us to change website IPs on that particular server to quickly pinpoint which site was being attacked. Another aspect that made this attack particularly difficult to deal with was the number of sites being attacked. More than one site was being attacked, but never simultaneously. As we continued to null route the target IPs, the attack moved to devices higher up in the chain, such as our routers, and our upstream provider’s routers. It was clear that the attacker did not want us to resume hosting the sites that were now off the air.

Improvements

We are looking into solutions which provide us protection from distributed attacks. The costs of such solutions may not make it viable. The solutions typically involve installing an additional connection from a specialist company that is able to handle distributed attacks. Once we are under attack, we would be able to switch to the provider. The traffic we receive during the attack would be filtered by this provider.

Due to the attacker being able to switch the target IP so quickly, we were under pressure to change IPs even quicker to determine which site was being attacked. In future, we will simply send out an email to all customers on the affected server that the IP has changed, and instructions on how to retrieve the new IP from their control panel. This means we would not need to publicly post any IP addresses, which may be within view of the attacker.

Downtime Summary

The attack did not affect our network or servers in Equinix. The downtimes mentioned apply to our infrastructure in Global Switch only.

The total network downtime caused by the attack was 17 minutes.

The total cabinet downtime where the server is located was 1.5 hours.

The total server downtime of the server being attacked was roughly 3-4 hours excluding propagation times.

International traffic was patchy throughout the attack. We believe this is because the attack was mainly coming from international hosts.

Last year, Mitch Keeler of The Webhosting Show wrote an interesting article entitled the “7 Dirty Words Every Hosting Customer Should Know.” In that article, he looked at what he called some web hosting “secrets” that every well-informed hosting customer ought to know about.  I particularly liked reading the article, and thought that it would be useful to look at how a few of his “dirty words” apply in the Australian hosting marketplace and to Net Logistics.

First on Mitch’s list was the term “server resources.”  He’s correct in implying that server resources aren’t much talked about, particularly when one is buying share hosting or reseller hosting.  In fact, quite often, discussions of server resources are relegated to the back corner of a hosting company’s Acceptable Use Policy.  If you’re a hosting customer, sometimes the first you hear about “server resources” is in a nasty e-mail from your hosting company, informing you your account has been shut down since it’s interfering with the entire server’s operations.

Sadly, although the nasty part isn’t necessary (and is never part of the equation at Net Logistics!), the shut-down part sometimes is.  Just like your personal computer, network servers are fundamentally limited by factors such as CPU speed, RAM, and disk space.  If a hosting customer on a shared server utilizes an excessive amount of resources, and does so suddenly, it can result in an emergency situation where the account has to be suspended.  Here at Net Logistics, we take a large number of steps to avoid situations like that, however.  For example, aggressive server performance monitoring almost always ensures that we proactively solve server resource issues before they affect other customers.  Also, a combination of high-end server specifications and “underselling” (discussed in more detail below) also enables us to avoid issues with excessive system resource usage on our shared and reseller packages.  As for VPS and Dedicated, being “self-contained” platforms, resource issues there don’t cause a negative effect on our other customers.

Another “dirty word” on the list is “unlimited.”  More than one hosting company has learned the hard way that when you apply this to resources (that are in reality always finite), you’re bound to get into trouble, as people associate “unlimited” with “infinite.”  That said, unlimited does make sense in other contexts, when its meaning is “not limited by the hosting provider.”  This is sometimes the case with things such as with subdomains, or MySQL databases, or e-mail accounts, where the total count of those are eventually capped by other resources.  To clarify what I mean, consider this — if you have 1GB of disk space to allocate, you could create 1,000 POP3 mailboxes, provided they were only 1MB in size each.  It is in that context that Net Logistics uses the term “unlimited” — i.e. the only caps we place are on the underlying “core” resources such as disk space.

One of the more interesting “dirty words” is “overselling.”  Have you ever wondered how some (predominantly American) hosting providers are able to offer ridiculous amounts of disk space and bandwidth at an amazingly low price?  They usually achieve this by overselling (and, some would suggest, sneaky Terms of Service, too): assuming that their customers will use (significantly) less server resources in total than are actually available to be used.  It’s the same strategy used by some airlines who will overbook their aircraft in hopes that at least a few people will cancel their flights before it’s time to actually board the plane.

Although it’s a dangerous game to play — as a sudden spike in customer resource utilisation can have a cascade effect for everyone on an oversold server — it’s a workable marketing strategy if you’re selling to an uninformed audience.  Our view at Net Logistics is that overselling never makes sense, which is why the server resource allocation you purchase is always what is available to you.  Although the consequence is that this raises costs and results in a higher hosting fee than a “bargain basement” provider, our customers have very consistently indicated they like the Hosting Confidence inherent to knowing that the server resources they’re promised will always be available when needed.  Because we also actually ensure that there are still additional unused resources as a contingency, we in fact can be said to “undersell” our hosting products.

So, in sum, there are quite a few hosting terms floating about — and knowing the details of at least a few of them can certainly help the typical hosting customer understand how to better select and evaluate hosting companies.