Now to get into the meat of what this post is about, that being updates that are being performed across various platforms.

On the billing side of things, some of you may have noticed the roll out of our new system, which is primarily used for domains at this moment. This new system is being actively developed and used in the management of domains and we will be rolling out updates soon to incorporate the existing billing accounts and packages across, however this functionality does require a large amount of time and planning to migrate people across and ensure a smooth transition.

On the technical side of things, many of our Kinetic, Momentum and Dedicated cPanel clients will start to see the rollout of the new WHM interface, which has changed somewhat more drastically compared to previous releases. The new interface has been updated based on usability that is especially useful for tablet devices and you can see more at cPanel (http://docs.cpanel.net/twiki/bin/view/AllDocumentation/WHMDocs/WHMUIChanges).

There will also be some updates coming to the main Net Logistics website, with an updated service status page in the works, providing updated information publicly compared to the existing method of forum announcements. We are also looking into mailing list announcements for both technical notices as well as providing hosting sales to our existing clients.

Now to get back to work and look into more up and coming tech.

Disclaimer: no tech were hurt in the production of these team days (much).

The server provided to Net Logistics came with thirty two 8GB RAM modules and four AMD Opteron 6180 SE Processors, with each CPU having twelve cores. So the question was – what were we going to do with this beast of a server with 48 CPU cores and 256GB of RAM? As you can imagine our technical team were just itching to get their hands on this and put it to work! We decided that we would use this server primarily for testing rather than deploying any live systems on it. We also decided that we would be testing three different layers on this server – hardware, virtualisation and applications.

Hardware Testing

Net Logistics is primarily an Intel environment and the vast majority of our servers run on CPUs produced by Intel so we were really interested in testing the AMD CPUs in this system to see how they fared. Obviously we were interested in testing the raw performance of these CPUs but our primary concern was to test their heat output and how well the server could dissipate the heat which was produced. Due to the density of the server componentry and the minimal space that this allowed for good air flow we certainly had our doubts.

We employed a variety of benchmarking and stress testing software across both Linux and Windows and we maxed out all 48 cores for up to 48 hours at a time. What we found was that no matter what we did, we could not get CPU temperatures to rise above 61°C and all other chassis temperature readings were well within acceptable limits. We were pleasantly surprised!

Virtualisation Testing

Since this server has been built by Dell from the ground up with virtualisation in mind we wanted to test a variety of Virtualisation software to see how it performed. We ended up testing VMWare ESXi, Microsoft Hyper-V, Citrix Xenserver and KVM (for Kernel-based Virtual Machine), which is the default virtualisation technology used in a number of major Linux distributions. The testing that we did on Hyper-V on this server ended up playing a direct role in our eventual adoption of this technology for our new “Ascend” Windows VPS Packages: http://www.netlogistics.com.au/hosting/vps/windows/

Application Testing

This server now and for at least the near future will be used for testing operating systems and applications. Once our virtualisation testing was complete we decided that this server would be running Citrix Xenserver in order for us to continue with our on-going in-house testing. It is currently running numerous virtual machines across a variety of different operating systems and we are using it to test many different classes of software and services such as hosting control panels, database servers, HTTP servers, high availability, clustering and many more.

This on-going testing will allow Net Logistics to keep abreast of the latest software developments and assess their potential to both provide our clients with a wider array of services and to improve upon the services we have already implemented.

• Please let me take this opportunity to remind you to use the helpdesk when submitting tickets. Try and avoid sending an email to submit a ticket. In the near future, we will be upgrading our helpdesk software, and we will be disabling email communication. There will be email notification, so you will receive the replies via email, but customers will be unable to submit a ticket via email. They will need to log in to the helpdesk interface to submit a ticket. This removes confusion as to whether we have received a ticket or not, and it is also more secure than email as we force the helpdesk to load using SSL.

Back to the topic, you are probably aware of the “Remember Password” feature in most browsers. You would probably also assume that the field name is used to store data in the browser, so that the data is only valid for that particular form, on that particular site. The following may be of interest to developers who deal with multiple forms on a single domain/site.

When we tried to implement similar field types for two separate forms within the same domain, what we found completely surprised us. Most browsers use the text description within the <form> tags to identify each field. This means that if you have two sets of forms, asking for the same type of data by text description, for example username and password, then the browser will attempt to autofill all forms under that domain with the data that may have been saved for another form. Most browsers completely ignore the fact that the field name/id is completely different in both forms.

Why is this relevant/important? Let’s say you have a billing system which allows the user to log in and check their invoices. At the same time, you have an option for that user to sign up for a new service and you ask for that user to provide a username and password for their new account. Most browsers will fill the username and password field with irrelevant data even though the html names of those fields are different to the login form. Not only is the data irrelevant but it is certainly can become a security issue if the multiple forms on a site are designed incorrectly.

There is a work around though. It is possible to tell the browser not to allow the “remember password” feature on certain forms. To achieve this, simply add the following code to the opening form tag:

autocomplete='off'

This, however, is only an official standard for HTML5, although from our testing most browsers will co-operate regardless of the document type. This technique can also be used to stop the browser from saving sensitive information such as credit card numbers.

On the client end, it is never a good idea to save passwords into a browser anyway. The data is not encrypted and can be easily viewed by anyone that has access to the browser. Instead, use tools such as Keepass (Open Source) or 1Password (Commercial) to store sensitive information such as login details.

The interesting component of this new release is the Parallels Cloud Storage. Instead of using traditional SAN storage to achieve high availability, it can use commodity storage, direct attached storage. This means, high availability containers or bare metal virtual machines can be achieved using just three normal servers with direct attached storage.

Here is a quick pro/con analysis:

• We can easily migration containers or virtual machines from existing VPS infrastructure on to parallels cloud storage.

• A traditional VPS will be highly available as a node failure will simply mean another node will boot the container.

• Migrations between two nodes using cloud storage are lightning fast, because the data is already available to each node that joins the storage cluster.

• It is more expensive to provide storage since it replicates each data chunk at least three times, though it can be configured to replicate each data chunk two times, Parallels still recommends a minimum of three. This basically means storage is at least three times more expensive. Parallels will also charge a license fee for each 100 GB available on Parallels Cloud Storage. Failure rates of hard drives are still relatively high, so RAID 10 will still need to be used.

• CloudLinux is still not available as a guest operating system for containers.

• From my testing, it doesn’t seem as though the replication to storage cluster nodes is synchronous, so there is a slight delay between data written and data replicated.

• The documentation is a bit lacking in terms of parameter configuration. Once we tune and optimise the systems for best possible configuration, I’ll be posting some benchmarks for the cloud storage component.

We encourage all customers to subscribe to the Announcements forum. You will receive all the latest news, as well as outage/services notifications directly to your email. Do not subscribe to the Announcements Discussion forum or you will receive email alerts for posts by customers.

What’s new?

• Cloud services have been publically launched now, which include High Availability VM’s, Load Balancing, Spam Filtering. We’ll soon be adding Virtual Desktop as well.
• Aspire VPS has been completely revamped, with bigger plan specifications, as well as a complete shift in platform from Virtuozzo for Windows to Hyper-V.
• We have dropped support for Plesk and DirectAdmin for Momentum VPS plans.
• Organisation Validated and Extended Validated SSL’s have been added.
• Velocity-based accounts with Ruby on Rails and Tomcat support now have their own set of plans.
• We have an entirely new category/section for Domains, since we’re now an auDA accredited registrar.
• Accelerate, Achieve, and Aspire plans now come with Plesk 11
• Accelerate and Achieve are now also based off the Hyper-V platform.

I plan to keep this blog up to date from now on! In future posts, I’ll be explaining some of our core decisions, such as the move away from Virtuozzo for Windows to Hyper-V, dropping Plesk for Linux VPS, and other ongoing issues at Net Logistics.

Stay tuned!

Prior to the attack, the core network infrastructure was upgraded to a pair of Cisco 7206VXR NPE-G2 routers and a pair of Cisco Catalyst 4507R switches. We believe if this network upgrade had not taken place before the attack, the attack would have been much more catastrophic.

We have dealt with DDoS attacks before, but we have never experienced such a large attack. The attack was bigger than 500 Mbps at certain times (from what we could see). At other times the attack was big enough to saturate several of our upstream providers’ links (probably bigger than 1 Gbps).

Apart from the size of the attack, what made this attack difficult to deal with was the frequency at which the attacker could change the target IP. This, however, allowed us to change website IPs on that particular server to quickly pinpoint which site was being attacked. Another aspect that made this attack particularly difficult to deal with was the number of sites being attacked. More than one site was being attacked, but never simultaneously. As we continued to null route the target IPs, the attack moved to devices higher up in the chain, such as our routers, and our upstream provider’s routers. It was clear that the attacker did not want us to resume hosting the sites that were now off the air.

Improvements

We are looking into solutions which provide us protection from distributed attacks. The costs of such solutions may not make it viable. The solutions typically involve installing an additional connection from a specialist company that is able to handle distributed attacks. Once we are under attack, we would be able to switch to the provider. The traffic we receive during the attack would be filtered by this provider.

Due to the attacker being able to switch the target IP so quickly, we were under pressure to change IPs even quicker to determine which site was being attacked. In future, we will simply send out an email to all customers on the affected server that the IP has changed, and instructions on how to retrieve the new IP from their control panel. This means we would not need to publicly post any IP addresses, which may be within view of the attacker.

Downtime Summary

The attack did not affect our network or servers in Equinix. The downtimes mentioned apply to our infrastructure in Global Switch only.

The total network downtime caused by the attack was 17 minutes.

The total cabinet downtime where the server is located was 1.5 hours.

The total server downtime of the server being attacked was roughly 3-4 hours excluding propagation times.

International traffic was patchy throughout the attack. We believe this is because the attack was mainly coming from international hosts.

First on Mitch’s list was the term “server resources.”  He’s correct in implying that server resources aren’t much talked about, particularly when one is buying share hosting or reseller hosting.  In fact, quite often, discussions of server resources are relegated to the back corner of a hosting company’s Acceptable Use Policy.  If you’re a hosting customer, sometimes the first you hear about “server resources” is in a nasty e-mail from your hosting company, informing you your account has been shut down since it’s interfering with the entire server’s operations.

Sadly, although the nasty part isn’t necessary (and is never part of the equation at Net Logistics!), the shut-down part sometimes is.  Just like your personal computer, network servers are fundamentally limited by factors such as CPU speed, RAM, and disk space.  If a hosting customer on a shared server utilizes an excessive amount of resources, and does so suddenly, it can result in an emergency situation where the account has to be suspended.  Here at Net Logistics, we take a large number of steps to avoid situations like that, however.  For example, aggressive server performance monitoring almost always ensures that we proactively solve server resource issues before they affect other customers.  Also, a combination of high-end server specifications and “underselling” (discussed in more detail below) also enables us to avoid issues with excessive system resource usage on our shared and reseller packages.  As for VPS and Dedicated, being “self-contained” platforms, resource issues there don’t cause a negative effect on our other customers.

Another “dirty word” on the list is “unlimited.”  More than one hosting company has learned the hard way that when you apply this to resources (that are in reality always finite), you’re bound to get into trouble, as people associate “unlimited” with “infinite.”  That said, unlimited does make sense in other contexts, when its meaning is “not limited by the hosting provider.”  This is sometimes the case with things such as with subdomains, or MySQL databases, or e-mail accounts, where the total count of those are eventually capped by other resources.  To clarify what I mean, consider this — if you have 1GB of disk space to allocate, you could create 1,000 POP3 mailboxes, provided they were only 1MB in size each.  It is in that context that Net Logistics uses the term “unlimited” — i.e. the only caps we place are on the underlying “core” resources such as disk space.

One of the more interesting “dirty words” is “overselling.”  Have you ever wondered how some (predominantly American) hosting providers are able to offer ridiculous amounts of disk space and bandwidth at an amazingly low price?  They usually achieve this by overselling (and, some would suggest, sneaky Terms of Service, too): assuming that their customers will use (significantly) less server resources in total than are actually available to be used.  It’s the same strategy used by some airlines who will overbook their aircraft in hopes that at least a few people will cancel their flights before it’s time to actually board the plane.

Although it’s a dangerous game to play — as a sudden spike in customer resource utilisation can have a cascade effect for everyone on an oversold server — it’s a workable marketing strategy if you’re selling to an uninformed audience.  Our view at Net Logistics is that overselling never makes sense, which is why the server resource allocation you purchase is always what is available to you.  Although the consequence is that this raises costs and results in a higher hosting fee than a “bargain basement” provider, our customers have very consistently indicated they like the Hosting Confidence inherent to knowing that the server resources they’re promised will always be available when needed.  Because we also actually ensure that there are still additional unused resources as a contingency, we in fact can be said to “undersell” our hosting products.

So, in sum, there are quite a few hosting terms floating about — and knowing the details of at least a few of them can certainly help the typical hosting customer understand how to better select and evaluate hosting companies.