Previously high availability was really limited to VRRP, which was great but had a couple of issues:

• You couldn’t use VRRP across VIF interfaces, which made high availability for ‘router on a stick solutions’ tricky.
• We experienced a few issues with interface bouncing, especially on gigabit interfaces.

VRRP is however a very nice solution, each virtual address is associated with a virtual MAC address that the currently actively router associates with the appropriate interface, the switchover is nearly instanteous.

The new clustering functionality in Vyatta is based upon the Linux HA project, which takes a slightly more simple but arguably more effective approach to the HA whereby when a failure is detected the virtual IP is reassigned to appropriate interface on the secondary router and a gratuitous arp sent out across the associated network segment to mitigate any arp cache issues.

The HA functionality also allows for failover of the ipsec vpn service, at the moment this works pretty simplistically by simply stopping or starting the service as needed, thus on the currently inactive server the VPN service and therefore tunnels simply aren’t up.

Lets take a look at a relatively simple multisite HA Vyatta solution and the associated configuration.

We have two sites, each with a pair of Vyattas configured as router, vpn, firewall, and nat. Behind them is a multi-segment internal network.

Interfaces

ldn-router1 interfaces:

interfaces { loopback lo { 

 address 10.1.1.251 { 

 	prefix-length: 24 

 } 

} 

ethernet eth0 { 

 description: "Internet" 

 address 98.76.54.31 

 	prefix-length: 28 

 } 

} 

ethernet eth1 { 

 description: "Servers" 

 address 10.1.10.251 { 

 	prefix-length: 24 

 } 

} 

ethernet eth2 { 

 description: "Workstations" 

 address 10.1.101.251 { 

 	prefix-length: 24 

 } 

} 

ldn-router2 interfaces:

interfaces { loopback lo { 

 address 10.1.1.252 { 

 	prefix-length: 24 

 } 

} 

ethernet eth0 { 

 description: "Internet" 

 address 98.76.54.32 { 

 	prefix-length: 28 

 } 

} 

ethernet eth1 { 

 description: "Servers" 

 address 10.1.10.252 { 

 	prefix-length: 24 

 } 

} 

ethernet eth2 { 

 description: "Workstations" 

 address 10.1.101.252 { 

 	prefix-length: 24 

 } 

} 

The important thing to notice here is that, the virtual ‘active’ addresses aren’t configured on the network interfaces themselves, instead they come later in the cluster configuration.

The New York site configuration is the same, except of course the IP addresses are changed accordingly.

Cluster

cluster { interface eth0 

 pre-shared-secret: "!secret!" 

 keepalive-interval: 2 

 dead-interval: 10 

 group "ldn-cluster1" { 

 	primary: "ldn-router1" 

 	secondary "ldn-router2" 

 	auto-failback: true 

 	monitor 12.34.56.73 

 	service "98.76.54.33" 

 	service ipsec 

 	service "10.1.10.1" 

 	service "10.1.101.1" 

 } 

} 

The cluster configuration on each router is identical (unless you want to do certain clever things such as run a different routing configuration in failover!). The interface definition is just for the interface that you want to monitor via. You can have multiple monitors however a failover will occur if any monitor returns a failure, in some ways this is a help and some ways its a hindrance, personally I prefer to just monitor an outside address and if its not available then go to failover where hopefully it will be (especially if we use different external blocks by router).

When a router becomes the active member of the cluster, it scans the route table for matches to the service IP and assigns the service IP to the appropriate interface, it then sends a gratuitous arp out of that interface to avoid any arp cache issues.

Routes

One downside of the Vyatta downing the ipsec tunnel when that router is not active, is that you can then only address that router on its dedicated addresses, for example if I wanted to do some remote maintenance ldn-router2 from the New York site while it wasn’t active, the only way I would be able to do so is either to log onto a machine on the London subnet and go via that, or use the public external IP (which I probably don’t want publically accessible anyway).

The solution is very simple, due to the way that VPN route matching works. When making a packet routing decision Vyatta checks the VPN tunnels for a local/remote match first, then checks against the routing table, therefore if we add a static route to each router for the whole internal network to go via its partner, we get a really neat solution:

protocols { static { 

 	route 10.0.0.0/8 { 

 	next-hop: 10.1.10.252 

 	} 

 } 

} 

Thus if a router has the VPN tunnel up (i.e. its active), it never checks the routing table and traffic goes direct, if the router has no VPN tunnel (i.e. its passive), it simply forwards the traffic to the active router.

VPN

The VPN configuration in a cluster is basically the same as a standard configuration, except the local and remote public IPs are the cluster addresses.

vpn { ipsec { 

 	ipsec-interfaces { 

 	interface eth0 

 } 

 ike-group "ike-ny" { 

 	proposal 1 { 

 		encryption: "aes256" 

 	} 

 	lifetime: 3600 

 } 

 esp-group "esp-ny" { 

 	proposal 1 { 

 		encryption: "aes256" 

 	} 

 	proposal 2 { 

 		encryption: "3des" 

 		hash: "md5" 

 	} 

 	lifetime: 1800 

 } 

 site-to-site { 

 	peer 12.34.56.73 { 

 		authentication { 

 		pre-shared-secret: "secret" 

 	} 

 	ike-group: "ike-ny" 

 	local-ip: 98.76.54.33 

 	tunnel 13 { 

 		local-subnet: 10.1.0.0/16 

 		remote-subnet: 10.3.0.0/16 

 		esp-group: "esp-ny" 

 	} 

 } 

} 

NAT

An easy pitfall on the NAT configuration is to forget that Vyatta processes source NAT before checking vpn or routing table matches. The fix is simply to exclude your internal network as a destination in the NAT configuration.

nat { rule 101 { 

 	type: "source" 

 	outbound-interface: "eth0" 

 	source { 

 		network: "10.1.101.0/24" 

 		} 

 	destination { 

 		network: "!10.0.0.0/8" 

 	} 

 	outside-address { 

 		address: 98.76.54.31 

 	} 

 } 

} 

VIFs

As i mentioned earlier, Vyattas implementation of VRRP doesn’t allow you to use VRRP on virtual VLAN interfaces, which is frankly a little annoying (although it will be fixed in the next release hopefully).

However under clustering it works perfectly, as the service IP can match and be assigned to any interface, real or virtual.

Conclusion

The clustering in Vyatta has added just enough simple HA clustering functionality that ‘just works’ to enable us to deploy far more complex and reliable solutions than was previously possible.

This is also just the tip of the iceberg, in future releases we can expect to see multiple cluster (allowing Active/Active configurations) and extra services added to the failover capability.

The workaround is to use the /etc/rc.local file to make IPTables do the job for you, heres how we did it:

#!/bin/sh -e
#
# rc.local
#
# Modified to forward to squid cache
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will “exit 0″ on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#

IPTABLES=”/sbin/iptables”
IP=”/sbin/ip”
SQUID=”10.1.1.1″      # Internal address of our squid box

# Webcache jump to cache
echo Setting up jump to webcache

# clear any existing entries
$IPTABLES -t mangle -F
$IPTABLES -t mangle -X

# Don’t mark webcache traffic
$IPTABLES -t mangle -A PREROUTING -j ACCEPT -p tcp –dport 80 -s $SQUID
# Internal subnets to exclude
$IPTABLES -t mangle -A PREROUTING -j ACCEPT -p tcp –dport 80 -d 10.0.0.0/8     #Don’t cache internal

# External sites to exclude
$IPTABLES -t mangle -A PREROUTING -j ACCEPT -p tcp –dport 80 -d 1.2.3.4 #IP address of site you want to exclude from going to the cache

# Now mark our traffic, we have a number of subnets on virtual interfaces we want to grab, if you aren’t using vifs simply use eth1 or whatever you are using
$IPTABLES -t mangle -A PREROUTING -j MARK –set-mark 3 -i eth3.102 -p tcp –dport 80
$IPTABLES -t mangle -A PREROUTING -j MARK –set-mark 3 -i eth3.103 -p tcp –dport 80

# Send the marked traffic to table 2 (you can actaully use whatever table you want, i used 2 because we are using eth2 for the subnet squid is on.
$IP rule add fwmark 3 table 2

# set the default route for table 2, change eth2 for the interface you are on
$IP route add default via $SQUID dev eth2 table 2

# Make sure we exit
exit 0

Probably the most important job it does for us though is for our firewalling, core network routing and traffic management for both bit10s internal systems and for the ISP hosting side of the business.

Our implementation is a highly bespoke customisation of Debian using things like iptables, IMQ, vconfig, etc. all good stuff and happily handles the routing and traffic management for our entire colocation and ISP services, however we would be the first to admit it isn’t the easiest to maintain and look after. The options in the commercial club (Cisco/Juniper) are simply way to expensive and don’t offer the flexibility we require.

A couple of months a go a client came to me needing some fairly significant network reorganisation. They had multiple offices around the world, at each site they had a very unsegmented ‘flat’ network, between the offices they had a mixture of MPLS and VPN tunnel solutions, and a number of single points of failure.

What we needed to do was to segment up each site into sensible subnets and bring additional resilience to the firewalls and routers.

Our initial reaction was to simply do another custom Linux configuration similiar to our own setup, however we were concerned about the time to get this right and the implications of future maintenance, so we look off the shelf and very quickly discovered Vyatta.

Vyatta for the uninitiated of you punts itself as ‘ The dawn of open-source networking’, personally I think this is a bit of pessimistic afterall we have been doing routing and firewalling with Linux as long as I can remember, its more of a ‘The late afternoon after a good siesta of open-source networking’.

However here at bit10 ‘Loving Doing Digital’ headquarters we can’t really criticise people on their taglines…

So why is Vyatta different?

The answer is simply its relatively easy to get to up and running, has a pretty web interface for those who have command line fear, and above all fantastic support.

Vyatta comes in two flavours, the fully open-source free ‘community edition’ and the ‘supported edition’. The community edition will suit you down to the ground if you have relatively simple requirements, basic routing/firewalling/etc., however if you have pushing the envelope you are going to the need the supported edition, which comes in two flavours ‘$647/£325′ for web only support, and ‘$897/£450′ for full telephone support, both supported flavours include free updates with the lastest fixes.

The telephone support is superb.

Vyatta does have limitations, especially if you are used to getting under the hood and having the full flexibility of Linux based routing, however the payback is a solution thats far simpler to manage.

Things we didn’t like:

1. We couldn’t configure the built in firewall to transparently push traffic to a Squid proxy server. We got around this by going under the hood and having a custom rc.local file that tag and forwarded the traffic (I will post our script on followup blog).
2. VRRP over VIFs. Vyatta supports VRRP (Virtual Router Redundancy Protocol) and VLANs out of the box, however you can only run VRRP on real ethernet inferfaces, which is troublesome if you are doing a ‘router on a stick’ solution. We have spoken to Vyatta about this and they have pencilled the functionality in for an upcoming build – good stuff!
3. The CLI, in a nutshell the CLI isn’t like IOS, its good and fulfill its job, its just that mental switch you have to make from IOS mode to something different.

Things we really liked:

1. VRRP (Virtual Router Redundancy Protocol) out of the box… VRRP was so simple to set up (on real ethernet interfaces), the village idiots really stupid cousin could have done it.
2. The separation of configuration from installation. Take a clean server, insert Vyatta CD, one line to install it to the local hard drive, copy your configuration onto it. Job done.
3. Support. The Vyatta team are passionate about their product, both on the telephone and on the web, and they know its limitations and will tell you so, so you don’t waste any time trying to make it do something it won’t.

Top Tips:

1. Unless you are doing something dead basic, go for the supported package, this ensures you get the latest version, you get the support and frankly you support Vyatta.
2. Don’t mess around trying to deploy it on that 5 yr old bodge of a server you have sat in the corner gathering dust, your firewall/routers will be business critical, so splash out on some decent hardware, we deployed on Dell 860s (<£1,000 each), which is all supported hardware and had no hardware related issues.
3. Think about what you are trying to achieve, plan it first.
4. Make sure you actually know a little about networking, while making life as simple as possible for the user, its not dumbed down to the level of a £50 ADSL router from PCWorld. If you don’t know the basics about routing, firewalling, etc. get someone to help you.

Performance:

I haven’t done any formal performance testing on Vyatta, however we have deployed it in bandwidth heavy environments, with upwards of 200 of LAN users across 10 network segments, and a single processor Dell 860 is running well under 10% load…
So is Vyatta Cisco on Shoestring… in my opinion if Vyatta can do the job you want, its definitely preferable to the Cisco option, probably the strongest reason beyond just price, is that Vyatta abstracts the software from the hardware, i.e. within reason you can redeploy a Vyatta configuration on any server with enough interfaces.

All in all well worth looking at.

In the end I figured out that that the difference was I could VPN into Windows 2000 servers but not Windows 2003.

In the logs I was getting:

LCP terminated by peer (random load of symbols)

I tracked it down in the end to one setting under the PPTP config.

Under KNetworkManager this appears under ‘Compression & Encryption’, ‘Encryption’, ‘Require 128 bit MPPE encryption’. Check this and it all works.

I kept getting a permissions error in my Exmerge.log file:

'Verify that the Microsoft Exchange Information Store service is running and that you have the correct permissions to log on. (0x8004011d)'

This is one of those annoying ‘exchange being oversecure for no good reason moments’, basically the administrator user (under which you are probably logged on for such an exercise) is denied permission to access individual mailboxes.

I found a couple of MS documents on the subject:

http://support.microsoft.com/kb/322312

and,

http://technet.microsoft.com/en-us/library/aa996410.aspx

Neither of them unfortunately resolved the problem in my case. It appears that even adding the administrator user to a temporary windows ‘exchange recovery’ group still didn’t allow access to the mailbox for extraction.

The solution is to:

1. Create a new Active Directory Group called ‘Exchange ExMerge’ (or whatever you like).
2. Give the new group full permissions on the store as per the 2nd MS article above.
3. Create a new user and add it to the Exchange ExMerge group and Domain Admins.
4. Logon to the Exchange server as the new user, run Exmerge and it should finally work.

You could of course just create the user and give it the appropriate permissions but I was just being ‘proper’!

Wow, you know you have hit the bigtime when one of your clients gets in The Register with a full on bit of espionage.

To cut a long story short a client of ours, VBi Limited, got broken into a load of kit stolen under extremely suspicious circumstances.

The ‘baddies’ broke in, gained access to the server room within minutes (the server room, located in a highly unobvious location), and whipped in excess of 30 drives out of the servers.

Sunday 1st April, the phone rang, I will never forget the moment I had a particularly aggressive hangover, ‘Hi Ben, its Chris here, we’ve been robbed!’ (well he used something a little more aggressive than ‘robbed’, but I am trying to keep my blog relatively expletive free’).

We broke out the disaster plan, the first hour was assessment i.e. sent the client, Chris, in to find out how bad it was – meanwhile i engaged on a major recovery program of my own, consisting of Berocca, bacon of eggs and enough water to drown an otter.

A couple of hours later the verdict came back, we had lost pretty much every removable drive – so that’ll be a full on recovery then.

Monday 2nd April, Les and I were in the car at 6am on the road to rainy Blackburn, enroute trying to acquire 30 odd replacement hard drives, Dell frankly were as much use as a server without hard drives, in fact we couldn’t even find the right phone number to get someone who could vaguely begin to think about the merest possibility of helping us within the next week let alone the next few hours.

Fortunately our good friends at Serversource, in Northampton, came to the rescue and had all the drives complete with caddies, with us in Blackburn, by lunchtime. Good work fellas!

Thanks to Les and I working our magic, by the end of play Monday we had restored the AD, all servers base restored, by the end of play Tuesday we had completed a full recovery of all services…. CRM, Intranet, Exchange, the whole works. 

Thanks for coming!

http://www.itoctopus.com/index.php?option=content&task=view&id=18&Itemid=27

I am just beyond part 1 of the problem, fingers crossed for a response from MS with a reset password email for me.

In the meantime my hacker continues to cause trouble!

Update (21/07/2008)

After recieving many posts and emails with people asking and pleading with me to help them get their hotmail back I have prepared a handy document that tells you everything you need to know. I am making a small charge for this of $10, you can buy it here.

My hotmail/msn messenger login has been stolen, I have for some years now had the hotmail address - king@hotmail.co.uk, which i won in a eBay charity auction (for NSPCC).

The address regularly attracts attention of hackers, I get frequent emails in my Hotmail ‘Reset your windows live password now’, not to mention at 10 people a day randomly adding me either to see who has the address or by mistake!

So how did they do it? Being in our business all our workstations are locked down, firewalled, security patched, anti-virussed, anti-spywared up to the hind teeth, so a workstation compromise seems unlikely.

The perpetrator (by all accounts a 19yr old Egyptian male), who has been boasting to my contacts about how ‘he works for msn, and has hacked my account’, by all accounts achieved this by the most artistic form of hacking ‘social engineering’.

‘Social engineering’, basically means he has convinced someone at MS Live to reset my account password, thus gaining him access to my account. He has then gone on to change the other pieces of information (secret phrase, alternate email address), thus stopping me resetting it back.

So I am in the annoying stage at the moment where someone else is logged onto my messenger, and I can’t do anything about it until MS sort it out. BOO!

Update (21/07/2008)

After recieving many posts and emails with people asking and pleading with me to help them get their hotmail back I have prepared a handy document that tells you everything you need to know. I am making a small charge for this of $10, you can buy it here.

Dell 2800, 2x36Gig (RAID 1 – Systems Boot)), 3x146Gig (RAID 5 – Data), primary file server, primary AD controller.

12PM Yesterday Dell Server Manager began reporting a fault on disc 1 (36Gig), the first step in this scenario is always to reseat the disc to rule out any possible connection issues.

This indeed brought up both drives again, only to be followed 5 minutes later by a failure of both drives in the set.

Rebooting the server to the RAID BIOS revealed that there were faults found  on disk 0 not disk 1 (as reported by the Dell Server Manager).

Lesson 1 – never underestimate the power of the hot spare.

At this current  moment in the time the company was without file, print and primary AD.

We decided to remove the disc 0 (that the bios said was faulty), and bring the disc 1 (that at bios said had no faults), back online.

Unfortunately a reboot in this scenario forced a windows chkdsk, which found faults on the drive. Although the system booted, AD did not come up and system was effectively broken.

We had no spare chasis available to drop the data disks  into, so we were in a fast reinstall situation.

The first step however was to seize the FSMO roles using NTDSUtil to one of our backup AD controllers, and cleanup all references to the old primary AD.

While these changes were propogating through AD we began the reinstall process. As we had no spare disks we were forced to reinstall to the one disk we thought was good.

All went well, and we were smart and didn’t repromote the server to AD yet.

Lesson 2 – only bring a server up as an AD controller when its a known good.

Unfortunately later that day the server failed again. Fortunately we were able to bring the disk online again without a problem.

The next day the first new disk arrived, we took the decision to not install it yet as we felt mirroring might force the remaining dodgy disk to fail…

… which it dutifully did.

We took the decision to then install the spare, allowing the RAID BIOS to resync the drives without rebooting the OS. We felt that we could have done it on the fly with OS running but that gave us a greater chance of the mirroring failing.

It took about 40 minutes for the mirror to take place.

We then removed the dodgy mirror, leaving us running on 1 disk until a further warranty replacement arrived.

Lesson 3: Always get disks of different ages, batches, brands if possible, two disks from the same batch can fail at the same time.

Sigh…