Nov 04
Ben KingOther BGP, Hp Procurve, Telecity Amsterdam, Vyatta
Steve and I recently popped over to the Netherlands to deploy Vyatta for a Dutch IT services company ‘Danego‘, Danego do a lot of managed hosting for their clients, and retained us to deploy Vyatta.
They are using Vyatta for:
- BGP Endpoints – one for each of their two upstream providers.
- External firewalling.
- Internal subnetting and routing ’router on a stick’ style using VLANs.
- All clustered for failover and total routing and switch level resilience.
Router Hardware
Typically we deploy Vyatta on Dell R200s (what were 860s), Danego however managed to get a stunning deal out Dell on 2950s, so the deployment was to what we thought would be similiar hardware to the 860s, Dual Core processor, 4Gigs of ram, hardware sata raid.
Interestingly though Vyatta wouldn’t recognise a mirrored container on the SAS 6i controller in the 2950 despite working fine on the 6i controller found in the R200s which we have used many times before. Howevr it would recognise and deploy happily to a single drive not configured as RAID.
It of course transpires that Dell use completely different chip sets for the 6i on the R200 and the 2950…
Anyway, we tracked the problem down and discovered it was due to be fixed in the next release of Vyatta (at the time we were running 3.0.6), so a quick support call to Vyatta and they immediately sorted us out a pre-release of 3.1.3 in which the issue was resolved. This is why we love Vyatta 
Switching
Anyone who knows Steve and I will tell you we like good switches, and by good in our opinion there are only two choices; Cisco or HP Procurve (that doesn’t mean there aren’t others, just we like these!). Cisco are obviously a little more expensive, however the argument goes that you never get fired for buying Cisco, the thing with the HP Procurves is that while I would never consider using them at the upper end of the scale (say much above 4500 series), at the lower 2000 and 3000 series end of the market the HPs represent very good bang for buck, including:
- Very full feature set, it starts to top out when you want to do exciting layer 3 stuff like BGP (but hey thats what a routers for!).
- Free updates for the life of the product, this represents a serious saving over Cisco in terms of Cost Of Ownership. Especially if you subscribe to our model of buy cheaper and more to deliver redundancy than relying on unreliable overpriced change-out support agreements.
- They are arguably easier to work with, especially once you get your head around adding ports to vlans, rather than the Cisco way of configuring a single port on a vlan.
Telecity South-East Amsterdam
It was the first time Steve and I had been to Telecity in Amsterdam, it allegedly carries more Internet traffic than any other Internet exchange in Europe (some say the world) and it is a very impressive setup and in some strange dutch way friendlier and nice than its London equivelents!
Photos
Telecity Zuid-Oost
Blue shoes
Steve Toiling Away
The two Dell 2950s and the HP switches.
Conclusion
Although this is not anything like the biggest Vyatta deployment we have done, I like it because it demonstrates how using HP and Vyatta you can very effectively deliver a relatively complex redundent solution for a fraction of the equivelent Cisco price.
Apr 20
Ben KingBusiness, Life, Vyatta Abu Dhabi, Benoy, Dell, Formula 1, Vyatta, Yas Island
I have deployed Vyatta to a lot of different locations, however the deployment I did last week was a little different…
Yas Island is a naturual island on the coast of the United Arab Emirates of about 2,500 hectares or which 1,700 hectares is being developed. It is to be a $40 billion playground of marinas, shops, theme park, water park, hotels and villas not to mention a Formula 1 track.
At the minute though it is little more than a lot of sand, some mounds of earth, a few roads and a lot of cranes, and I get the pleasure on behalf of my client Benoy (architects), of extending their existing Vyatta network to cover both their Abu Dhabi city office and their Yas Island site office.
There were a number of challenges with the deployment:
- The connectivity; we had ordered a 2mbit/s leased line from Etisalat, the UAE telco, this was being delivered via a microwave link back to Abu Dhabi, at the point of landing in the country, we had no idea of the reliability, IP Scheme and weren’t even confident about the presentation!
- Disruption; the users were using a shared network provided by the client, which was painfully slow, but worked to give them email and basic web access, we had to minimise downtime.
- Reliability; we had to do everything we could to ensure reliability and remote maintainability of the network once we had left.
The Kit
Vyatta was the natural choice not only because we were using it across the rest of the Benoy network, but also because of the cost effectiveness of the hardware required to deploy a resilient configuration.
At each site we deployed 1U Dell 860s, with:
- Dual core Xeon processors
- 2GBs of Ram
- Hardware mirrored Sata drives
- Additional Intel Dual NIC card (giving 4 ethernet interfaces in total)
- Vyatta 2.3.1
The Configuration
- 4 Subnets: Workstations, Servers, Internet 1 (leased line), Internet 2 (ADSL)
- All subnets clustered across the two routers
- DHCP for workstation subnets (split across the two routers)
- Masquerade NAT for internal subnets
- Incoming NAT for email and video conferencing
- IPSec VPN tunnels back to the UK network and the other Abu Dhabi site
- Internal and external firewalling
The Microwave Link
The microwave link was a V35 serial presentation that we passed through a Cisco 1841 before passing onto the Vyattas, the resulting connection performed remarkably well giving us about 14ms round trip on pings back to the main Abu Dhabi office.
The Result
The end result is fantastic, speed and response of performance at both sites far exceeded expectations. At the main site we were replacing a Firebox VPN tunnel back to London, which had proved to be a little unreliable and extremely slow, we were putting this down to the quality of the Etisalat connection, however once we replaced it with the Vyatta VPN the network response and reliability was far in excess of expectations and performs as well as the MPLS circuits we have connecting other sites.
Martin Neal, IT Director of Benoy, said ’I am really pleased with the speed and also the “feel” of the network.‘
Photos
The Yas Island site office…
The Benoy team at Yas Island…
Our Microwave Link…
Mar 31
Ben KingNetworks, Vyatta bgp convergence, FusionCLI, glendale, qos, router, vrrp, Vyatta
The next major release of Vyatta (VC4 – codename Glendale) is due for final release on 22nd April, and I am soo excited!
This release promises to deliver plethora of new features and functionality, including:
1) FusionCLI – A new CLI based around bash that gives simultaneous access to Vyatta configuration and the underlying Linux shell. Woohoo – no more exiting from xorpsh to do basic bash stuff.
2 ) Remote Access VPN – Remote vpn clients are now supported via both PPTP and L2TP/IPSec.
3) Tunnel Interfaces - GRE support means that we can now tunnel both non-ip tunnels (Appletalk, etc. – should you really want to), as well as more importantly ip in ip tunnels, which are somewhat more useful, not least for doing resilient routing via backup VPN links.
4) QOS - Although QOS via TC has long been available to the ‘hardcore’ under the hood via tc and the shell, this is the first time that Vyatta have exposed this functionality to masses via the CLI. The initial release notes suggest that it will just be SFQ (Stochastic Fair Queuing), and shaping around packet marking. As is always the case with Vyatta they start simple and grow from there, so you can expect to see HTB, etc. added later on. Personally I am hoping they have ticked the 2.6 kernal option to enable IFB (Intermediate Functional Block), to enable traffic shaping over multiple interfaces… we will see!
5) Redesign of Routing Protocols – Vyatta has in the past been criticised for its routing protocol performance particularly in terms of BGP convergence, I am guessing this is one of the many reasons they have completely revisited the router manager. And they are keen to shout about it, this Tolly report demonstrates $8000 of Vyatta running on an IBM server, giving $30,000+ of Cisco 7204 a complete and utter kicking.
6) VRRP interfaces by VIF - This one is worth special mention because it was my first Vyatta enhancement request that has made it through to release (given I have only ever submitted 2 thats not bad going!). Basically VRRP could only previously be deployed on real ethernet interfaces, great unless like me you subnet networks up and run a ‘router on a stick’ configuration, in which case you need to be able to deploy VRRP across VIFs, you now can! woohoo!
This is but the tip of the iceberg, there are many other great improvements.
I will be deploying a version of Glendale soon, in a test environment, and I will report back!
Jan 04
Ben KingBusiness, Internet, Networks, Vyatta clustering, high availability, nat, Systems, vpn, vrrp, Vyatta
The latest subscription release of Vyatta, 2.3, has seen the addition of clustering capability, which has added greatly to the high availability features of the product.
Previously high availability was really limited to VRRP, which was great but had a couple of issues:
- You couldn’t use VRRP across VIF interfaces, which made high availability for ‘router on a stick solutions’ tricky.
- We experienced a few issues with interface bouncing, especially on gigabit interfaces.
VRRP is however a very nice solution, each virtual address is associated with a virtual MAC address that the currently actively router associates with the appropriate interface, the switchover is nearly instanteous.
The new clustering functionality in Vyatta is based upon the Linux HA project, which takes a slightly more simple but arguably more effective approach to the HA whereby when a failure is detected the virtual IP is reassigned to appropriate interface on the secondary router and a gratuitous arp sent out across the associated network segment to mitigate any arp cache issues.
The HA functionality also allows for failover of the ipsec vpn service, at the moment this works pretty simplistically by simply stopping or starting the service as needed, thus on the currently inactive server the VPN service and therefore tunnels simply aren’t up.
Lets take a look at a relatively simple multisite HA Vyatta solution and the associated configuration.
We have two sites, each with a pair of Vyattas configured as router, vpn, firewall, and nat. Behind them is a multi-segment internal network.
Interfaces
ldn-router1 interfaces:
interfaces { loopback lo {
address 10.1.1.251 {
prefix-length: 24
}
}
ethernet eth0 {
description: "Internet"
address 98.76.54.31
prefix-length: 28
}
}
ethernet eth1 {
description: "Servers"
address 10.1.10.251 {
prefix-length: 24
}
}
ethernet eth2 {
description: "Workstations"
address 10.1.101.251 {
prefix-length: 24
}
}
ldn-router2 interfaces:
interfaces { loopback lo {
address 10.1.1.252 {
prefix-length: 24
}
}
ethernet eth0 {
description: "Internet"
address 98.76.54.32 {
prefix-length: 28
}
}
ethernet eth1 {
description: "Servers"
address 10.1.10.252 {
prefix-length: 24
}
}
ethernet eth2 {
description: "Workstations"
address 10.1.101.252 {
prefix-length: 24
}
}
The important thing to notice here is that, the virtual ‘active’ addresses aren’t configured on the network interfaces themselves, instead they come later in the cluster configuration.
The New York site configuration is the same, except of course the IP addresses are changed accordingly.
Cluster
cluster { interface eth0
pre-shared-secret: "!secret!"
keepalive-interval: 2
dead-interval: 10
group "ldn-cluster1" {
primary: "ldn-router1"
secondary "ldn-router2"
auto-failback: true
monitor 12.34.56.73
service "98.76.54.33"
service ipsec
service "10.1.10.1"
service "10.1.101.1"
}
}
The cluster configuration on each router is identical (unless you want to do certain clever things such as run a different routing configuration in failover!). The interface definition is just for the interface that you want to monitor via. You can have multiple monitors however a failover will occur if any monitor returns a failure, in some ways this is a help and some ways its a hindrance, personally I prefer to just monitor an outside address and if its not available then go to failover where hopefully it will be (especially if we use different external blocks by router).
When a router becomes the active member of the cluster, it scans the route table for matches to the service IP and assigns the service IP to the appropriate interface, it then sends a gratuitous arp out of that interface to avoid any arp cache issues.
Routes
One downside of the Vyatta downing the ipsec tunnel when that router is not active, is that you can then only address that router on its dedicated addresses, for example if I wanted to do some remote maintenance ldn-router2 from the New York site while it wasn’t active, the only way I would be able to do so is either to log onto a machine on the London subnet and go via that, or use the public external IP (which I probably don’t want publically accessible anyway).
The solution is very simple, due to the way that VPN route matching works. When making a packet routing decision Vyatta checks the VPN tunnels for a local/remote match first, then checks against the routing table, therefore if we add a static route to each router for the whole internal network to go via its partner, we get a really neat solution:
protocols { static {
route 10.0.0.0/8 {
next-hop: 10.1.10.252
}
}
}
Thus if a router has the VPN tunnel up (i.e. its active), it never checks the routing table and traffic goes direct, if the router has no VPN tunnel (i.e. its passive), it simply forwards the traffic to the active router.
VPN
The VPN configuration in a cluster is basically the same as a standard configuration, except the local and remote public IPs are the cluster addresses.
vpn { ipsec {
ipsec-interfaces {
interface eth0
}
ike-group "ike-ny" {
proposal 1 {
encryption: "aes256"
}
lifetime: 3600
}
esp-group "esp-ny" {
proposal 1 {
encryption: "aes256"
}
proposal 2 {
encryption: "3des"
hash: "md5"
}
lifetime: 1800
}
site-to-site {
peer 12.34.56.73 {
authentication {
pre-shared-secret: "secret"
}
ike-group: "ike-ny"
local-ip: 98.76.54.33
tunnel 13 {
local-subnet: 10.1.0.0/16
remote-subnet: 10.3.0.0/16
esp-group: "esp-ny"
}
}
}
NAT
An easy pitfall on the NAT configuration is to forget that Vyatta processes source NAT before checking vpn or routing table matches. The fix is simply to exclude your internal network as a destination in the NAT configuration.
nat { rule 101 {
type: "source"
outbound-interface: "eth0"
source {
network: "10.1.101.0/24"
}
destination {
network: "!10.0.0.0/8"
}
outside-address {
address: 98.76.54.31
}
}
}
VIFs
As i mentioned earlier, Vyattas implementation of VRRP doesn’t allow you to use VRRP on virtual VLAN interfaces, which is frankly a little annoying (although it will be fixed in the next release hopefully).
However under clustering it works perfectly, as the service IP can match and be assigned to any interface, real or virtual.
Conclusion
The clustering in Vyatta has added just enough simple HA clustering functionality that ‘just works’ to enable us to deploy far more complex and reliable solutions than was previously possible.
This is also just the tip of the iceberg, in future releases we can expect to see multiple cluster (allowing Active/Active configurations) and extra services added to the failover capability.
RSS
Recent Comments