Continue reading »]]> As part of an investigation into several other things, my Internet research dredged up some work by Jim Gettys on Bufferbloat, which was related closely enough to my intended target to get stuck into it a little more.

It has been previously been believed that adding buffers across a network to help mitigate against packet loss is a “good thing™”, especially if we’re going to be dealing with VoIP and other protocols sensitive to packet-loss. Bufferbloat is the phenomenon caused by these buffers when the data source(s) can generate enough data to fill these (often deep) buffers, and thereby destroy the linkage between TCP’s congestion control mechanism and the actual data rate.

This leads to added latency, increased packet loss, lowered network efficiency (all those retransmitted packets take up space that should be used for other data), and lower throughput. These are all situations that we have experienced in the Internet, every day, and is typically shown by the sporadic performance in downloading large files across the Internet.

There is a nice introduction to bufferbloat on their page, which explains the problem, which is basically that each and every buffer within a network is a potential problem, as if data can be sent into the network faster than the buffer can empty, this will have a knock on impact on the performance of the flow (of whatever application or protocol). Whilst this does optimise the sue of bandwidth in the network, (i.e. it keeps it full), it doesn’t optimise the performance of the flows, and this means that the network efficiency drops as it’s carrying many packets that are retransmissions of the original (still buffered) data, because it hasn’t yet arrived.

The solution in the past has been to implement technologies such as random early discard (RED) in order to determine if the buffer is filling, and to selectively discard packets on flows that occupy more of the buffer than others. But it hasn’t always fixed the problem.

Whilst the Bufferbloat project is looking at ways to minimise buffer size (and potentially self tune the buffers within devices), the situation exists in each and every network, because buffers are cheap (both in parts and logic) to implement. But it may take a long time until router and switch (and all the other bits of comms kit) suppliers fix their environments to adjust automatically. The internet might be a challenge for a while, but it’s certainly possible to fix the enterprise environment.

So how do you manage the performance of a network if the feedback mechanisms are broken? I believe that the best mechanism is the management of the performance of each flow against the available bandwidth at the edge of the network. How can we implement one of these?

We need:

• A system with intelligence at the edge
• A way of understanding the available bandwidth in use at each point
• A mechanism to ensure that packet loss is managed to protocols that are less ‘important’
• A mechanism to determine if there is bufferbloat in the network core

It sounds like just the sort of things that the Ipanema Technologies‘ Autonomic Networking System delivers.

Continue reading »]]> Over time the use of multiple paths within the modern corporate network has increased. We’ve gone through a whole series of phases, and now are emerging into the 4th generation of the technology. I’ve had the (good?) fortune to experience all of these at various points, and to design networks around the deployment of these, so can speak from experience that this isn’t a particularly easy thing to do.

Lets look at some of the options:

• static host routing
• application specific routing (Policy Based routing)
• path selection technology
• application path selection against objectives using smart|path

The static routing option was quick and easy to implement, but was terrible from an operational standpoint, since a failure of a link somewhere in the network would most likely cause the traffic to blackhole, (since the static routes invariable wouldn’t fail out of the network.

Policy Based routing was more successful, especially if you wrapped the traffic in a GRE tunnel across the network, since this could be setup in later Cisco IOS releases to send keepalives that failed the tunnel if there was an issue in the network, and PBR would then fold back to the other route in the network. However, like the static routing, it’s still a bit of a nightmare trying to isolate the faults, and to scale the solution (especially if there are changes to the PBR rules for a different application or host, since you’d need to change lots (possibly 100′s or 1000′s in large networks) of router configs)

Only recently has the software and equipment been available to make this easier, and now it’s possible using Ipanema to use two networks (e.g. MPLS and Internet, or dual supplier), and to still gain all the objective based application benefits that Ipanema Technologies provides.

Admittedly, we’re not first to market with a path-selection routine, most notably Juniper has a similar feature in the WX platform. But we are the first (since we’re the only ones that have the capability) to extend our network-wide per-flow objective based management into a autonomous multi-path solution.

Smart|path allows for the selection of specific user classes, (a group of applications with similar performance requirements) to be optionally routed over upto 3 paths in the network. This means that you could have all the benifits of a path-selection technology to split traffic onto multiple routes to take advantage of some of those expensive tail circuits that are used for backup (your network provider may want to charge you for the additional IP bandwidth in use in the network core too!).

Unlike traditional path selection techniques, which force traffic down a single link until either delay or packet-loss or failure cause it to reroute, the Ipanema solution will dynamically select a path based on the delay, jitter and loss requirements of the user class. This means that should all the paths in the network be capable of delivering an application, it could route seperate flows over upto three different links. For the security-minded, this can be disabled on a user class by user class basis, so that sensitive information never leaves a more secure route.

So smart|path provides the following benifits to an organisation:

• Use of the expensive tails that are provided for backup (you might need to spend a little more for your provider to allow the bandwidth on the spare link to be utilised at the same time)
• objective based management of user classes to deliver applications against business requirements
• comprehensive monitoring and reporting on traffic
• the ability to send different traffic flows over multiple links at the same time (other technologies limit this to a single path)

Continue reading »]]> So Network Instruments have published their latest State of the Network report, and it makes for some worrying reading for network managers. The report is gathered from questionnaires answered by 592 network professionals, covering geographically diverse locations, with differing numbers of users. This time Network Instruments have concentrated on three elements, Time Consuming Troubleshooting, VoIP, and adoption of 10Gb and MPLS technologies.

“Over the last two years, while IT staffs purchased new tools to optimize applications and traffic, the amount of time spent troubleshooting performance problems increased,” said Charles Thompson, manager of systems engineering for Network Instruments. “It’s clear that relying on new tools or increasing bandwidth doesn’t address the performance problems.” – Source

Network managers need to solve some of the problems created in today’s networks, but it’s not just about choosing a tool or solution that gives you a working solution on the data side, you also need to ensure that you have the reports and ability to analyse the data to ensure that the network keeps on delivering. It’s no use having a swanky new network toy that speeds up the data, if when the crunch comes, it’s totally obscured your ability to understand where today’s particular performance issue lies.

We’ll take each of the elements in turn, starting with the time consuming troubleshooting.

The basic information I’ve distilled out of this so far indicates the following:

• 75% percent of respondents claim “identifying the source of the problem” as the key concern in troubleshooting
• 27% percent spend between 26-50 days replicating network issues each year
• 41% percent spend upto 25 days doing the same.
• 17% percent spend more than 75 days a year determining the source of performance issues.
• 33% percent see bandwidth consumption as thier biggest challenge

Network instruments’ own interpretation of this data is as follows:

A partial explanation for the large amount of hours spent troubleshooting may be found in the relatively low number of respondents that used tools and technologies to monitor application performance and health.

This is a huge amount of time being sunk into fixing performance issues, which by their very nature often are impacted by things outside the direct control of the network manager, and the nature of replicating the issues often removes (in my experience) the original source of the problem. This is obviously true in relation to the deployment of tactical optimisation technologies, as these often mask the data being transferred across the network as optimised flows. These flows also have different behaviours compared to the single end-to-end TCP flow, and providing coherent linkage from client to server of all the involved flows is a major challenge in understanding application performance related issues.

Some of the traditional site-centric application performance tools can provide monitoring of some of the issues being seen, but only with respect to the location at which they are installed. Other solutions such as NetFlow can not provide that end-to-end management of the total data path as it reports on each of the various legs of the data’s journey, rather than the end-to-end client / server route. Other solutions are available that provide total end-to-end application performance monitoring by installing agents at each end of the network, in the actual clients and servers. This provides the most accurate data, but distilling this into useful information that can be used to resolve problems is most often still quite a challenge.

On the VoIP question, the respondents answer:

• 55% have already implemented VoIP solutions
• 48% worry about VoIP call quality.
• 43% worry about VoIP’s impact on other applications.

This implies that VoIP as a technology is here to stay, but that deployments haven’t yet matured enough to integrate the reporting necessary for the network professionals’ piece of mind on the issues their users are most likely to complain about. It also shows that although they may have brought into the technology, they haven’t found a suitable integrated monitoring platform that provides them information about both the voice and data traffic in their network.

On the network deployment side, there appears to be a bias to towards the US marketplace, where 56% of respondents came from, and this appears to my mind to bias down the reported figures for MPLS deployment with 27% of organisations having deployed it, with an 8% growth set for this year. The 10Gb deployment also has an impact on the mix of respondents, since most use smaller networks with only 25% of the respondents having a userbase of more than 2500 users.

However, those that have deployed MPLS networks (particularly for voice) know that the advantages of “routing directly within the cloud” and any-to-any connectivity also has implications on the control of traffic flows between sites, since this normally means implementing points of control at each site, increasing cost and complexity.

To my mind these can all be solved with a single solution that provides the following things:

• Visibility of all network traffic flows, including performance parameters
• Understanding of VoIP flows and needs
• Control and management of traffic flows based on business objectives
• Reporting simply the performance on applications on a traffic light scale: Green, Yellow, Red (or Relax, Worry, Panic)

To my mind the Ipanema solution does all of these things:

• Ensures the delivery of business critical applications via an objective based ruleset
• Understands VoIP traffic and directly measures MOS voice call scoring, and provides AQS (application quality score) as an equivalent score for application
  performance.
• Provides network-wide visibility of traffic flows with instant (helpdesk) and historical (management) reporting

Since the bandwidth of the applications in use is dynamically adjusted constantly by the system across the network, the business traffic should always be given the bandwidth it needs to function effectively, whilst less critical traffic is also delivered within the requirements of the business-based rules. So not only does this provide the needed visibility, but also provides the control needed by network managers to solve the problems they have:

• factual information on the usage and performance of the applications transiting the network
• proactive tools for anticipating incidents and easily managing changes
• accurate real-time information on application performance across the network
• protection of business critical applications by the system even in congested network states
• control of network bandwidth based on the business requirements
• ensures VoIP or video conference call quality, without impacting business critical applications

All this leads to the Ipanema solution providing a business optimized network.

Continue reading »]]> So I’ve started the new job.. and the first week is over.

Three days of training in Paris, in which it feels like I’ve met more than half of the company, and the other two days of presentation and demonstration training. Luckily I’ve been able to hit the ground running with the technology side of things, so it’s really just a case of learning the Ipanema “tone of voice”.

The travel is a bit of bind, since I’m still finding the busy times of different parts of the motorway network, traffic has been a bit lighter than normal this week with half-terms, so we’ll have to see what it’s like next week.

Continue reading »]]> Well it’s come to that time again (nearly 11 years later), and I’m off to a different company, Ipanema Technologies

I joined GlobalOne at the beginning of March 1995, doing network technical support. Over time I graduated to being a project engineer, looking after the design of networks for customers, including Unilever and GlaxoSmithkline. After a round of changes of staff in the engineering team, I ended up managing the team, and changed the focus from individual network projects to a more long-term relationship with customers and ensuring that the network evolved over time to keep pace with the customer requirements, rather than having revolutionary changes every three years.

Then there were more changes in the organisation as GlobalOne first lost Sprint as an owner, and then was taken over completely by France Telecom. At this point the company merged with Equant, that France Telecom brought from SITA.

After a while, I moved into the WAN Optimization arena, and consultancy, which wasn’t really to big a shift, since most of my focus has been on consistent user experience.

I’m somewhat sad to leave all the colleagues and friends I’ve made there, but this being a small industry, I’m bound to keep bumping into them in the future at various locations.