Page 1

White Paper

White Paper: 7 Steps to Network Lab Automation

How to automate the physical layer of a network system’s test lab resulting in dynamic test beds, shorter test cycles, and higher quality testing

Š 2008-2010 Gale Technologies, Inc. All rights reserved.

7 Steps to Network Lab Automation | 2

Table of Contents Introduction ....................................................................................................... 3 Step 1. Assessing the Need .............................................................................. 4 Step 2. Deciding What to Automate .................................................................. 5 Step 3. Choosing the Infrastructure................................................................... 6 Step 4. Lab Management: Choosing the Right Solution.................................... 7 Step 5. The Architecture of the Solution............................................................ 8 Step 6. The Administrative Framework ............................................................. 9 Step 7. Integration ............................................................................................. 9 Conclusion ...................................................................................................... 10 About Gale Technologies ................................................................................ 10

Š 2008-2010 Gale Technologies, Inc. All rights reserved.

7 Steps to Network Lab Automation | 3 Introduction As IP networks are becoming ever more pervasive, networking equipment is becoming more sophisticated and must be tested for operation in more heterogeneous environments. Testing is time sensitive, and labs are affected by a host of factors that impact performance – collaborative work with offsite colleagues, partners or customers, changes in network topologies to support growth, and emerging voice, storage, and video data services to name a few. Technology advancements, new higher speed interfaces, and the corresponding increase in test equipment complexity and cost have put pressure on testing departments and test lab managers to do more with less. This has spawned an increased interest in test automation using test scripts. While many test labs have evolved to embrace some form of test script automation where appropriate, labs of all kinds are now eyeing the next logical step in this evolution: lab automation at the physical layer. At its very basic element, this involves deploying physical-layer switches that serve as interconnects in conjunction with what might be called a "lab operating system" (lab OS) – a software-based solution that manages dynamically configurable test topologies. In this new generation test environment, it is common for test gear, devices, and systems under test to be permanently connected to physical-layer switches, typically OEO (Optical-Electrical-Optical) matrix switches, OOO (all-optical) switches, or other copper or fiber cross-connect devices. In this scenario, the lab OS software provides a front-end graphical user interface (GUI) and/or application programming interface (API) for scheduling and controlling the connections between devices. Ideally, it integrates seamlessly with the lab’s existing scripts and test tools as well as provides features for managing user access and tracking the usage of the lab resources it is interconnecting. This new lab infrastructure provides many benefits, including remote access for off-premises developers and partners, the ability to run multiple tests in parallel, and the elimination of manual reconfigurations, resulting in dramatically shortened test cycles. In addition, when used in conjunction with test automation, dynamic physical configurations have dramatically increased the speed in which tests can be run. This white paper offers a practical how-to approach for moving to the new generation test lab using physical-layer automation.

© 2008-2010 Gale Technologies, Inc. All rights reserved.

7 Steps to Network Lab Automation | 4

Step 1. Assessing the Need Automating a test lab begins with a thorough assessment of the lab's environment. This assessment should meet two key criteria: one strategic, the second tactical. On a strategic level, first prove a business case for automating the physical layer of your lab. This is a critical analytical step for managers of both large and small test labs. Consider, for instance, a lab with a finite set of equipment whose configuration never changes or changes only once every six months or so. In this situation, an assessment may show there is no case for automating the physical layer of the lab. In a large or more complex test lab, where the return-on-investment (ROI) benefits of physical-layer automation are readily understandable, the assessment is more a tactical issue. One critical early step in the assessment process is developing an inventory of the lab's resources, which may be geographically dispersed in the hands of engineers at remote offices. This will pinpoint how many devices – routers, switches, gateways, Fiber Channel directors, servers, PCs, traffic generators, analyzers, and the like – the lab contains, along with their port counts. Another area to explore is the cabling infrastructure. Is it a well-organized plant, or a morass of wires? How much time do lab personnel spend re-cabling equipment in preparation for a given test and what is the impact of that time on the project backlog of the lab? The complete assessment should also include information detailing where each device is located, which devices it typically connects to and whether it is available for use by both remote and local personnel. Just as importantly, the assessment should contain a breakdown of the types of tests normally run in the lab and on each piece of equipment, their frequency, their goal, and the additional equipment required for each type of test. This part of the assessment can be eye opening as it often fully reveals the processes and organization of the lab, which can differ dramatically from management’s perspective as to how efficient the lab processes have been. Oftentimes, an assessment will uncover policies that are detrimental to the lab's efficiencies. For instance, many labs allow engineers to check out test equipment for months at a time without tracking granular information about whether the equipment is being used regularly or sitting idle. This phase may uncover significantly underutilized resources. The format of the assessment depends on the nature of the lab and its personnel. For small groups, an informal visual assessment, with an inventory written on paper, can suffice. For larger labs, a more structured process is needed. Most vendors of physical-layer automation tool sets provide customizable surveys that facilitate this process. The most sophisticated of these survey tools, which offer a variety of operations simulations, allow lab managers to perform an appropriate business-case analysis they can present to executive management to justify the implementation of lab automation.

© 2008-2010 Gale Technologies, Inc. All rights reserved.

7 Steps to Network Lab Automation | 5

A lab assessment can measure the business case for automation through factors such as increased capital utilization and efficient use of labor.

In addition to proving the business case for physical-layer automation, the information collected in the assessment is needed to support several other steps in the evaluation and implementation phases.

Step 2. Deciding What to Automate The next step is to determine exactly which sections of a lab will benefit most from physical-layer automation. Seldom does it make sense to connect every single port in a test lab to a physical-layer switching infrastructure, especially early in the automation process. Therefore, lab managers must spend a fair amount of time during the assessment process to pinpoint which areas of their labs undergo the most reconfiguration or can be better utilized or shared. The goal here is to understand where dynamic configuration of the lab’s physical infrastructure can deliver the most benefits. For some labs, it will be most beneficial to put the most expensive pieces of equipment -- traffic generators, for example -- on the switched environment. Doing so makes it easier to share those resources among multiple users or across multiple functions, as well as give the organization the best return on investment on these devices. For others, the best action to take may be to consolidate multiple patch panels and cabling systems that usually take hours to move between each test. Alternatively, it could be to interconnect dedicated test equipment –used by individual engineers at their workstations – to a set of shared equipment within the lab.

© 2008-2010 Gale Technologies, Inc. All rights reserved.

7 Steps to Network Lab Automation | 6 Most labs will want to take a phased approach to the automation process, starting where they need help the most, and adding resources to the infrastructure over time. As such, the lab OS software should allow all devices, whether connected to the switching infrastructure or not, to be designed into test topologies and reserved and tracked equally.

Step 3. Choosing the Infrastructure The capabilities of physical-layer switches vary, so a careful analysis is important to get the best infrastructure fit for a particular lab environment. Making this decision will involve examining a wide range of factors, ranging from form factor to interfaces and technologies supported by the various vendors’ switches to the conditions in the particular lab being automated. So, what exactly is a physical-layer switch? In basic terms, a physical-layer switch operates similarly to a manual patch panel, except that patching is controlled electronically through software. A physical-layer switch patches physical circuits, essentially simulating the actual plugging/unplugging of cables in/out of a manual patch panel. A physical-layer switch, or Layer 1 switch, operates at Layer 1 of the Open System Interconnection (OSI) model. A physical-layer switch merely establishes a physical connection between two ports, and can provide media conversion. It does not read, modify or route data based on packet or frame headers. Data traversing the circuit is unimpeded by the overhead of packet processing. In addition, physical-layer switches are non-blocking (to a point), protocol-independent devices that operate at full line speed. Contrast this with a typical Ethernet switch, which operates at Layer 2 or 3 in the OSI model and switches packets by reading packet or frame headers and routing data to the destination ports designated in the packet header. Some labs use Layer 2/3 switches rather than Layer 1 switches for interconnecting devices into a physical topology, but doing so is not appropriate for many labs. First of all, Layer 2/3 switches can affect the results of tests. A Layer 2/3 switch, for instance, will filter bad packets or fragments, or determine where to send a packet based on IP address, for example, which are behaviors that may not be appropriate in a test environment. Additionally, Layer 2/3 switches may be costly to scale, and do not provide the flexibility of Layer 1 switches to support a mix of media types and media conversion. When assessing which physical-layer switches to deploy, lab managers should consider these factors: • the performance characteristics required by their lab • the OSI layer at which they are testing • the number of ports they need to interconnect • data rates and interfaces in use • the any-to-any capabilities their lab requires • the copper and fiber cabling requirements of the lab • the requirements for inserting test, analysis, or impairment equipment into connections between devices Most labs today deploy a mix of physical-layer switches from different manufacturers. This allows the lab to better support a variety of data and interface types and the various characteristics of the technologies that need to be tested. In addition, feature and function differences of otherwise similar switches may dictate deploying several different switches for addressing a variety of needs within a lab. Economic factors may also come into play here. Certain types of ports may be available in many types of switches, but may be more cost-effective in a particular type or brand depending on scale and other factors.

© 2008-2010 Gale Technologies, Inc. All rights reserved.

7 Steps to Network Lab Automation | 7 Step 4. Lab Management: Choosing the Right Solution Choosing a lab OS software solution is a critical decision, because it provides the support structure for managing and automating the infrastructure of your lab. Many labs use home-grown solutions for managing various lab processes. For the most part, the use of home-grown solutions has been perpetuated by a lack of commercially available solutions for lab management. Disparate solutions for managing individual devices, test scripts, or specific test gear have not been designed with the holistic lab in mind, requiring front-end interfaces to be designed by lab personnel. But with the addition of physical-layer automation and the solutions available today, choosing and deploying an off-the-shelf lab-management solution has become a viable and cost-effective solution. Thoroughly investigate your options, and focus on the features that are important for your unique environment. Here are some key features and capabilities to look for when choosing a lab-management solution: ƒ

An open solution that manages multiple vendors' physical-layer switches: As mentioned in Step 3, most labs today are heterogeneous and require a mix of switch technologies accommodating different media types or special features provided by different switches. Proprietary solutions from switch vendors themselves or other customized solutions will lock the lab into a single hardware vendor, which restricts lab management’s choice of switches. A hardware independent solution allows the lab manager to select switches based on features, performance, port density, quality, and value rather than on a limitation of the lab’s deployed management software. And because this hardware-independence forces competition among the switch vendors, it also helps to commoditize the switches and compel vendors to provide additional value to their customers. Further, a common front end or set of commands that masks the particulars of the switching infrastructure is a key factor in the efficiency, reusability, and portability of test scripts. While hardware-independence is important, feature integration is also important, ensuring the software provides intuitive access to hard-to-use switch features such as connection tapping, multicasting, rate and framing settings, and SFP (small form-factor pluggable) diagnostics.


A GUI that allows users to easily design, save, organize, and recall an infinite number of test topologies: Look for a drag-and-drop interface that masks the underlying switching infrastructure, while providing troubleshooting capabilities that expose the infrastructure when necessary. Any subset of a device or set of devices and the interconnections between them should be able to be configured.


A scheduler that manages multiple users contending for lab resources: The software should allow reservation of lab resources at specific dates and times, as well as prioritized queuing of automated tests so that tests are run as soon as the appropriate equipment becomes available. The scheduler should also automate the process of identifying available devices and assigning them to tests as required. For example, you may want to define a test topology by describing the types of devices involved and required characteristics of the devices, and have the system find and assign matching and available devices at reservation time.


Control over user permissions: You should be able to control who can access which devices and test topologies, when, and for how long. If you have multiple departments sharing a lab, or local or remote user communities for whom limited access is appropriate, you should look for a solution that provides fine-tunable permissions and priorities on devices, topologies, and scheduling. This is discussed further in Step 6.


An API that enables automated test scripts to incorporate control of the physical infrastructure:

© 2008-2010 Gale Technologies, Inc. All rights reserved.

7 Steps to Network Lab Automation | 8 Seamless integration with the lab’s existing scripts and script managers is an important success factor that will keep engineers from having to re-create existing automation processes. This is discussed further in Step 7. ƒ

An extensible database that allows the assignment of customizable device attributes: Every lab is different, and the software should accommodate user-customizable device information that may be important for your lab to track or for engineers or managers to use for search criteria – technical data, asset data, geographic data, and the like. You should, for example, be able to search for an available traffic generator that supports a particular type of test on a specified protocol.

There are a few additional capabilities discussed in the remaining steps, and many others that you may want to look for, such as built-in reporting on activity and resource utilization, appropriate platform support for the platforms use in your organization, and integrated access to devices’ management consoles, power control, and software configurations. A lab OS designed specifically for lab environments should provide these capabilities.

The new paradigm for dynamically creating test beds remotely through lab-management software.

Step 5. The Architecture of the Solution Planning the lab architecture is a critical deployment step. As noted in Step 1, it requires a careful assessment of a lab’s existing infrastructure to determine where and how devices are generally used and connected to each other. This requires understanding, for instance, that there may be communities of interest or devices that are typically connected to each other for specific types of tests or logical reasons; a traffic generator is typically not connected directly to another traffic generator, for instance, but to some device or system under test. In developing this architecture, a lab manager should consider what needs to be achieved and what does not because all decisions have ramifications. For instance, one major mistake would be to connect all of the test equipment to one switch, and all devices under test to a separate switch. That would essentially force a situation in which most connections, which are typically between the test gear and the devices under test, are forced to cross to a different switch. This is costly, because it eats up switch ports for unnecessary inter-switch linking, and it is an inefficient way to run a test environment.

© 2008-2010 Gale Technologies, Inc. All rights reserved.

7 Steps to Network Lab Automation | 9 So, if the number of ports under test is larger than the number of ports on the largest switch available – in the case of Ethernet, this is probably around 288 ports – care must be taken to consider the connection patterns expected and to distribute the equipment among multiple switches so as to minimize the need to use inter-switch links to connect devices. Divide the devices so most typical or common ports that a device needs to connect to can be found on same switch. This minimizes having to cross switches to make a typical connection, which not only minimizes the number of switch ports that are required, but also reduces potential signal degradation factors that may result from many hops through the switching infrastructure. In many labs, however, in order to achieve the required connectivity of devices across the lab, daisychaining or otherwise interconnecting physical-layer switches cannot be avoided. Transparent set-up, management, use, and optimization of such links is another key component of a lab-management software solution. Understanding how the devices need to be interconnected to each other, planning how to connect them appropriately to the switching infrastructure and interconnecting the switches to each other in such a way as to allow required connections to be made is a very important step in the lab automation process.

Step 6. The Administrative Framework The next step in automating the physical layer of a lab is developing an administrative framework. This framework will determine user access rights to ensure that all users and groups have expected and appropriate access to lab resources. Before deploying the physical-layer automation solution, it is essential to clearly understand who will have access to lab resources, and establish clear access and permission parameters for different user communities within and outside of the organization. This also involves naming and grouping lab resources in such a way that those user communities can easily identify and use the resources available to them. All of this requires that the software solution selected to facilitate this process should be able to assign various “see,” “use,” “read/write” permissions to any device, group of devices, topology, or similar resource. An administrative framework should also accommodate special projects or ad hoc situations. That is, it should permit giving specific users or groups exclusive or prioritized access to domains of equipment for critical or special situations. Flexibility is key for developing an automated physical-layer lab infrastructure that supports operational efficiency. By eliminating a pre-defined set of generic permission sets, lab management can respond more quickly to changing business needs that require new or special projects.

Step 7. Integration Integrating a lab-management solution into an existing test environment is the last major step in automating the lab’s physical layer. To deliver the most value, the lab-management system must fit seamlessly into the existing test environment and integrate with any test-automation tool platforms in use. The areas of integration driven by the lab-management software may include system-under-test configuration, including loading a particular firmware version or disk image on a device under test, script integration with automated test logic, and discrete device control, including direct console access or scripted control of devices in the lab. To streamline test processes for greater productivity, integrate the features provided by the labmanagement software with existing tools in the lab. For example, you may have a test management solution that launches a test script or series of scripts. The software managing your physical layer infrastructure should be controllable from that test management system so that engineers can © 2008-2010 Gale Technologies, Inc. All rights reserved.

7 Steps to Network Lab Automation | 10 dynamically set up the appropriate test topology before the script or scripts are launched. Likewise, if the lab OS includes a sophisticated scheduler, use it as a launch pad for activity on script managers or test equipment to trigger scripts at the time scheduled reservations begin or end. Through this integration, lab management and test engineers can significantly improve their productivity in both automated and manual test modes.

Conclusion A variety of market forces are driving the move to increased lab automation. Cutthroat competition that demands fast time-to-market and shorter lifecycles are forcing networking device manufacturers to do more – and manage resources more cost effectively and efficiently – in their test labs. An emerging solution is a standard for the automatic control of traffic generation and analysis devices. With this standard in place, lab personnel could use a single script on test equipment from multiple vendors, dramatically reducing the number of different test scripts and greatly facilitating the sharing and reuse of test scripts across a large organization. Efficiencies and cost savings gained in the lab can translate to enterprise-wide operational benefits and reduced capital expenditures. For example, test automation can help dramatically shorten development lifecycles and time-to-market. The ability to share remote resources gives lab managers the power to create globally distributed teams, which further bring speed and efficiency to test processes. In much the same way that implementing test automation is not a short-term project, automating a lab is a long-term commitment that requires an upfront investment in time and resources, but promises an ROI in terms of both capital utilization and lab personnel productivity.

For More Information To learn more about Gale Technologies and its products, please call us at +1 866-450-3366 toll free in North America or +1 408-213-4922 worldwide, visit us at , or email .

About Gale Technologies Gale Technologies provides advanced software solutions to automate, orchestrate, and optimize resources – transforming the process of infrastructure service delivery. As a pioneer of innovative solutions for provisioning and workflow automation across networking, server, storage, and virtualization technologies, Gale enables the automated and self-service provisioning of private and public cloud environments. The company’s end-to-end solutions enable organizations to reduce capital and operational expenditures, set up any dynamic lab, demo, data center, or cloud in just minutes, and provide secure access to resources 24x7 from anywhere in the world. Gale Technologies is based in Santa Clara, CA, and serves a global customer base with offices in North America and Asia. Gale Technologies, Inc. 2350 Mission College Blvd, Suite 825 Santa Clara, CA 95054 +1 408-213-4900

© 2008-2010 Gale Technologies, Inc. All rights reserved.

Gale Technologies Explains Seven Steps To Network Lab Automation  
Gale Technologies Explains Seven Steps To Network Lab Automation  

In this document, Gale Technologies explains how to automate a network lab in seven steps. It further elaborates on how recent developments...