2 | CENIC Staffing & Operations

EXECUTIVE LEADERSHIP

CENIC’s executive staff brings decades of experience in a variety of fields that have prepared them to lead this unique nonprofit organization in its daily operations with an eye to the future.

Louis B. Fox is President & Chief Executive Officer of CENIC, where he is responsible for the strategic direction and overall management of this nonprofit that operates the California Research and Education Network (CalREN). Prior to joining CENIC in 2012, Louis served for nearly three decades as a researcher, faculty member, and administrator at the University of Washington and Duke University. He has been an active leader in national and international R&E networking efforts for 30 years, with a particular focus on broadband and social equity in the US and abroad.

Bill Paik is Treasurer and Chief Financial Officer at CENIC, where he oversees and is responsible for accounting and human resources roles. Prior to joining CENIC, Bill was the accounting manager in two private companies, NI&D and American Fish, where he was responsible and performed the day-to-day accounting functions, as well as the monthly and annual financial closings. Bill earned his bachelor of science degree in finance at the University of Southern California and attained his Certified Public Accountant license in 2001.

Tony Nguyen is Vice President of Engineering and Security at CENIC, where he is responsible for the systems, software, and network engineering departments. He also oversees the Program Management Office and Security Information Office. Tony joined CENIC in 2003 as part of a team that engineered, implemented, and operated CENIC’s first-generation statewide optical and packet infrastructure. Prior to joining CENIC, Tony was the senior network architect at Ameriquest Mortgage, and lead systems engineer at the regional Internet Service Provider Digilink, where he oversaw the day-to-day technical operations as well as research and development of new technologies that would create growth for the company.

Raul Rincon is Vice President of Operations and Human Resources at CENIC, where he is responsible for technical operations and personnel. Raul started at CENIC 17 years ago as a network engineer and has advanced through several roles. Prior to joining CENIC, Raul was a network engineer at a startup in the late 90s that was acquired by Verio and then NTT, one of the largest telecoms in the world. He worked on the core team responsible for datacenter network engineering in the dedicated and enterprise hosting environments.

DEPARTMENT ORGANIZATION & FUNCTIONS

CENIC’s greatest asset is its staff. There are 69 full-time employees working in the two offices in La MIrada and Berkeley, California. Our workgroups, whether technical or non-technical, are dedicated to CENIC’s mission and work tirelessly to realize the highest possible quality of service for our members. These groups collaborate closely with each other, with our providers, with our members, and with the broader community to achieve CENIC’s goals.

Network Operations

The Network Operations group runs the NOC (Network Operations Center) and keeps CalREN (California Research and Education Network) running seamlessly for thousands of organizations throughout the state. Its primary responsibilities are to address outages and disruptions on the network; perform maintenance tasks (e.g., replace and/or update software and equipment); and support Core Engineering on new and upgrade service implementations.

CENIC’s Network Operations Center is available to assist CENIC members 24x7x365 (24 hours per day, 7 days per week, 365 days per year). Contact information (phone: 714-220-FIXIT and email: noc@cenic.org) is published on CENIC’s website and disseminated widely to members through meetings, presentations, and publications.

Alarms and requests: Alarms from any of CENIC’s network monitoring systems (NMS) and requests for assistance (generated via phone call, email, or entries in CENIC’s ticketing system) trigger a standardized triage process. Network triage alarms and service requests span a broad range of events, such as circuit bounces, single-site outages, multi-site outages, performance issues, and major issues or outages on the backbone. Engineers are required to troubleshoot and resolve issues with CENIC equipment, as well as work with commercial service providers to resolve issues on leased circuits, and work with members to resolve local site issues. CENIC has established a triage process to create a predictable environment for handling incoming requests, thereby increasing productivity by reducing interruptions for the majority of engineers on shift. One or two engineers per shift are designated to be responsible for triage on a rotating schedule, and they assess all incoming requests and assign tickets to themselves or other staff.

Escalations: Ticket escalations to Lead Engineers or Core Engineers are available 24x7x365 whenever needed. However, CENIC’s culture encourages professional development of network engineers and sets an expectation that engineers continually enhance their technical skills. As a result, less than 1% of the requests that NOC receives require escalation. For our member institutions CENIC has defined an escalation path to provide members a method to raise visibility on any issue to the attention of CENIC management. Contact information for escalation is published on CENIC’s website and distributed to authorized contacts at every member institution and service provider.

Level 1: CENIC Network Operations Center at 714-220-3494 and noc@cenic.org. Level 2: Stanley Han, Mgr. Network Operations | Office: 714-220-3468 | Email: shan@cenic.org Level 3: Tony Nguyen, V.P. Engineering & Security | Office: 714-220-3446 | Email: tony@cenic.org

Service implementation: NOC supports Core Engineering in the implementation of both new and upgraded services. For major deployments and maintenance, the Core Engineering team develops a “method of procedures” (MOP) for NOC engineers to follow. An example of a major deployment would be installing a new backbone router at a hubsite. NOC engineers ship the necessary equipment, which may include chassis, line cards, optics, and fiber jumpers, to the site. Typically, NOC engineers do not travel to backbone sites around the state; they coordinate work with field technicians who work for our colocation facilities. To minimize disruption to users, CENIC’s NOC schedules the majority of maintenance work to occur between midnight and 6:00 AM, and issues maintenance notices at least four business days in advance, as well as posting notices publicly in CENIC’s online maintenance calendar. Engineers capture pre-maintenance and post-maintenance snapshots to ensure affected services restore within operating specifications.

Staffing: To ensure 24x7x365 coverage, the NOC has three shifts: day, swing, and grave. These are designed as rolling shifts, with start and end times that facilitate smooth transitions between shifts. Day shift starts between 5:30 and 8:30 AM. Swing shift starts between 9:30 AM and 1:30 PM. Grave shift starts between 8:30 and 9:30 PM. Engineers work four 10-hour days per week to provide optimal coverage. Each shift has a lead engineer with supervisory responsibility for five to seven engineers. Total staffing of the NOC includes one manager, three shift leads, 18 network engineers, and two interns.

Core Engineering

The Core Engineering team is instrumental in connecting millions of Californians with reliable and forward-thinking solutions, from basic broadband to advanced services for experimental networks. This team architects and designs the various networks that CENIC manages and supports, and is responsible for identifying new technologies and planning future directions in network infrastructure and services.

Its main functions are to:

1. Deploy hundreds of new circuits each year. 2. 3. 4. Identify, assess, design, and implement upgrades on the network backbone. Provide expertise for escalation of requests from the NOC. Identify, assess, and employ new and emerging technologies.

5. Furnish technical expertise for a broad range of community projects.

In performing this work, Core Engineering interacts with every team within CENIC, with members across California, with providers of goods and services, and with colleagues, both nationally and internationally.

Examples of work undertaken by Core Engineering include:

● Analysis of how members connect, how they use the various networks, overall utilization, growth projections, and capacity planning, including for LEARN and GPN. Design and implementation of network services to connect member institutions to CalREN. “Future WAN Working Group”: Forward-looking to plan for how to change the network to better support members now and in the future while reducing costs. This group coordinates with the New Network Group. Broadly scoped projects (beyond the boundaries of CalREN), such as NSF International Networking (IRNC), Pacific Wave, WRN, and GRP. Virtual Customer Equipment (VCE) pilot. Initiatives to use more automated processes on the network, such as network monitoring; explore use of non-native (alien) transponders; and collapsing the HPR and DC hardware into a single platform. (For details on future planning, see the Strategic Plan in Chapter 8.) Participation on CENIC’s Technical Advisory Councils.

The team is made up of a director, a principal network architect, three senior-level network engineers, and six mid-level “associate core” network engineers.

Network Consultation & Design

This functional area provides technical consultation, both for prospective customers and existing member institutions. For prospective new members, this effort may include explaining the benefits of connecting to CalREN. For both prospective and existing members, this effort entails technical consultation on network design requirements for connectivity, including design changes and enhancements. In this capacity, the staff must coordinate closely with other CENIC teams: Core Engineering to identify equipment cost, Project Management Office to obtain circuit costs, and Contract Management to develop and execute contracts with new customers. The tools most often used by the Network Consultation & Design staff include Cricket data utilization graphs, the Jira ticketing system, and two of CENIC's homegrown systems: Asset Management System (AMS) for inventory information and PARC for contract information.

Though not a frequent activity, there is also a consultative role this staff offers to charter members requiring an outside perspective on their local (i.e., enterprise-level) network planning, such as upgrading a campus network.

This area is currently staffed by a single employee with back-up provided through coordination with other teams at CENIC.

Project Management Office

The Project Management Office provides a centralized role that oversees project activities and ensures that projects are completed as efficiently as possible by coordinating tasks internally and externally throughout numerous project stages.

For example, this team is responsible for oversight of the entire lifecycle of network service deployment milestones, managing processes which span initiation, procurement, planning, site prep, and turn-up of service, through active production use of the service. While the majority of projects focus on service deployment, the PMO also handles major internal projects, such as the software project to transition from a legacy ticketing system (RT) to Jira.

The PMO is staffed by a manager, three project managers (PMs), two associate PMs, and two interns (an apprenticeship role). The genesis of this department began in September 2015, when two statewide initiatives dramatically increased the number of new circuit deployments: funding for libraries to join CENIC as the fifth public segment; and funding ($75M) to provide last-mile broadband connectivity to K–12 schools in historically underserved areas. This expansion caused a surge in activity that CENIC needed to manage effectively, and creation of a PMO was integral to addressing this need.

The workflow diagram below of a new circuit deployment provides an example of the many tasks PMs track, including their interactions with other CENIC departments.

Internet Services

The Internet Services team oversees and manages our network peering and exchange relationships with key Internet service providers, content providers, and cloud service companies.

The staff work with network peers such as ESnet, AMLIGHT, Google, Amazon, and Microsoft to build and maintain high-quality and high-capacity settlement-free interconnection (SFI) points between their networks and CENIC’s. Exchanging traffic with other large networks this way supports routes and connection points to enhance network performance, offers caching strategies that bring content closer to users, and expands collaboration opportunities for CENIC members.

Most importantly, this long-standing SFI strategy means that CENIC is able to address the astronomical growth in commodity traffic without increasing ISP costs. CENIC peers 90% of its non-research traffic, which means only 10% of our current traffic traverses paid commodity links. Given the scale of CENIC’s network, this is still a significant amount of network traffic.

Peering requires physical interconnection of the networks, an exchange of routing information, and is accompanied by peering agreements. CENIC works with hundreds of peers from both research and commodity networks, and has built out some of the highest-quality direct connectivity available across all R&E networks to all major cloud providers. This "built-in" connectivity generally obviates the need for CENIC members to pay for expensive direct connections to those cloud providers. CENIC’s peering infrastructure supports researchers moving their computing to the cloud, campus administrators storing data off-site, and students using online and interactive resources.

The work also involves analyzing historical data and keeping close tabs on industry trends to anticipate and stay ahead of network demand.

Internet Services is staffed by the Associate Director of Internet Services and one network engineer.

Security

The Security group leads strategy on enterprise and network security and privacy issues as well as develops enterprise policies and operates a Security Incident Event Management (SIEM) system.

In January 2016, the CENIC security group spent the bulk of their first year cataloging the people, processes, computational, networked, and data (both online and offline) assets underlying CENIC’s operations, including ancillary physical resources (buildings, physical plant operations, physical maps) to understand and prioritize the security risks. In November 2016, the team began implementation of key security efforts.

CENIC’s core security team strategizes high-level issues and manages day-to-day cybersecurity operations. Another team, the “Virtual Security Team,” was assembled with representatives from each of CENIC’s business units, as a means to collect and disseminate security-related info among all staff, and to propagate a wider culture in the organization in support of more hardened cybersecurity practices. Additionally, the CENIC Security & Privacy Technical Activities Committee (SEC-TAC) was created in July 2017 to convene discussions around security and privacy concerns within CENIC’s constituency. CENIC and ESnet established a joint cybersecurity initiative in 2016, meeting on a periodic basis to exchange tactical and strategic information as well as to provide mutual review of internal security approaches.

Systems and Tools

This functional area is composed of two teams: Systems and Tools. Systems and Tools is staffed by a manager, five systems engineers, three software engineers, a network tools engineer, and is supplemented by temporary interns as needed.

The Systems team handles operational support and maintenance of the hardware and software systems and platforms that support CENIC’s business needs. Their charge involves management of current systems, whether developed in-house or purchased, as well as support for new requirements as changes in technologies and tools may require.

The Tools team is responsible for supporting legacy systems, as well as developing new software tools. This team is currently endeavoring a complete overhaul of CENIC’s suite of tools. Historically, CENIC has relied heavily on open-source “freeware” and “homegrown” applications, which have met CENIC’s basic operational needs. However, as the size and complexity of CENIC’s infrastructure have grown, so has the need for a more sophisticated set of tools. To that end, CENIC initiated the Infinity Project, a multi-year software development roadmap, to modernize the systems architecture, and create network tools and application services to facilitate workflow automation and network orchestration.

Business, Finance, and Human Resources

As a nonprofit, CENIC relies on our accounting and human resources team to be excellent stewards of our finances, track our expenditures, and help us pass savings on to our members. The four department members are: Chief Financial Officer, Accounting Manager, Staff Accountant, and Human Resources Coordinator.

The department contracts with various vendors to support its work. TIAA administers the 403(b) retirement savings plan. KPMG has been performing annual certified financial audits for about 10 years. The accounting firm HKG LLP is contracted to: cut vendor checks, receive checks, assist with quarterly financial statements, and prepare tax returns. (AP checks are not cut in house for greater security.) QuickBooks is our accounting software. When CENIC receives federal grant funds in excess of $750,000 we are subject to an additional oversight audit.

The accounting staff rely on the Asset Management System (AMS) database to track and depreciate the inventory of network equipment such as devices, routers, and switches.

HR supports recruitment efforts by managers, issues employment offer letters, manages staff enrollment in benefit plans, negotiates for group coverage benefits, and coordinates annual employee performance reviews. The vendor Paychex provides a payroll platform and contracts with our carriers for benefits. Payroll is processed in house and approved by the CEO. (Recruitment for most technical and engineering positions is managed by Raul Rincon.)

Administrative Services and Facilities

The Administrative Services team encompasses a wide range of administrative functions: general administrative support, contract management, asset management and inventory control, executive and Board support, meeting coordination and logistics, purchasing, shipping and receiving, expense reimbursement, and the procurement and maintenance of office supplies, services, furniture, and equipment.

There are six full-time staff members with various titles and reporting relationships who staff these efforts, along with two part-time student interns. The tools used by this staff include: Jira ticketing system, PARC (contract management and purchase order system), and AMS (asset management and inventory control system).

Communications

This team is responsible for external communications for the following brands: CENIC, Pacific Wave, Western Regional Network, and the Pacific Research Platform.. A broad spectrum of deliverables is produced against an annual schedule for both a general and segmented set of audiences.

The in-house team produces CENIC events, promotes network news, progress, and uses, and develops partnering initiatives. The team consists of three full-time staff and one contractor for assistance with producing the annual conference.

All of the audience touchpoints are integrated and include the websites, blog articles, social media (LinkedIn, Facebook, and Twitter), and press releases, for each brand. Content is created, managed, and updated regularly across all.

Other materials that are produced and continually refreshed include the Annual and Biennial reports for Pacific Wave and CENIC respectively, as well as illustrated maps of network footprints, media kits, presentations, branded design collateral and logo marks, the annual conference and Innovation Awards, workshops, and other CENIC-sponsored events.

Communications staff also handle a considerable number of internal communications needs, such as design and formatting of data, reports, and special presentations for the executives, staff, and Board at meetings and conferences.

Refreshing the Talent Pool

Finding talented individuals in the technology field has always been difficult, but the ever-tightening job market for technically proficient workers in the last several years, has been even more challenging. One way we've taken on that challenge is by developing an internship program. We began with a modest effort: hiring interns into the Network Operations group in 2011. Initially, we brought in two, part-time interns referred by the faculty at a local community college. As we developed the program, we were able to create a great pipeline of network engineers. As an added benefit, we found talented interns who were able to fit into other areas of the organization as well. Interns that began in Network Operations have gone on to full-time positions in Systems Engineering, Software Engineering, and Project Management. We've expanded the program and created similar intern positions in other departments to develop multiple talent-specific pipelines. To date, we've made 13 full-time hires through our internship program; with as many as seven interns working in four different departments at any given time.

CENIC EVENTS & AWARDS

CENIC’s position in California affords us a unique perspective on advanced networking and on the needs of the communities that depend on it, as well as the responsibility to ensure that innovations and best practices are disseminated throughout those communities.

CENIC Annual Conference

CENIC has, since its inception, held annual conferences, bringing together researchers and educators throughout California, along with their colleagues from around the nation and the world. Attendees come together for presentations, demonstrations, keynote addresses, the annual Innovations in Networking Awards, latest news from networking vendors, and, of course, the ad hoc human networking at the conference.

The CENIC Annual Conference, themed “The Right Connection,” typically is a three-day event alternating each year between locations in northern and southern California in order to enable Associate researchers and educators from all over the state to attend. Conferences taking place during odd years are located in southern California at the Estancia La Jolla, while even-year conferences take place in northern California at the Hyatt Regency Monterey. Previously, conferences have taken place at Associate campus locations, including UC Davis (2016), UC Irvine (2015), Sonoma State University (2014), and UC San Diego (2013).

CENIC works to maintain a reasonable registration fee for attendees, as we understand the travel and other budget challenges facing Associate researchers and educators. CENIC does however understand that many researchers and educators in California who may wish to attend may still not be able to justify the expenditure, and hence a portion of the Annual Conference programming is streamed live via

YouTube and archived there for remote viewing within the CENICNews channel.

The Conference Committee and Program Committee that typically organize the conference and plan the programming are members of the CENIC research and education community who generously donate their time and expertise to ensure that the event is useful and enlightening and features a wide spectrum of innovations.

CENIC Innovations in Networking Awards

Each year, CENIC seeks to discover and recognize exemplary innovations that leverage CalREN and have the potential to improve the way instruction and research is conducted. We request nominations from the community of applications and projects that represent innovative uses of the network. Each year’s awards are presented during the CENIC Annual Conference. Award winners are invited to give presentations on their projects.

CENIC Innovations in Networking Awards are presented annually in some or all of the following categories:

Educational Applications Research Applications Experimental/Developmental Applications Gigabit/Broadband Applications Individual Contributions Technical Contributions Corporate Contributions

CENIC encourages nominations of applications and projects focused on research, teaching, and learning activities involving collaboration across multiple segments of the K–20 community, and/or international collaborations that have been developed within the fiscal year prior to the conference.

More recently, CENIC created the Christine Haska Distinguished Service Award to recognize extraordinary individual service to the CENIC community and its partners — local, regional, national, and international. The award is named in honor of Dr. Christine Haska (1951-2017). After a long career in higher education, Dr. Haska joined the Naval Postgraduate School (NPS) in Monterey, California, in 2002, where she served as Vice President of Information Resources and Chief Information Officer, and she served on the CENIC Board of Directors.

CENIC Research Workshops

Aside from the CENIC Annual Conference, other topic-specific research workshops are occasionally sponsored by CENIC and other advanced networking organizations and held at various locations. For example, in February 2013, a research workshop focused on 100-plus-gigabit networking, sponsored by CENIC, the California Institute for Telecommunications and Information Technology (Calit2), and the Energy Sciences Network (ESnet), was held at UC San Diego. A series of workshops focused on promoting network utilization and cross-border research between the United States and Mexico was also held in Ensenada (October 2013) and UC San Diego (February 2014), organized by CENIC, Calit2, CICESE, CUDI, and Florida International University.

In its second year, the National Research Platform Workshop brought together representatives from interested institutions to discuss implementation strategies for deployment of interoperable Science DMZs at a national, and potentially international, scale. The viewpoints of researchers, cyberinfrastructure experts, CIOs, as well as perspectives from regional and national networks, helps to shape the program. Recent experiences from projects such as the National Science Foundation-funded Pacific Research Program, along with analogous national and international projects, are essential features of the workshop, as well.

Sessions at the NRP Workshop are devoted to science-driver application researchers, who describe their needs for high-speed data transfer, including their successes and frustrations. Discussions focus on requirements from the domain scientists and the networking architecture, policies, tools, and security necessary to deploy a National Research Platform. The NRP Workshop is co-sponsored by Calit2/PRP, Internet2, ESnet, Montana State University, and CENIC.

Through these and other more targeted events, CENIC seeks to maintain its position of leadership and collaboration with other advanced networking organizations while ensuring that future networking developments within California are informed, found useful, and adopted by the communities that CENIC was created to serve.

5 | CENIC Networks

Network Infrastructure Overview Connecting to CalREN Future Technology Strategies Addressing Growth Demands & Evolving Requirements