Attachment 5
COMPLETED SIEM & IDS INSTALLATION REPORT & INITIAL DEPLOYMENT STATUS REPORT
Completion of SIEM & IDS Installation The installation of the SIEM and IDS system was completed the afternoon of 10/31/11 and was accomplished after seven days of concerted effort between USDN and CCSF’s Internet & Data Security Specialist. At this point in time all necessary hardware and software to include the SIEM/IDS server, firewalls and taps have been installed and configured. USDN has verified that network data capture is taking place and that the SIEM/IDS system is functioning as intended. USDN notes that although the installation task had been originally initiated on August 23, 2011 it was not until October 31, 2011 that the fully functioning SIEM/IDS system could be deployed, a period of over two months. The reason for this is that considerable setbacks were experienced due to the failure of CCSF’s Technical Operations Manager to follow USDN’s requests and instructions regarding the setup and configuration of the SIEM/IDS hardware and his refusal to cooperate in the installation process as ordered by CCSF’s Chief Information Technology Officer (CITO). As a result the CITO removed the Technical Operations Manager from the project and assigned his responsibilities to the Internet & Data Security Specialist on 10/13/11. With this change in CCSF personnel, the SIEM/IDS installation project was completed within a week’s time. The details attributed to the delays and failed initial installation of the SIEM/IDS system are described in the sections below.
Initial SIEM & IDS Deployment Status The installation and deployment of the SIEM and IDS system became significantly behind schedule as noted in the previous section and remediation has been halted until the issues in this report are addressed and fixed. Furthermore, all of the issues and statements of fact included in this document are supported by “direct” evidence and relates specifically to the issue it is being asked to substantiate. All information presented as evidence is in a manner that CCSF can independently verify, which USDN strongly encourages.
ROOT CAUSE #1 The CCSF infrastructure has not been deployed as we had agreed, which has caused multiple failures to the USDN IDS System. BACKGROUND USDN is following a standard deployment model which uses mirrored ports on the different network segments to collect packets with completely passive network interfaces. The CCSF’s switches have the ability to chart, log and display the CPU, Port, and Backplane usage statistics. If the usage on any metric climbs above 70% for more than a second or reaches 80% at any time, USDN will install hardware network taps to alleviate the bottlenecks.
CCSF REMDIATION 1) The IDS system has multiple network cards installed for the express purpose of separating the capture interfaces from the administration interface. The IDS admin interface (CCSF Assigned IP 147.144.1.26) is supposed to be connected to the CCSF admin network. The IDS server and the admin interface must be classified with the same level of security reserved for the servers which store and transmit the most privileged CCSF data. The reason for this directive can be discerned using common sense alone, however, USDN presents the example below as further explanation: The IDS deployment is shown here:
CCSF SECURE APP SERVER
Privileged Data Transmissions
USDN IDS BOX
CCSF SECURE DATABASE SERVER
The USDN IDS is sniffing the packets of all the servers. It contains the same data that the highest privilege servers contain, since it mirrors that data. Therefore, not only should the USDN IDS server be kept as secure as the privileged one, but the network segment that the CCSF critical assets reside upon should be protected as well. Unfortunately this did not happen. Instead, the USDN IDS was assigned an IP address and to a VLAN of public servers. This is the exact opposite of what was requested.
IDS SYSTEM F AILURES
The inability to send mail to the mailserver was caused by the placement of the IDS server in a non privileged network segment, specifically the one used as a pseudo CCSF DMZ. It is common practice to deny connections to the SMTP port of the mailserver from any hostile or otherwise “external” segment but if the server was deployed as requested the connection issue would never have presented itself.
SUPPORTING EVIDENCE This is the vlan configuration that the USDN IDS is connected to and samples can be obtained from the switch itself, from backups, or from the running configuration copied from the switch. vlan 256 name "inst-PublicSrvrs" untagged B18 qos priority 0 ip helper-address 147.144.1.254 ip helper-address 147.144.1.253 ip address 147.144.1.1 255.255.255.0 tagged A11,B1,Trk1-Trk4,Trk10 ip igmp exit
ROOT CAUSE #2 The hard disk keeps filling up which then causes mysql to stop working. BACKGROUND USDN has deployed IDS servers on networks with far greater throughput than the CCSF network and they are nowhere near the capacity of events that the CCSF IDS receives. Specifically, the networks USDN refers to that were deployed in the same month as the CCSF IDS are:
Fry’s Electronics (Corporate Office & 38 large nationwide retail stores) The County of San Benito
As of 10/13/2011, a 30 day sample has Fry’s at 12% and they have a 39 location deployment. The County of San Benito is at 8% for the same 30 day period.
IDS SYSTEM F AILURES

The entire storage space of the server fills up every 2 days on average. This amount of data to process has made it very hard to diagnose the errors listed in this report.
SUPPORTING EVIDENCE This switch configuration has some parameters listed in the mirror port and monitor port setup that need to be changed. The configuration as it is now throws data into loops due to the direction of the port mirrors. The loops are then closed which has the effect of exponentially multiplying the bandwidth until it floods the server.
Core Switch VLAN config: vlan 611 name "PortMirroring" untagged A19 no ip address exit
Port config <snip> mirror 1 port A19 </snip> <snip> interface A1 monitor all both mirror 1 exit interface A2 monitor all both mirror 1 exit interface A3 monitor all both mirror 1 exit interface A4 monitor all both mirror 1 </snip>
Main config: Network Monitoring Sessions Status Type Sources Mirror‐Policy ‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐ 1 active port 5 no 2 not defined 3 not defined 4 not defined There are no Remote Mirroring endpoints currently assigned. BAT11S3H6# show monitor endpoint Remote mirroring destination configuration. <1‐4> Mirror destination number. <cr> Network Monitoring Session: 1 Session Name: Mirror Policy: no mirror policy exists Mirror Destination: A19 (Port) Monitoring Sources Direction ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ Port: A1 Both Port: A2 Both Port: A3 Both Port: A4 Both Port: A5 Both CCSF REMDIATION 2) The ports should not be mirrored in both directions as it is now on all of the switch ports. The only one that should be set to “both” is the port that is right in front of the main firewall. This needs to be set to “both” since once the packets leave from that port they are gone and the only way to get both sides of the TCP/IP conversation is at that point. All the other ports should be either set to Rx or Tx (not “both”) depending on flow direction and what is on the network segment. USDN and CCSF can work up a list together.
Summary USDN’s installation and deployment of the CCSF SIEM & IDS system has been challenging for a variety of reasons. As illustrated within the previous sections of this report, the root cause of the implementation delay of the functioning SIEM/IDS system lies with the inability of the CCSF Technical Operations Manager to execute technical requests regarding the implementation of the SIEM/IDS. The exact reason for the Technical Operations Manager’s behavior is unknown, but at best it is due to a lack of understanding of basic knowledge of the network for which the individual is in charge and at worst it is due to purposeful obstructionism against the mutually agreed upon network monitoring plan and directive from the CCSF CITO. Regardless, the behavior of the Technical Operations Manager created not only a delay in the monitoring of the CCSF network but resulted in additional USDN consulting expenses and unnecessarily wasted utilization of CCSF resources
required to (1) analyze data generated from the first SIEM/IDS deployment implemented with the incorrect switch configuration, (2) troubleshoot the SIEM/IDS system without transparent knowledge of the switch configuration, (3) reinstall and redeploy the correctly configured SIEM/IDS system. It is USDNâ&#x20AC;&#x2122;s opinion that although success was ultimately achieved regarding the deployment of the SIEM/IDS system at CCSF, future IT objectives may face similar roadblocks. Additionally, the outcome of any IT project going forward is in jeopardy and at risk of compromise given the behavior pattern of the Technical Operations Manager.