Gaining Visibility with Flow Data
Above all else, we know that network visibility is critical in the modern threat landscape. In a perfect world organizations could collect and store mountains of full packet capture data for long periods of time. Unfortunately, storing packet data for an extended duration doesn't scale well, and it can be cost prohibitive for even for small networks. Even if you can afford to store some level of packet data, parsing and filtering through it to perform network or security analysis can be incredibly time consuming.
Network Flow data is ideal because it provides a significant amount of context with minimal storage overhead. This means that it can be stored for an extended amount of time, providing historical data that can account for every connection in to and out of your network. The storage footprint is so minimal, that most organizations measure the amount of flow data they store by years rather than by hours or days. This provides an unbelievable amount of flexibility while investigating events or breaches that have occurred in the past.
Flow data is based on the network communication "5-tuple", which includes the Source IP Address, Source Port, Destination IP Address, Destination Port, and the communication Protocol. In addition, flow data records the start and end of the communication sequence, as well as the amount of data transferred between the two network endpoints. The result looks like this:
Network Flow data is ideal because it provides a significant amount of context with minimal storage overhead. This means that it can be stored for an extended amount of time, providing historical data that can account for every connection in to and out of your network. The storage footprint is so minimal, that most organizations measure the amount of flow data they store by years rather than by hours or days. This provides an unbelievable amount of flexibility while investigating events or breaches that have occurred in the past.
Flow data is based on the network communication "5-tuple", which includes the Source IP Address, Source Port, Destination IP Address, Destination Port, and the communication Protocol. In addition, flow data records the start and end of the communication sequence, as well as the amount of data transferred between the two network endpoints. The result looks like this:
Analyzing Network Flow Data via Command Line
Introducing FlowBAT
Even though flow data is so versatile, its adoption has been slowed because most of the tools available for performing flow data analysis can be challenging to use. These tools are often command-line based and lack robust analysis features. We developed the Flow Basic Analysis Tool (FlowBAT) to address this need by providing an analyst-focused graphical interface for analyzing flow data. FlowBAT was designed by analysts, for analysts and provides a feature set that is applicable to many use cases:
Intrusion Detection and Network Security Monitoring
Detecting the presence of an intruder is the first step towards eradicating them from your network. This is often done though the analysis of intrusion detection system alerts, or by manually "hunting" through network data. Flow data is ideal for this type of analysis because you can review it and pivot between data sets rapidly. Using FlowBAT, you can setup saved queries to look for hosts involved in potentially malicious communication, and then zero in on those hosts by listing all of the devices they communicated with in a set period of time. All of this can be done with just a few clicks.
Incident Response and Network Forensics
Once you know that an adversary is inside your network, it becomes crucial to quickly identify the scope of the breach so that you can contain it. FlowBAT allows you to use network flow data to determine exactly how an infected device communicated with inside and outside of the network. You can then trace this communication all the way from your network border through interior network segments. Because the storage footprint of flow data is so small, you can review hostile communication that occurred weeks, months, and even years prior to a newly discovered intrusion. Since most breaches are discovered months after originally occurring, this kind of flexibility is critical in determining the long term impact of an intruder on your network.
Internal Network Intelligence
In order to understand how attackers might gain access to your network, it is critical to identify how devices on your network communicate. This includes identify which hosts exists on certain network segments, which devices listen on specific ports, and the common communication partners for individual devices. FlowBAT makes this very easy with the ability to create saved queries and asset modeling charts. You can even proactively identify changes in these baselines by placing your saved queries on to a dashboard for continual updates and review.
System and Network Troubleshooting
FlowBAT isn't just a security tool. How often do network administrators receive complaints about devices not communicating with one another or about the network being slow? FlowBAT is a great tool for tracking network communications throughout your network to determine exactly where a problem might be occurring. This can save a great deal of time for administrative staff, and provides cross-functional usage between IT and Information Security groups.
Standards Compliance and Auditing
Whether you are dealing with Sarbanes-Oxley or HIPAA, compliance is becoming a part of most information security practitioners daily job duties. Several of the more prevalent and newly emerging standards bodies are now requiring some form of network auditing that can be used to demonstrate the paths sensitive devices are using to communicate inside and outside of your network. FlowBAT provides a mechanism to pull this information quickly. Why just show someone a network diagram when you can show them actual data records as communication moves through your network? FlowBAT lets you back up what your network architecture diagrams show. You can't make an auditor any happier than that.
Intrusion Detection and Network Security Monitoring
Detecting the presence of an intruder is the first step towards eradicating them from your network. This is often done though the analysis of intrusion detection system alerts, or by manually "hunting" through network data. Flow data is ideal for this type of analysis because you can review it and pivot between data sets rapidly. Using FlowBAT, you can setup saved queries to look for hosts involved in potentially malicious communication, and then zero in on those hosts by listing all of the devices they communicated with in a set period of time. All of this can be done with just a few clicks.
Incident Response and Network Forensics
Once you know that an adversary is inside your network, it becomes crucial to quickly identify the scope of the breach so that you can contain it. FlowBAT allows you to use network flow data to determine exactly how an infected device communicated with inside and outside of the network. You can then trace this communication all the way from your network border through interior network segments. Because the storage footprint of flow data is so small, you can review hostile communication that occurred weeks, months, and even years prior to a newly discovered intrusion. Since most breaches are discovered months after originally occurring, this kind of flexibility is critical in determining the long term impact of an intruder on your network.
Internal Network Intelligence
In order to understand how attackers might gain access to your network, it is critical to identify how devices on your network communicate. This includes identify which hosts exists on certain network segments, which devices listen on specific ports, and the common communication partners for individual devices. FlowBAT makes this very easy with the ability to create saved queries and asset modeling charts. You can even proactively identify changes in these baselines by placing your saved queries on to a dashboard for continual updates and review.
System and Network Troubleshooting
FlowBAT isn't just a security tool. How often do network administrators receive complaints about devices not communicating with one another or about the network being slow? FlowBAT is a great tool for tracking network communications throughout your network to determine exactly where a problem might be occurring. This can save a great deal of time for administrative staff, and provides cross-functional usage between IT and Information Security groups.
Standards Compliance and Auditing
Whether you are dealing with Sarbanes-Oxley or HIPAA, compliance is becoming a part of most information security practitioners daily job duties. Several of the more prevalent and newly emerging standards bodies are now requiring some form of network auditing that can be used to demonstrate the paths sensitive devices are using to communicate inside and outside of your network. FlowBAT provides a mechanism to pull this information quickly. Why just show someone a network diagram when you can show them actual data records as communication moves through your network? FlowBAT lets you back up what your network architecture diagrams show. You can't make an auditor any happier than that.
FlowBAT Features
FlowBAT has several features that make it applicable for analysts with multiple goals operating in a wide array of environments. This includes:
Multiple Deployment Scenarios
FlowBAT can be deployed in an existing SiLK environment or as a part of a new installation. You can deploy SiLK in two ways: local or remote. A local FlowBAT installation requires that you install FlowBAT on the same system as your SiLK database. This method is fastest as it doesn't have to traverse the network to query flow data. A remote FlowBAT installation allows you to install SiLK on a system separate from your SiLK database. In this scenario, FlowBAT queries flow data by utilizing the SSH capability of an existing server running SiLK. This allows FlowBAT to transmit queries and receive data securely with minimal additional setup. You can even deploy FlowBAT on a cloud based system as long as it can reach your SiLK database over SSH. In either deployment scenario, FlowBAT can be up and running in a matter of minutes.
Multiple Deployment Scenarios
FlowBAT can be deployed in an existing SiLK environment or as a part of a new installation. You can deploy SiLK in two ways: local or remote. A local FlowBAT installation requires that you install FlowBAT on the same system as your SiLK database. This method is fastest as it doesn't have to traverse the network to query flow data. A remote FlowBAT installation allows you to install SiLK on a system separate from your SiLK database. In this scenario, FlowBAT queries flow data by utilizing the SSH capability of an existing server running SiLK. This allows FlowBAT to transmit queries and receive data securely with minimal additional setup. You can even deploy FlowBAT on a cloud based system as long as it can reach your SiLK database over SSH. In either deployment scenario, FlowBAT can be up and running in a matter of minutes.
Quick Query Interface
Analysis is all about getting data and getting it quickly. While we have included an interface that makes this easy for seasoned flow analysis pros, we also provide a query interface designed to present all of the possible data retrieval options to analysts who might not be as experience, or who simply want a more visual way of getting the data they want. The quick query interface allows the analyst to iteratively build data queries and easy tweak them after the queries initial execution. This means that you don't have to spend a ton of time looking up commands to get the exact data you want.
Analysis is all about getting data and getting it quickly. While we have included an interface that makes this easy for seasoned flow analysis pros, we also provide a query interface designed to present all of the possible data retrieval options to analysts who might not be as experience, or who simply want a more visual way of getting the data they want. The quick query interface allows the analyst to iteratively build data queries and easy tweak them after the queries initial execution. This means that you don't have to spend a ton of time looking up commands to get the exact data you want.
Rapid Data Pivoting
When you are hunting through large amounts of data, you need to move quick. Using traditional analysis techniques this requires a lot of typing, multiple open terminals, and constantly copying and pasting commands. With FlowBAT, you can simply click on field values in a set of query results to add additional parameters to your existing query or to create a new query. For example, while looking at a series of flow records associated with an individual service on a specific port, you can click on a specific IP address and pivot to a data set showing all communication to and from that host. From there, you can click on a timestamp from an individual flow record and automatically retrieve flow records occurring five minutes before and five minutes after that time frame. This can all occur within a matter of seconds. This same workflow using traditional command-line analysis tools could easily take several minutes or more.
When you are hunting through large amounts of data, you need to move quick. Using traditional analysis techniques this requires a lot of typing, multiple open terminals, and constantly copying and pasting commands. With FlowBAT, you can simply click on field values in a set of query results to add additional parameters to your existing query or to create a new query. For example, while looking at a series of flow records associated with an individual service on a specific port, you can click on a specific IP address and pivot to a data set showing all communication to and from that host. From there, you can click on a timestamp from an individual flow record and automatically retrieve flow records occurring five minutes before and five minutes after that time frame. This can all occur within a matter of seconds. This same workflow using traditional command-line analysis tools could easily take several minutes or more.
Saved Queries and Dashboards
Analysts often find queries they like and will reuse them constantly. In the past, this resulted in dozens of text files thrown haphazardly in multiple directories that contain commonly used queries. Using FlowBAT's saved queries feature, you can store these queries right in the tool and execute them with a single click. Furthermore, if you use these saved queries very often, you can save them to an interactive dashboard and schedule them to periodically update over set time intervals. Using this mechanism you can stay constantly up to date on specific activity on your network. For instance, you can configure a saved query that is used to identify web servers on your network. With this query configured to execute on a periodic basis, you will be the first to know if an unexpected device starts receiving data on a common HTTP port on your network.
Analysts often find queries they like and will reuse them constantly. In the past, this resulted in dozens of text files thrown haphazardly in multiple directories that contain commonly used queries. Using FlowBAT's saved queries feature, you can store these queries right in the tool and execute them with a single click. Furthermore, if you use these saved queries very often, you can save them to an interactive dashboard and schedule them to periodically update over set time intervals. Using this mechanism you can stay constantly up to date on specific activity on your network. For instance, you can configure a saved query that is used to identify web servers on your network. With this query configured to execute on a periodic basis, you will be the first to know if an unexpected device starts receiving data on a common HTTP port on your network.
Graphing and Statistical Capability
One of the most powerful features of flow data is the power to generate statistics from aggregated data. This can yield very powerful detection capabilities such as:
While some of these statistics are best interpreted as text, sometimes it becomes easier to interpret statistical data when it is presented visually. FlowPlotter allows you to send statistical data to a graphing engine to automatically generate bar, line, column, and pie charts. This level of visualization is useful for analysis, and for helping to provide visual examples of flow data in various forms of reporting that may be required as a part of your analysis duties.
One of the most powerful features of flow data is the power to generate statistics from aggregated data. This can yield very powerful detection capabilities such as:
- Calculating Device Throughput
- Identifying Top Talking Devices
- Identifying Odd Inbound/Outbound Traffic Rations
- Examining Throughput Distribution Across Network Segments
- Locating Unusual Periodic and Repetitive Traffic Patterns
While some of these statistics are best interpreted as text, sometimes it becomes easier to interpret statistical data when it is presented visually. FlowPlotter allows you to send statistical data to a graphing engine to automatically generate bar, line, column, and pie charts. This level of visualization is useful for analysis, and for helping to provide visual examples of flow data in various forms of reporting that may be required as a part of your analysis duties.
Flexible Data Display
Every analysts processes and interprets information differently. As analysts, one thing we hate is when a tool locks you into viewing data in a very specific manner. With FlowBAT, we designed the display of flow data so that it is extremely customizable to each analysts needs. With this in mind, you can rearrange, sort, and add/remove columns as needed. This provides an analysis experience that can be customized to your personal taste, as well as to specific scenarios.
Every analysts processes and interprets information differently. As analysts, one thing we hate is when a tool locks you into viewing data in a very specific manner. With FlowBAT, we designed the display of flow data so that it is extremely customizable to each analysts needs. With this in mind, you can rearrange, sort, and add/remove columns as needed. This provides an analysis experience that can be customized to your personal taste, as well as to specific scenarios.
FlowBAT Demos
Getting Data w/ CLI Query Interface
Getting Data w/ Query Builder Interface
Manipulating Flow Data
Pivoting with Flow Data
