Export security rule hitcount to CSV
note
This tutorial covers how to dump a security rulebase hit count into a CSV file for offline processing. A typical use case leveraging rule hit counts is to identify rules with zero or very few hits that, otherwise, might be eligible for cleanup or deletion.
Requirements#
To follow this tutorial, it is recommended that that you are familiar with the concepts of Palo Alto Networks Next-Generation Firewalls, Security Policies and APIs. Some basic understanding of XML is also recommended.
Make sure you have a Palo Alto Networks Next-Generation Firewall deployed and that you have administrative access to its Management interface via HTTPS. To avoid potential disruptions, it's recommended to run all the tests on a non-production environment.
In this tutorial you'll find code examples leveraging the pan-python and pan-go. It is assumed that the reader has basic PAN-OS API knowledge (curl) and that owns an API KEY for his PAN-OS device (Grab the API Key)
Rule hit count#
Starting with PAN-OS 8.1, the firewall web and command line interface displays the hit count and additional metadata for traffic matching rules in different rulesets.
These runtime statistics can provide value in some automation use cases. For instance:
- In large deployments (thousands of rules) it might be desirable to export specific datapoints (i.e. rule name, hit count and last hit timestamp) into a CSV file for out-of-band processing (periodic security audit checks)
- SOC operations might be interested on displaying and analyzing time-based graphs of hit-count on selected rules.
Extracting runtime statistics using PAN-OS API#
The above use cases justify the need for a programmatic way of extracting runtime data points (the rule hit count in this case) from the PAN-OS device.
A network security operations engineer might be accustomed to using the device's CLI (PAN-OS CLI in this case) to access that data. The following is an example of CLI command displaying the rule hit count on a Palo Alto Networks firewall.
For years, the only way to programmatically access this type of data was using CLI automation tools (Expect / Puppet / ...). But, nowadays, most teams look to leverage APIs to support automation use cases.
The "type=op" PAN-OS API command#
PAN-OS API supports many different type of requests:
type=op(operation commands: i.e. show commands)type=config(configuration management)type=log(get log events)type=user-id(dataplane real-time object update)type=keygen(generate an API KEY out of user and password data)type=report(request report generation)- ... and few other types
The case we're covering in this tutorial requires us to use a type=op API requests. Examples of CLI commands that can be emulated with API type=op requests are show, clear, delete, test, request, etc.
type=op API requests requires a mandatory cmd=<xml-command> parameter. <xml-command> is a XML document describing the command to be executed.
For fluent PAN-OS CLI engineers, the easiest way to get the XML document of a CLI command is by enablig CLI debug on their terminal session as in the following example:
Example output:
The output of any CLI command after debug has been turned on will include two XML blocks. The first one contains data about the API request and the second one the raw XML response (in the example above only the first XML block is shown).
The request will look like <request><operations>[xml-command]</operations></request>
With all information we have so far we're ready to construct the URL for our API call. It would be:
The following linux command sequence can be used to perform the previous API call:
Example response
The output is a large XML document (it contains hit count details for all rules in the security rule-set) with the following structure:
As you can see the job of extracting runtime data from a PAN-OS firewall using the API is quite straightforward. Processing the XML output, though, can be a bit tough using plain POSIX/GNU CLI tools.
Let's leverage two of the most used PAN-OS SDK's ( pan-python and pan-go ) to get the job done.
Exporting rule hit count to CSV using Python#
Look at the following python3 snippet
The first try / except block attemps to leverage pan-python to initialize a xapi object connected to our PAN-OS device API
The second try / except block performs the following steps in sequence:
- Execute the operational command
- Parse the XML response
- Process the result data set (order by hit count number) and print CSV to sdtout
Let me highline some pieces of code:
pan-python's xapi object features automatic XML parsing of the API response. A field named element_root holds the xml.etree object corresponding to the parsed XML response.
That allows us to use the iterfind method to produce a generator (rules) that will yield all elements with XML tag entry at the XPATH .//rules/entry (look above for the XML response schema of this operational command)
A loop iterating the rules generator will convert the raw data into a python dictionary whose key will be the rule name and having the tuple (hit_count, last_hit_ts) as its value.
Inside the loop, rule becomes an xml.etree object that will be used to:
- extract the name XML attribute using its
get()method (rule name) - extract the XML text node from the XML elements hit-count and last-hit-timestamp using its
findtext()method
The XML text values are casted to integer (hit-count) and formatted (date representation of the UNIX timestamp)
The last step is a basic sort by hit-count and formatted print (CSV) to stdout.
Exporting rule hit count to CSV using GO#
Look at the following GO snippet
Let's comment some parts of the code.
This is how you define a Firewall struct for your PAN-OS device and how you Initialize() it to be able to use its Op() function.
The Op(req interface{}, vsys string, extras interface{}, ans interface{}) method executes the API req (a string representing a XML command in this case) and returns a []byte slice with the XML response (we're discarding it). In case you provide a non-nil value to the ans parameter (we are providing &response) then the Op() function will try to unmarshall the XML response in your structure.
The following is the struct of the response we're passing to Op(). Using this strategy we leverage pan-go's XML parser and data conversion features simplifying data processing on our end.
The last part of the code is a simple sort and CSV print to stdout code.
Summary#
Many automation use cases require the DevOps engineers to be able to extract runtime data from PAN-OS firewalls. Any data that can obtained from CLI commands can also be accessed programmatically using the XML API.
Although the API can be consumed directly (i.e. using curl commands) it is highly recommended to leverage any available SDK. In this tutorial we've introduced both the pan-python and pan-go SDK's and highlined the benefits of using them (i.e. XML response parsing and data type conversion)