Exporting Rulebases to CSV

note

In this tutorial we will implement a tool to export Security and NAT Rulebases from PAN-OS NGFW in CSV format, using the API (what else?). We will build the tool using Go.

Requirements#

To follow this tutorial, it is recommended that that you are familiar with the concepts of Palo Alto Networks Next-Generation Firewalls, Security Policies and APIs.

Make sure you have a Palo Alto Networks Next-Generation Firewall deployed and that you have administrative access to its Management interface via HTTPS. To avoid potential disruptions, it's recommended to run all the tests on a non-production environment.

We will use Go in this tutorial, but the same concepts can be reused in Python using PAN-OS-Python or pan-python.

Our tool: pan-export#

Hello World! (pango version)#

Let's start from basics, just some code to start a connection to PAN-OS API:

package main
import (    "log"
    "github.com/PaloAltoNetworks/pango")
func main() {    var err error
    c := &pango.Firewall{Client: pango.Client{        Hostname: "192.168.10.1",        Username: "admin",        Password: "admin",        Logging:  pango.LogAction | pango.LogOp,    }}    if err = c.Initialize(); err != nil {        log.Printf("Failed to initialize client: %s", err)        return    }    log.Printf("Initialize ok")}

Change the values in the highlighted lines to adapt the code to your environment. Save the code in panos-export.go file and run:

go get github.com/PaloAltoNetworks/pangogo run panos-export.go

The output should look like:

2019/11/21 11:42:44 192.168.10.1: Retrieving API key2019/11/21 11:42:45 (op) show system info2019/11/21 11:42:45 Initialize ok

Before moving forward#

Let's remove those ugly, against-all-the-security-best-practices constants from the code and move them to the command line as arguments:

package main
import (    "flag"    "log"    "os"
    "github.com/PaloAltoNetworks/pango")
func main() {    var err error
    hostnamePtr := flag.String("hostname", "", "PAN-OS NGFW hostname (Required)")    usernamePtr := flag.String("username", "", "PAN-OS NGFW username")    passwordPtr := flag.String("password", "", "PAN-OS NGFW password")    apiKeyPtr := flag.String("apikey", "", "PAN-OS NGFW API Key")    verifyCertificate := flag.Bool("k", false, "Skip PAN-OS certificate verification")    flag.Parse()
    if *hostnamePtr == "" {        flag.PrintDefaults()        os.Exit(1)    }    if *apiKeyPtr == "" && (*passwordPtr == "" || *usernamePtr == "") {        log.Printf("apikey or username & password should be specified")        os.Exit(1)    }
    c := &pango.Firewall{Client: pango.Client{        Hostname:          *hostnamePtr,        ApiKey:            *apiKeyPtr,        Username:          *usernamePtr,        Password:          *passwordPtr,        VerifyCertificate: !*verifyCertificate,        Logging:           pango.LogAction | pango.LogOp,    }}    if err = c.Initialize(); err != nil {        log.Printf("Failed to initialize client: %s", err)        return    }    log.Printf("Initialize ok")}

note

Note that we have added the -k flag to disable certificate verification. By default certificate verification is enabled in this code. If your code was working before but now the tool fails with cannot validate certificate, you may want to add -k to your command line.

Let's try on the shell:

$ go get github.com/PaloAltoNetworks/pango$ go run panos-export.go -hostname 192.168.10.1 -username admin -password admin2019/11/21 11:42:44 192.168.10.1: Retrieving API key2019/11/21 11:42:45 (op) show system info2019/11/21 11:42:45 Initialize ok

Get the candidate rulebase#

Now that we have been able to connect to the PAN-OS API we need to grab the security rulebase in order to export it. If we were using the XML API directly we could do this with the get command of the config API, with the right XPath. The result is an XML dump of the security rulebase.

Sample shell session:

$ export API_KEY="<Your API Key>"$ export ELEMENT_XPATH="/config/devices/entry[@name='localhost.localdomain']/vsys/entry[@name='vsys1']/rulebase/security"$ curl -g "https://<firewall>/api/?key=$API_KEY&type=config&action=show&xpath=$ELEMENT_XPATH"<response status="success"><result><security>  <rules>    <entry name="Block Unknowns" uuid="6dff2589-28cb-48d2-8854-46c90d95d50e">      <to>        <member>External</member>      </to>[...]

PAN Go makese our life lot easier by wrapping all of this in some handy abstractions. We can just use the client.Policies.Security namespace to get access to the candidate Security Rulebase:

package main
import (    "flag"    "log"    "os"
    "github.com/PaloAltoNetworks/pango")
func main() {    var err error
    hostnamePtr := flag.String("hostname", "", "PAN-OS NGFW hostname (Required)")    usernamePtr := flag.String("username", "", "PAN-OS NGFW username")    passwordPtr := flag.String("password", "", "PAN-OS NGFW password")    apiKeyPtr := flag.String("apikey", "", "PAN-OS NGFW API Key")    verifyCertificate := flag.Bool("k", false, "Skip PAN-OS certificate verification")    flag.Parse()
    if *hostnamePtr == "" {        flag.PrintDefaults()        os.Exit(1)    }    if *apiKeyPtr == "" && (*passwordPtr == "" || *usernamePtr == "") {        log.Printf("apikey or username & password should be specified")        os.Exit(1)    }
    c := &pango.Firewall{Client: pango.Client{        Hostname:          *hostnamePtr,        ApiKey:            *apiKeyPtr,        Username:          *usernamePtr,        Password:          *passwordPtr,        VerifyCertificate: !*verifyCertificate,        Logging:           pango.LogAction | pango.LogOp,    }}    if err = c.Initialize(); err != nil {        log.Printf("Failed to initialize client: %s", err)        return    }    log.Printf("Initialize ok")
    policies, err := c.Policies.Security.GetList("vsys1")    if err != nil {        log.Printf("Failed retrieving the security rulebase: %s", err)        return    }
    for _, policy := range policies {        entry, err := c.Policies.Security.Get("vsys1", policy)        if err != nil {            log.Printf("Failed retrieving details for policy %s: %s", policy, err)            continue        }
        log.Printf("%+v", entry)    }}

First we grab the list of policy names with client.Security.Policies.GetList, and then we iterate over it to retrieve the details of each single policy with client.Security.Policies.Get.

On the shell:

$ go run pan-export.go -hostname 192.168.10.1 -password admin -username admin2019/11/21 13:34:03 192.168.10.1: Retrieving API key2019/11/21 13:34:04 (op) show system info2019/11/21 13:34:05 Initialize ok2019/11/21 13:34:05 {Name:Block Unknowns Type: Description: Tags:[] SourceZones:[InternalL3] SourceAddresses:[any] NegateSource:false SourceUsers:[any] HipProfiles:[any] DestinationZones:[External] DestinationAddresses:[any] NegateDestination:false Applications:[unknown-tcp unknown-udp] Services:[any] Categories:[any] Action:reset-client LogSetting: LogStart:true LogEnd:false Disabled:false Schedule: IcmpUnreachable:false DisableServerResponseInspection:false Group: Targets:map[] NegateTarget:false Virus: Spyware: Vulnerability: UrlFiltering: FileBlocking: WildFireAnalysis: DataFiltering:}[...]

Checking for pending changes#

With our code we are retrieving the candidate rulebase. It would be nice if the tool could also generate a warning when there are pending changes, to notify the user that running config may be out sync with the candidate config we are exporting.

We can perform this check using the op command show config list changes and then look if there are pending changes on the security rulebase. An easy way to check the response schema of the command is using the API web UI available on PAN-OS at https://<firewall>/api:

Op Command on API UI

note

We use the following op command to select only pending changes affecting policies and objects: <show><config><list><changes><partial><device-and-network>excluded</device-and-network><shared-object>excluded</shared-object></partial></changes></list></config></show>

Result:

Op Command result

In PAN Go we can use the native XML support built into Go runtime to help unmarshaling the response. The PAN Go code to run the command and check the result looks like this:

func checkForChanges(c *pango.Firewall) {    const ChangesCmd = "<show><config><list><changes><partial>" +        "<device-and-network>excluded</device-and-network>" +        "<shared-object>excluded</shared-object>" +        "</partial></changes></list></config></show>"    var err error    type Entry struct {        XMLName xml.Name `xml:"entry"`        XPath   string   `xml:"xpath"`    }    type Response struct {        XMLName xml.Name `xml:"response"`        Entries []Entry  `xml:"result>journal>entry"`    }    ans := Response{Entries: nil}
    _, err = c.Op(ChangesCmd, "vsys1", nil, &ans)    if err != nil {        log.Printf("Error checking for changes: %s", err)        return    }
    for _, e := range ans.Entries {        if strings.Contains(e.XPath, "rulebase/security") {            log.Printf("WARNING: pending changes for security rulebase in candidate config")            return        }    }}

When changes are detected the output looks like:

$ go run pan-export.go -hostname 192.168.10.1 -password admin -username admin2019/11/21 14:09:52 192.168.10.1: Retrieving API key2019/11/21 14:09:53 (op) show system info2019/11/21 14:09:53 Initialize ok2019/11/21 14:09:54 WARNING: pending changes for security rulebase in candidate config[...]

Dump to CSV#

Going back to our rulebase, we now have all the rules - we just need to convert them into CSV format. Easily done using the Go package csv. Final result:

package main
import (    "encoding/csv"    "encoding/xml"    "flag"    "log"    "os"    "strconv"    "strings"
    "github.com/PaloAltoNetworks/pango"    "github.com/PaloAltoNetworks/pango/poli/security")
func translate(e security.Entry) []string {    var result []string
    result = append(        result,        e.Name,        e.Type,        e.Description,        strings.Join(e.Tags, ", "),        strings.Join(e.SourceZones, ", "),        strings.Join(e.SourceAddresses, ", "),        strconv.FormatBool(e.NegateSource),        strings.Join(e.DestinationZones, ", "),        strings.Join(e.DestinationAddresses, ", "),        strconv.FormatBool(e.NegateDestination),        strings.Join(e.Applications, ", "),        strings.Join(e.Services, ", "),        strings.Join(e.Categories, ", "),        e.Action,        e.LogSetting,        strconv.FormatBool(e.LogStart),        strconv.FormatBool(e.LogEnd),        strconv.FormatBool(e.Disabled),        e.Schedule,        strconv.FormatBool(e.IcmpUnreachable),        strconv.FormatBool(e.DisableServerResponseInspection),        e.Virus,        e.Spyware,        e.Vulnerability,        e.UrlFiltering,        e.FileBlocking,        e.WildFireAnalysis,        e.DataFiltering,    )
    return result}
func checkForChanges(c *pango.Firewall) {    const ChangesCmd = "<show><config><list><changes><partial>" +        "<device-and-network>excluded</device-and-network>" +        "<shared-object>excluded</shared-object>" +        "</partial></changes></list></config></show>"    var err error    type Entry struct {        XMLName xml.Name `xml:"entry"`        XPath   string   `xml:"xpath"`    }    type Response struct {        XMLName xml.Name `xml:"response"`        Entries []Entry  `xml:"result>journal>entry"`    }    ans := Response{Entries: nil}
    _, err = c.Op(ChangesCmd, "vsys1", nil, &ans)    if err != nil {        log.Printf("Error checking for changes: %s", err)        return    }
    for _, e := range ans.Entries {        if strings.Contains(e.XPath, "rulebase/security") {            log.Printf("WARNING: pending changes for security rulebase in candidate config")            return        }    }}
func main() {    var err error
    ruleFields := []string{        "Name", "Type", "Description", "Tags",        "Source Zones", "Source Addresses", "Negate Source",        "Destination Zones", "Destination Addresses", "Negate Destination",        "Applications", "Services", "Categories",        "Action",        "Log Setting", "Log Start", "Log End", "Disabled",        "Schedule", "ICMP Unreachable", "DSRI",        "Virus", "Spyware", "Vulnerability", "Url Filtering",        "File Blocking", "WildFire Analysis", "Data Filtering",    }
    hostnamePtr := flag.String("hostname", "", "PAN-OS NGFW hostname (Required)")    usernamePtr := flag.String("username", "", "PAN-OS NGFW username")    passwordPtr := flag.String("password", "", "PAN-OS NGFW password")    apiKeyPtr := flag.String("apikey", "", "PAN-OS NGFW API Key")    verifyCertificate := flag.Bool("k", false, "Skip PAN-OS certificate verification")    flag.Parse()
    if *hostnamePtr == "" {        flag.PrintDefaults()        os.Exit(1)    }    if *apiKeyPtr == "" && (*passwordPtr == "" || *usernamePtr == "") {        log.Printf("apikey or username & password should be specified")        os.Exit(1)    }
    c := &pango.Firewall{Client: pango.Client{        Hostname:          *hostnamePtr,        ApiKey:            *apiKeyPtr,        Username:          *usernamePtr,        Password:          *passwordPtr,        VerifyCertificate: !*verifyCertificate,        Logging:           pango.LogAction | pango.LogOp,    }}    if err = c.Initialize(); err != nil {        log.Printf("Failed to initialize client: %s", err)        return    }    log.Printf("Initialize ok")
    checkForChanges(c)
    policies, err := c.Policies.Security.GetList("vsys1")    if err != nil {        log.Printf("Failed retrieving the security rulebase: %s", err)        return    }
    w := csv.NewWriter(os.Stdout)    w.Write(ruleFields)
    for _, policy := range policies {        entry, err := c.Policies.Security.Get("vsys1", policy)        if err != nil {            log.Printf("Failed retrieving details for policy %s: %s", policy, err)            continue        }
        w.Write(translate(entry))    }
    w.Flush()}

On the shell:

$ go run pan-export.go -hostname 192.168.10.1 -password admin -username admin > security.csv2019/11/21 15:38:23 192.168.10.1: Retrieving API key2019/11/21 15:38:23 (op) show system info2019/11/21 15:38:24 Initialize ok2019/11/21 15:38:24 WARNING: pending changes for security rulebase in candidate config$$ head -2 security.csvName,Type,Description,Tags,Source Zones,Source Addresses,Negate Source,Destination Zones,Destination Addresses,Negate Destination,Applications,Services,Categories,Action,Log Setting,Log Start,Log End,Disabled,Schedule,ICMP Unreachable,DSRI,Virus,Spyware,Vulnerability,Url Filtering,File Blocking,WildFire Analysis,Data Filteringrule1-1,,,,InternalL3,any,false,External,any,false,any,any,,allow,,false,true,false,,false,false,default,strict,default,LogAll,,default,

Build it for multiple platforms#

One amazing thing of Go is the support for multiple platform with the same code and toolchain. You can build the tool for multiple platform/OS on the same devel environment without special tools. Just set GOOS and GOARCH environment variables to what you need before running go build. Example, for building for Linux, Mac OS X, Raspberry PI and Windows:

env GOOS=linux GOARCH=amd64 go build -o pan-export.linux pan-export.goenv GOOS=darwin GOARCH=amd64 go build -o pan-export.macosx pan-export.goenv GOOS=linux GOARCH=arm go build -o pan-export.rpi pan-export.goenv GOOS=windows GOARCH=amd64 go build -o pan-export.exe pan-export.go