Exporting Rulebases to CSV

note

In this tutorial we will implement a tool to export Security and NAT Rulebases from PAN-OS NGFW in CSV format, using the API (what else?). We will build the tool using Go.

Requirements

To follow this tutorial, it is recommended that that you are familiar with the concepts of Palo Alto Networks Next-Generation Firewalls, Security Policies and APIs.

Make sure you have a Palo Alto Networks Next-Generation Firewall deployed and that you have administrative access to its Management interface via HTTPS. To avoid potential disruptions, it's recommended to run all the tests on a non-production environment.

We will use Go in this tutorial, but the same concepts can be reused in Python using PAN Device or pan-python.

Our tool: pan-export

Hello World! (pango version)

Let's start from basics, just some code to start a connection to PAN-OS API:

package main

import (
    "log"

    "github.com/PaloAltoNetworks/pango"
)

func main() {
    var err error

    c := &pango.Firewall{Client: pango.Client{
        Hostname: "192.168.10.1",
        Username: "admin",
        Password: "admin",
        Logging:  pango.LogAction | pango.LogOp,
    }}
    if err = c.Initialize(); err != nil {
        log.Printf("Failed to initialize client: %s", err)
        return
    }
    log.Printf("Initialize ok")
}

Change the values in the highlighted lines to adapt the code to your environment. Save the code in panos-export.go file and run:

go get github.com/PaloAltoNetworks/pango
go run panos-export.go

The output should look like:


2019/11/21 11:42:44 192.168.10.1: Retrieving API key
2019/11/21 11:42:45 (op) show system info
2019/11/21 11:42:45 Initialize ok

Before moving forward

Let's remove those ugly, against-all-the-security-best-practices constants from the code and move them to the command line as arguments:

package main

import (
    "flag"
    "log"
    "os"

    "github.com/PaloAltoNetworks/pango"
)

func main() {
    var err error

    hostnamePtr := flag.String("hostname", "", "PAN-OS NGFW hostname (Required)")
    usernamePtr := flag.String("username", "", "PAN-OS NGFW username")
    passwordPtr := flag.String("password", "", "PAN-OS NGFW password")
    apiKeyPtr := flag.String("apikey", "", "PAN-OS NGFW API Key")
    verifyCertificate := flag.Bool("k", false, "Skip PAN-OS certificate verification")
    flag.Parse()

    if *hostnamePtr == "" {
        flag.PrintDefaults()
        os.Exit(1)
    }
    if *apiKeyPtr == "" && (*passwordPtr == "" || *usernamePtr == "") {
        log.Printf("apikey or username & password should be specified")
        os.Exit(1)
    }

    c := &pango.Firewall{Client: pango.Client{
        Hostname:          *hostnamePtr,
        ApiKey:            *apiKeyPtr,
        Username:          *usernamePtr,
        Password:          *passwordPtr,
        VerifyCertificate: !*verifyCertificate,
        Logging:           pango.LogAction | pango.LogOp,
    }}
    if err = c.Initialize(); err != nil {
        log.Printf("Failed to initialize client: %s", err)
        return
    }
    log.Printf("Initialize ok")
}

note

Note that we have added the -k flag to disable certificate verification. By default certificate verification is enabled in this code. If your code was working before but now the tool fails with cannot validate certificate, you may want to add -k to your command line.

Let's try on the shell:


$ go get github.com/PaloAltoNetworks/pango
$ go run panos-export.go -hostname 192.168.10.1 -username admin -password admin
2019/11/21 11:42:44 192.168.10.1: Retrieving API key
2019/11/21 11:42:45 (op) show system info
2019/11/21 11:42:45 Initialize ok

Get the candidate rulebase

Now that we have been able to connect to the PAN-OS API we need to grab the security rulebase in order to export it. If we were using the XML API directly we could do this with the get command of the config API, with the right XPath. The result is an XML dump of the security rulebase.

Sample shell session:

$ export API_KEY="<Your API Key>"
$ export ELEMENT_XPATH="/config/devices/entry[@name='localhost.localdomain']/vsys/entry[@name='vsys1']/rulebase/security"
$ curl -g "https://<firewall>/api/?key=$API_KEY&type=config&action=show&xpath=$ELEMENT_XPATH"
<response status="success"><result><security>
  <rules>
    <entry name="Block Unknowns" uuid="6dff2589-28cb-48d2-8854-46c90d95d50e">
      <to>
        <member>External</member>
      </to>
[...]

PAN Go makese our life lot easier by wrapping all of this in some handy abstractions. We can just use the client.Policies.Security namespace to get access to the candidate Security Rulebase:

package main

import (
    "flag"
    "log"
    "os"

    "github.com/PaloAltoNetworks/pango"
)

func main() {
    var err error

    hostnamePtr := flag.String("hostname", "", "PAN-OS NGFW hostname (Required)")
    usernamePtr := flag.String("username", "", "PAN-OS NGFW username")
    passwordPtr := flag.String("password", "", "PAN-OS NGFW password")
    apiKeyPtr := flag.String("apikey", "", "PAN-OS NGFW API Key")
    verifyCertificate := flag.Bool("k", false, "Skip PAN-OS certificate verification")
    flag.Parse()

    if *hostnamePtr == "" {
        flag.PrintDefaults()
        os.Exit(1)
    }
    if *apiKeyPtr == "" && (*passwordPtr == "" || *usernamePtr == "") {
        log.Printf("apikey or username & password should be specified")
        os.Exit(1)
    }

    c := &pango.Firewall{Client: pango.Client{
        Hostname:          *hostnamePtr,
        ApiKey:            *apiKeyPtr,
        Username:          *usernamePtr,
        Password:          *passwordPtr,
        VerifyCertificate: !*verifyCertificate,
        Logging:           pango.LogAction | pango.LogOp,
    }}
    if err = c.Initialize(); err != nil {
        log.Printf("Failed to initialize client: %s", err)
        return
    }
    log.Printf("Initialize ok")

    policies, err := c.Policies.Security.GetList("vsys1")
    if err != nil {
        log.Printf("Failed retrieving the security rulebase: %s", err)
        return
    }

    for _, policy := range policies {
        entry, err := c.Policies.Security.Get("vsys1", policy)
        if err != nil {
            log.Printf("Failed retrieving details for policy %s: %s", policy, err)
            continue
        }

        log.Printf("%+v", entry)
    }
}

First we grab the list of policy names with client.Security.Policies.GetList, and then we iterate over it to retrieve the details of each single policy with client.Security.Policies.Get.

On the shell:


$ go run pan-export.go -hostname 192.168.10.1 -password admin -username admin
2019/11/21 13:34:03 192.168.10.1: Retrieving API key
2019/11/21 13:34:04 (op) show system info
2019/11/21 13:34:05 Initialize ok
2019/11/21 13:34:05 {Name:Block Unknowns Type: Description: Tags:[] SourceZones:[InternalL3] SourceAddresses:[any] NegateSource:false SourceUsers:[any] HipProfiles:[any] DestinationZones:[External] DestinationAddresses:[any] NegateDestination:false Applications:[unknown-tcp unknown-udp] Services:[any] Categories:[any] Action:reset-client LogSetting: LogStart:true LogEnd:false Disabled:false Schedule: IcmpUnreachable:false DisableServerResponseInspection:false Group: Targets:map[] NegateTarget:false Virus: Spyware: Vulnerability: UrlFiltering: FileBlocking: WildFireAnalysis: DataFiltering:}
[...]

Checking for pending changes

With our code we are retrieving the candidate rulebase. It would be nice if the tool could also generate a warning when there are pending changes, to notify the user that running config may be out sync with the candidate config we are exporting.

We can perform this check using the op command show config list changes and then look if there are pending changes on the security rulebase. An easy way to check the response schema of the command is using the API web UI available on PAN-OS at https://<firewall>/api:

Op Command on API UI

note

We use the following op command to select only pending changes affecting policies and objects: <show><config><list><changes><partial><device-and-network>excluded</device-and-network><shared-object>excluded</shared-object></partial></changes></list></config></show>

Result:

Op Command result

In PAN Go we can use the native XML support built into Go runtime to help unmarshaling the response. The PAN Go code to run the command and check the result looks like this:

func checkForChanges(c *pango.Firewall) {
    const ChangesCmd = "<show><config><list><changes><partial>" +
        "<device-and-network>excluded</device-and-network>" +
        "<shared-object>excluded</shared-object>" +
        "</partial></changes></list></config></show>"
    var err error
    type Entry struct {
        XMLName xml.Name `xml:"entry"`
        XPath   string   `xml:"xpath"`
    }
    type Response struct {
        XMLName xml.Name `xml:"response"`
        Entries []Entry  `xml:"result>journal>entry"`
    }
    ans := Response{Entries: nil}

    _, err = c.Op(ChangesCmd, "vsys1", nil, &ans)
    if err != nil {
        log.Printf("Error checking for changes: %s", err)
        return
    }

    for _, e := range ans.Entries {
        if strings.Contains(e.XPath, "rulebase/security") {
            log.Printf("WARNING: pending changes for security rulebase in candidate config")
            return
        }
    }
}

When changes are detected the output looks like:


$ go run pan-export.go -hostname 192.168.10.1 -password admin -username admin
2019/11/21 14:09:52 192.168.10.1: Retrieving API key
2019/11/21 14:09:53 (op) show system info
2019/11/21 14:09:53 Initialize ok
2019/11/21 14:09:54 WARNING: pending changes for security rulebase in candidate config
[...]

Dump to CSV

Going back to our rulebase, we now have all the rules - we just need to convert them into CSV format. Easily done using the Go package csv. Final result:

package main

import (
    "encoding/csv"
    "encoding/xml"
    "flag"
    "log"
    "os"
    "strconv"
    "strings"

    "github.com/PaloAltoNetworks/pango"
    "github.com/PaloAltoNetworks/pango/poli/security"
)

func translate(e security.Entry) []string {
    var result []string

    result = append(
        result,
        e.Name,
        e.Type,
        e.Description,
        strings.Join(e.Tags, ", "),
        strings.Join(e.SourceZones, ", "),
        strings.Join(e.SourceAddresses, ", "),
        strconv.FormatBool(e.NegateSource),
        strings.Join(e.DestinationZones, ", "),
        strings.Join(e.DestinationAddresses, ", "),
        strconv.FormatBool(e.NegateDestination),
        strings.Join(e.Applications, ", "),
        strings.Join(e.Services, ", "),
        strings.Join(e.Categories, ", "),
        e.Action,
        e.LogSetting,
        strconv.FormatBool(e.LogStart),
        strconv.FormatBool(e.LogEnd),
        strconv.FormatBool(e.Disabled),
        e.Schedule,
        strconv.FormatBool(e.IcmpUnreachable),
        strconv.FormatBool(e.DisableServerResponseInspection),
        e.Virus,
        e.Spyware,
        e.Vulnerability,
        e.UrlFiltering,
        e.FileBlocking,
        e.WildFireAnalysis,
        e.DataFiltering,
    )

    return result
}

func checkForChanges(c *pango.Firewall) {
    const ChangesCmd = "<show><config><list><changes><partial>" +
        "<device-and-network>excluded</device-and-network>" +
        "<shared-object>excluded</shared-object>" +
        "</partial></changes></list></config></show>"
    var err error
    type Entry struct {
        XMLName xml.Name `xml:"entry"`
        XPath   string   `xml:"xpath"`
    }
    type Response struct {
        XMLName xml.Name `xml:"response"`
        Entries []Entry  `xml:"result>journal>entry"`
    }
    ans := Response{Entries: nil}

    _, err = c.Op(ChangesCmd, "vsys1", nil, &ans)
    if err != nil {
        log.Printf("Error checking for changes: %s", err)
        return
    }

    for _, e := range ans.Entries {
        if strings.Contains(e.XPath, "rulebase/security") {
            log.Printf("WARNING: pending changes for security rulebase in candidate config")
            return
        }
    }
}

func main() {
    var err error

    ruleFields := []string{
        "Name", "Type", "Description", "Tags",
        "Source Zones", "Source Addresses", "Negate Source",
        "Destination Zones", "Destination Addresses", "Negate Destination",
        "Applications", "Services", "Categories",
        "Action",
        "Log Setting", "Log Start", "Log End", "Disabled",
        "Schedule", "ICMP Unreachable", "DSRI",
        "Virus", "Spyware", "Vulnerability", "Url Filtering",
        "File Blocking", "WildFire Analysis", "Data Filtering",
    }

    hostnamePtr := flag.String("hostname", "", "PAN-OS NGFW hostname (Required)")
    usernamePtr := flag.String("username", "", "PAN-OS NGFW username")
    passwordPtr := flag.String("password", "", "PAN-OS NGFW password")
    apiKeyPtr := flag.String("apikey", "", "PAN-OS NGFW API Key")
    verifyCertificate := flag.Bool("k", false, "Skip PAN-OS certificate verification")
    flag.Parse()

    if *hostnamePtr == "" {
        flag.PrintDefaults()
        os.Exit(1)
    }
    if *apiKeyPtr == "" && (*passwordPtr == "" || *usernamePtr == "") {
        log.Printf("apikey or username & password should be specified")
        os.Exit(1)
    }

    c := &pango.Firewall{Client: pango.Client{
        Hostname:          *hostnamePtr,
        ApiKey:            *apiKeyPtr,
        Username:          *usernamePtr,
        Password:          *passwordPtr,
        VerifyCertificate: !*verifyCertificate,
        Logging:           pango.LogAction | pango.LogOp,
    }}
    if err = c.Initialize(); err != nil {
        log.Printf("Failed to initialize client: %s", err)
        return
    }
    log.Printf("Initialize ok")

    checkForChanges(c)

    policies, err := c.Policies.Security.GetList("vsys1")
    if err != nil {
        log.Printf("Failed retrieving the security rulebase: %s", err)
        return
    }

    w := csv.NewWriter(os.Stdout)
    w.Write(ruleFields)

    for _, policy := range policies {
        entry, err := c.Policies.Security.Get("vsys1", policy)
        if err != nil {
            log.Printf("Failed retrieving details for policy %s: %s", policy, err)
            continue
        }

        w.Write(translate(entry))
    }

    w.Flush()
}

On the shell:


$ go run pan-export.go -hostname 192.168.10.1 -password admin -username admin > security.csv
2019/11/21 15:38:23 192.168.10.1: Retrieving API key
2019/11/21 15:38:23 (op) show system info
2019/11/21 15:38:24 Initialize ok
2019/11/21 15:38:24 WARNING: pending changes for security rulebase in candidate config
$
$ head -2 security.csv
Name,Type,Description,Tags,Source Zones,Source Addresses,Negate Source,Destination Zones,
Destination Addresses,Negate Destination,Applications,Services,Categories,Action,Log Setting,
Log Start,Log End,Disabled,Schedule,ICMP Unreachable,DSRI,Virus,Spyware,Vulnerability,
Url Filtering,File Blocking,WildFire Analysis,Data Filtering
rule1-1,,,,InternalL3,any,false,External,any,false,any,any,,allow,,false,true,false,,false,
false,default,strict,default,LogAll,,default,

Build it for multiple platforms

One amazing thing of Go is the support for multiple platform with the same code and toolchain. You can build the tool for multiple platform/OS on the same devel environment without special tools. Just set GOOS and GOARCH environment variables to what you need before running go build. Example, for building for Linux, Mac OS X, Raspberry PI and Windows:

env GOOS=linux GOARCH=amd64 go build -o pan-export.linux pan-export.go
env GOOS=darwin GOARCH=amd64 go build -o pan-export.macosx pan-export.go
env GOOS=linux GOARCH=arm go build -o pan-export.rpi pan-export.go
env GOOS=windows GOARCH=amd64 go build -o pan-export.exe pan-export.go

PAN-OS

Cortex

Cortex XSOAR

PAN-OS

Cortex

Cortex XSOAR

Exporting Rulebases to CSV

note

Requirements

Our tool: pan-export

Hello World! (pango version)

Before moving forward

note

Get the candidate rulebase

Checking for pending changes

note

Dump to CSV

Build it for multiple platforms

PAN-OS

Cortex

Cortex XSOAR

PAN-OS

Cortex

Cortex XSOAR

note

#Requirements

#Our tool: pan-export

#Hello World! (pango version)

#Before moving forward

note

#Get the candidate rulebase

#Checking for pending changes

note

#Dump to CSV

#Build it for multiple platforms

Requirements

Our tool: pan-export

Hello World! (pango version)

Before moving forward

Get the candidate rulebase

Checking for pending changes

Dump to CSV

Build it for multiple platforms