Integrity implementation of RPZ

Hi Tim, 
Sorry for late on this as I was digging for 15 years history to understand why was this implemented this way. Unfortunately my history here is only 3 years so I had to reach out to Robert Barnhardt (SvP) to get the response and here is Robert’s response. 

Response Policies in Integrity were implemented (-- 15 years ago!) exclusively for customers who were managing or planned to manage flat lists, i.e. blacklist, whitelist.  There was little awareness of BIND's RPZ as a first-class feature in our customer base at the time: RPZ was simply leveraged as the implementation behind those lists.  That's one reason we applied our own terminology ("Response Policy" and "Response Policy Item" vs. Response Policy Zone and... er, "Resource Records with special meanings") as an abstraction: because it wasn't meant to be RPZ, it simply used RPZ under the hood.  At the same time, we kept our terminology close to RPZ because we had a vision of updating the implementation to include "real" RPZs as well, which would have given our users RP Lists (~easy mode), RP Zones (~ expert mode), and then I guess the Threat Protection feed on top of that.However, to date, there's been relatively little demand for that ("expert mode") middle ground.  One challenge there is that -- as you said -- RPZ is much more flexible, and part of that involves the semantic meanings of existing RRs being "adapted" in ways both straightforward and ...less straightforward to the application of policy.  Do we simply let users define DNS Zones that could be published as RPZ (-- requiring users understand the semantics of RRs in an RPZ), or would we try to abstract that away?  The former is much easier, but the audience for that would be more limited.(There's another technical reason.  Because we were told to handle lists of up to 4 million items, the most efficient way to store and update these lists was definitely not as full Resource Records.  The Integrity mechanisms for persisting, managing and deploying RP Lists are actually completely distinct from those of Zones/Resource Records...)

Thanks. 

Are there any guides to doing this in BlueCat?  I’m guessing it’s similar to adding custon feeds which require using DNS Raw options but having a guide would make it easier to implement and test.

I think this was a hypothetical.  There’s currently no way to do this outside of aggressively hacking up named.conf via postDeploy.sh and/or completely managing RPZ outside of Integrity with include files or what not.

​@cgnanapragasam  thank you for the background from Robert.  It’s good to know how it came about, but unfortunately the 15 year old implementation is showing its age.  I would love to see this part of the product re-imagined so to speak - but I understand that doing so without probably more customer demand than currently exists probably doesn’t make sense for Bluecat.

All my non-BlueCat DNS servers use RPZ with our own threat feed.  I need that on the BlueCat servers, but they won’t accept a zone transfer of my RPZ zones, so I need to write an API script someday to load and update that data in real time.  Most of my RPZ zones are just allow/deny, but I would like the option to use other RPZ actions in certain cases/

I’ve been able to use the ioc2rpz feeds with BlueCat.  I had to do some manual setup but it works.

Yeah I’d imagine you could get secondary (feed) RPZs set up without too much fuss with raw options - but it probably would get more complicated quickly if you need to also support Integrity-driven/hosted RPZs as well.

​@dittman   Any clues on how you did that would be appreciated.

There’s an internal document that support should be able to provide.  If not let me know and I’ll try to dig it up.

Hey Guys,

So we are currently considering a ThreatProtect+ feature which may include more advanced capabilites around RPZ specifically, as Robert’s commentary points out it very easy mode at the moment verses full RPZ control. We’re considering adding DoH/DoT/DGA detection/Tunnelling 

Now what I can share is something I wrote awhile ago to inject a pihole flat domain list into a response policy items list, we have a number of customer that have to utilise free domains lists from gov entities etc. I didn’t have access to their list so I just pulled a flat block list normally used for piHole
 

import requests
from io import BytesIO
import urllib3
import json

# Suppress SSL warnings for self-signed certs
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

url = "https://192.168.1.2/api/v2/responsePolicies/100964/imports"

# Remote URI for domains.txt
domains_uri = "https://hole.cert.pl/domains/v2/domains.txt"

# Download domains.txt from URI
domains_response = requests.get(domains_uri)
domains_response.raise_for_status()

# Create in-memory JSON for blob
blob_content = {"type": "ResponsePolicyItemImport", "name": "domains.txt"}
blob_bytes = BytesIO(json.dumps(blob_content).encode('utf-8'))

files = [
    ('fileInfo', ('blob', blob_bytes, 'application/json')),
    ('fileData', ('domains.txt', BytesIO(domains_response.content), 'text/plain'))
]

headers = {
    'Authorization': 'Basic YWRtaW46Tm02aEFGREd2N1R0dGUyRENMV2RieGk0RWJac2hUckFkekxPY2lMYg=='
}

# Make the request
response = requests.post(url, headers=headers, files=files, verify=False)

print(response.text)

End result 62K blocking items loaded dynamically into Integrity X 25.1
You’d need to update the url string to the RP policy and update the basic auth header
 

Brian
 

Don’t you love it when as the VP of a Product Line you’re post gets sent to moderation?!? Let me start again

We’re considering post 26.1 an updated ThreatProtection+ module with more enhanced RPZ, along with DOT/DOH/DGA detection and tunneling. Would welcome any security related requirements from the community to help guide that effort

In the mean time, I wrote something for another customer who had to injest a flat domain list into a TP list as a requirement
 

import requests
from io import BytesIO
import urllib3
import json

# Suppress SSL warnings for self-signed certs
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

url = "https://192.168.1.2/api/v2/responsePolicies/100964/imports"

# Remote URI for domains.txt
domains_uri = "https://hole.cert.pl/domains/v2/domains.txt"

# Download domains.txt from URI
domains_response = requests.get(domains_uri)
domains_response.raise_for_status()

# Create in-memory JSON for blob
blob_content = {"type": "ResponsePolicyItemImport", "name": "domains.txt"}
blob_bytes = BytesIO(json.dumps(blob_content).encode('utf-8'))

files = [
    ('fileInfo', ('blob', blob_bytes, 'application/json')),
    ('fileData', ('domains.txt', BytesIO(domains_response.content), 'text/plain'))
]

headers = {
    'Authorization': 'Basic YWRtaW46Tm02aEFGREd2N1R0dGUyRENMV2RieGk0RWJac2hUckFkekxPY2lMYg=='
}

# Make the request
response = requests.post(url, headers=headers, files=files, verify=False)

print(response.text)

Result 62k of domains added to a block list, its dynamic too, so if an entry is removed from the listing its removed from the response policy items
 

Your mileage may vary, and acknowledge this isn’t an RPZ zone

Do you have anything similar for redirects?

Since this is just bulk-populating Response Policy Items on a Response Policy, you can use this method on any Response Policy regardless of type - e.g. redirect, block list, black hole,  or allow list.

​@bshorland thanks for the info.  If the proposed ThreatProtection+ functionality (or, specifically, whatever enables one to use RPZ as it is intended to be used) requires a Threat Protection subscription (or any additional add on license) then my knee jerk reaction is “why do I need to pay more to fully use built in BIND functionality?”.  Understanding that Bluecat is in the business of selling software, it still hurts my soul to pay for things like RPZ.  I’d hate to see Bluecat go the way of Infoblox where you get nickled and dimed on licenses for every little piece of functionality.

Thanks.  I thought redirects were treated differently but that may just be from having used the current BC implementation.