Address Manager notification processing

Hey Tim,

Any AAAA record updates occurring? There is a known performance issue with the processing of AAAA in particular that can cause a large slow down on the processing of the notifications, this has yet to be addressed

Brian

Hi Brian - no, no AAAA updates.  Our PNA monitoring scripts (which alert if there are > 5 PNA files on BAM) have been triggering more often lately, and I’ve started looking into database optimization (we have about 700k orphaned MACs that need to be cleaned up) and perhaps a reindexing is due.  The past month or two we’ve had, usually on Monday mornings, a backlog of up to around 50-75 PNA files, but last week we went from 30 to over 350 in the space of an hour.  It took almost 24hrs for BAM to get caught up.  We don’t currently watch the notification-related SNMP OIDs so I’m not really sure if our volume of notifications spiked or something else occurred (there were some deadlock messages logged by postgres but nothing that lined up directly at this time).  For this reason, I really like the 25.1 notification-related stats now graphed in BAM so looking forward to getting there next year.  

But, as previously mentioned, it rankles a bit to see the BAM at overall ~95-98% idle with one thread pegged while it struggles to dig itself out of the hole all day.

Hi Tim, damn was hoping it was going to be something at least known … I would recommend in this instance getting a Care case opened up so Defenders & Pheonix teams can start looking at the root cause of what’s occurring, it could be as simple as the orphaned MACs slowing processing but those are supposed to be cleaned up during the audit data retention process automatically I thought since 9.3 days

Brian

Thanks Brian.  I woke up just now to an alert for 584 PNA files so something is definitely amiss.  I’ll be opening that case shortly.  I already have a case open to deal with the 700k orphaned macs (my theory is that they exist pre-implementation of the 9.3 cleanup behavior and so I need to get to a “clean slate” first to allow for normal automatic maintenance).

Follow on question for ​@bshorland or anyone else who may know - is the Notifications Per Second stat in an Integrity X BAM dashboard the number of Notifications received (and potentially queued) by BAM, or the number of notifications processed by BAM?  

AFAIK ​@tmaestas 

The notifications per second metric refers to the number of notifications processed by Address Manager per second. For reference, the upper limit for processing notifications on a BAM8500 is up to 700 notifications per second

Brian