Layer 8 Lounge Recap: The Hidden Gaps in Network Visibility
If you’ve ever looked at your IPAM, your cloud console, and your monitoring tools and thought, “These cannot all be right,” this Layer 8 Lounge was very much your crowd.
For our final session of the year, we brought the community together to talk about network visibility: the messy, day-to-day work of trying to actually know what’s on the network across firewalled data centers, multi-cloud, and sprawling virtual estates.
How visible is your network, really?
We kicked things off with two simple questions:
-
When was the last time your network surprised you?
-
On a scale of 1–10, how would you rate your visibility?
Answers landed all over the spectrum:
-
A few teams rated themselves 7–8/10, usually because they’ve invested in automation – scripts that scrape router configs, pull data from cloud, and sync into IPAM on a regular schedule.
-
Others were closer to 4–5/10, especially where cloud visibility is still maturing or network discovery is limited to what’s behind DHCP.
-
And a few were somewhere around 2–3/10, dealing with years of acquisitions and inherited networks that have never been fully discovered.
The common thread: no one felt like a perfect 10. Even teams with strong processes know there are blind spots – overlapping subnets that can’t be modeled cleanly, devices behind firewalls they’re not allowed to see, or cloud segments that suddenly go dark when permissions change.
The truth problem: discovery vs. reality
From there we moved into the “truth” question: once you discover something, how do you decide what’s real?
Most teams are doing some version of:
-
Nightly or regular scripts that:
-
Pull router configs and BGP/ARP tables
-
Compare against IPAM and/or CMDB
-
Generate lists of “things that need fixing”
The challenge isn’t collecting data. It’s what happens next.
Several people described getting daily emails full of discrepancies – overlapping subnets, missing entries, stale networks – and simply not having time to reconcile them all. The issues that get fixed are the ones that either break something, trigger a complaint, or keep showing up until someone has bandwidth.
A few specific pain points came up repeatedly:
-
Overlapping subnets – especially between on-prem and cloud – that can’t be represented neatly without splitting configurations, losing relationships, or breaking downstream tooling.
-
Shadow IP space in cloud – teams in Azure or other clouds creating networks outside of the established process, then connecting them back later (if at all).
-
Ownership and accountability – discovery can show you a network, but often not who owns it, who requested it, or who should fix it.
One participant summed it up neatly:
“The stuff we don’t know is there is scarier than the overlapping subnet we do know about.”
Cloud vs. on-prem: different worlds, different rules
A clear distinction emerged between on-prem and cloud environments:
-
On-prem is often tightly controlled. Only the network team touches the routers, so visibility issues tend to show up quickly as broken routes or services.
-
Cloud feels more like the wild west. Multiple groups can create VNETS/VPCs, permissions change frequently, and visibility can disappear overnight when roles or policies are updated.
Some teams are mitigating this with strict policies:
-
Allocating defined address space ranges for cloud use
-
Blocking users from attaching their self-created networks to the corporate core without going through the network team
-
Enforcing automation-driven allocation to reduce overlapping subnets
But even with good policy, there’s still the problem of seeing what you don’t control – especially in large organizations with many independent business units or acquired companies.
Discovery methods: SNMP, SSH, APIs… and Perl
When we dove into the “how,” almost everyone is running some combination of:
-
SNMP – still widely used and still useful, but limited in depth and sometimes painful to maintain.
-
SSH and config scraping – backups, table dumps, and scripted parsing of router configs.
-
ARP / BGP table collection – to track where devices have been and which address space is actually in use.
-
API-based discovery – increasingly used for VMware and cloud platforms, with some teams trying to “piggyback” on existing monitoring tools via their APIs.
-
Homegrown scripts – often in Python, Ansible, or long-lived Perl.
Two big concerns surfaced here:
-
Security and compatibility drift
SSH ciphers and versions change, old devices can’t always be upgraded, and “fixing” one thing often means trading known bugs for unknown ones.
-
People risk: the “rockstar script maintainer” problem
Many organizations have one or two key people who wrote the discovery scripts years ago. If they leave or retire, decoding and safely evolving that code becomes a major risk and time sink.
AI: junior developer or future operator?
We closed the session looking ahead at AI and what role, if any, it should play in network discovery and reconciliation.
The mood was cautiously open, but skeptical:
-
Most participants agreed that AI could be treated like a “junior developer”: useful for suggestions, code completion, and pattern-spotting, but not something you let make changes on its own.
-
There was strong concern around:
-
Accuracy and hallucinations – “If AI can’t reliably count strawberries, do I trust it with my networks?”
-
Data provenance – where the training data comes from, and whether any network context might leak or be blended with unreliable sources.
-
Identity and ownership inference – using AI to guess who owns a network sounded attractive, but also risky without clear, auditable sources like ITSM, AD, or tagging systems.
For now, most participants see AI as a promising assistant for analysis and correlation rather than an autonomous actor in discovery or reconciliation workflows.
Thanks, and what’s next
Huge thanks to everyone who joined, shared honestly, and admitted their network isn’t a perfect 10. One of the big takeaways from this Lounge was simply this: you’re not alone. Whether you’re sitting at 8/10 or 2/10 for network discovery, everyone is dealing with a mix of segmentation, network and cloud sprawl, governance, and automation challenges.
If you couldn’t attend live:
-
We’re running a short survey on network discovery and visibility. It takes 3–5 minutes, and we’ll share the anonymized results with the community so everyone can see how their peers are tackling these problems.
-
As a small thank-you, there’s a $25 gift card (or the option to donate it to charity) for completing the survey, where your organization allows it.
-
Complete the survey: https://app.userevidence.com/surveys/ce64e3b7-099d-4258-8741-d950be61d437/share