UniFi Protect DirectConnect fails across VLANs with Windows DNS

UniFi Protect was complaining that it couldn't establish a direct connection, even though my PC and the Protect console were on the same local network. They were on different VLANs, which gave me a perfectly plausible place to start looking. The browser console gave me a better clue.

net::ERR_NAME_NOT_RESOLVED

DirectConnect could not be established,
but WebRTC connection should succeed.

Same hostname, two answers

I took the full id.ui.direct hostname from the failed request and looked it up from the affected PC. The normal Windows DNS path returned NXDOMAIN. Asking the UniFi gateway for the same name returned the correct LAN address for the Protect console.

Here's how to repeat that comparison in PowerShell. The hostname and IP addresses here are anonymised. Replace the example hostname with the full one from your failed request and 192.0.2.1 with your UniFi gateway's DNS address.

$protectName = 'example-console.id.ui.direct'

Resolve-DnsName -Name $protectName -Type A -DnsOnly
Resolve-DnsName -Name $protectName -Type A -DnsOnly -Server 192.0.2.1

That gave me a DNS failure I could reproduce without opening Protect. It also answered my question about whether the console needed to register in Windows DNS. A record for its ordinary hostname in our AD zone would have been for a different name. Protect was asking for the id.ui.direct name, and I'd already found a resolver that could answer it.

Forwarding id.ui.direct to the gateway

I added a conditional forwarder for id.ui.direct in Windows DNS and pointed it at the UniFi gateway. I briefly considered a stub zone, which helps Windows DNS discover the authoritative servers for a zone. Here I had a specific resolver that had already given me the answer I needed, so I wanted Windows DNS to send queries for that suffix directly to it. Microsoft's guide to DNS zone types covers stub zones, and its DNS forwarding guide explains how Windows selects a conditional forwarder.

To set up the same forwarder, first repeat the direct query to the gateway from the Windows DNS server. A successful query from a workstation only proves the gateway is reachable from that workstation. In DNS Manager, create a new conditional forwarder, enter id.ui.direct and the gateway's IP, and store it in Active Directory with domain replication. The PowerShell equivalent below runs in an elevated session on a writable domain controller running DNS. Replace 192.0.2.1 with your gateway's IP. Microsoft's setup instructions cover both methods.

Add-DnsServerConditionalForwarderZone `
    -Name 'id.ui.direct' `
    -MasterServers '192.0.2.1' `
    -ReplicationScope 'Domain'

Domain scope replicates the forwarder to DNS servers running on domain controllers in that domain. Each of those servers will use the gateway for matching lookups across the whole id.ui.direct suffix. If you have multiple gateways, check that the one you choose can answer for the consoles your clients need to reach. Microsoft's replication scope reference lists the available scopes.

Checking it from the client

If your clients can use several DNS servers, query each one explicitly with Resolve-DnsName -Server while checking replication. On the affected PC, you can then clear any cached negative answer and repeat the lookup through its normal DNS settings. Use the real Protect hostname in place of the example.

ipconfig /flushdns
Resolve-DnsName -Name 'example-console.id.ui.direct' -Type A -DnsOnly

The normal lookup returned the console's LAN address after the change, and I confirmed that DirectConnect worked on both Protect consoles across VLANs. I still don't know why the original DNS path started failing. The difference between the two resolver answers was enough to find a fix.