Conversation
Domains with wildcard DNS records (*.example.com) cause every predicted subdomain to resolve successfully, producing massive false positive counts. On a tested domain with 21 known subdomains, this inflated results from ~500 predictions to 2,199 "resolved" subdomains — all matching the same wildcard catch-all endpoint. This adds wildcard detection by probing random subdomains before resolving predictions. When a wildcard is detected, only predictions that resolve to IPs outside the wildcard set are returned. - Add detect_wildcard() that probes 3 random subdomains per apex - Modify get_registered_domains() to accept apex_domain for wildcard filtering - Pass apex_domain from main.py recursive resolution loop - Add tests for wildcard detection on non-wildcard domains - Backward compatible: apex_domain is optional, existing behavior preserved Co-Authored-By: WHO ELSE BUT!!!! AI-Session-Id: 8ef18548-aeed-4766-afa2-7d0cfcfcc6a2 AI-Tool: claude-code AI-Model: unknown
klaasmeinke
left a comment
There was a problem hiding this comment.
Thanks for your contribution and sorry it took me a while to get back to you!
I definetly think wildcard detection is a good idea, but was wondering whether it might be better to do this right at the start, before the expensive inference with the transformer. So first check that the targeted domains don't have wildcards. If they do, exit early with an error message. What do you think?
|
Thanks for taking a look! Agree on moving detection to the start, the probe is just a few DNS lookups, so it's cheaper to check once up front than to re-derive the wildcard set each recursion round. I'll hoist it into run(). The one part I'd push back on is exiting with an error. The motivating case was that the wildcard domain still had 21 real subdomains pointing outside the wildcard set a hard exit would return none of those, and with --multi-apex it'd also kill the clean apexes. So I'd suggest: detect up front (your idea), but warn and fall through to the IP filter instead of erroring. Can add a --wildcard filter|skip flag if you want an escape hatch. Happy to restructure the PR that way if it sounds good. |
|
Sounds good let's do that! |
|
Opened #38 with this: wildcard detection now runs once per apex up front (before inference), and instead of erroring it warns and falls through to the IP-based filter, keeping subdomains that resolve outside the wildcard set. Added a Happy to close this PR in favor of #38, or fold it in — whichever you prefer. The CI workflows on #38 are waiting on your "Approve and run workflows" since it's from a fork. |
Problem
Domains with wildcard DNS records (
*.example.com) cause every predicted subdomain to resolve successfully, producing massive false positive counts that compound through the recursive inference loop.Tested on a real domain with 21 known subdomains:
The recursive loop in
_get_domains_for_group()amplifies the problem — wildcard-resolved predictions are fed back as seeds for the next inference round, generating even more predictions that all resolve again.Fix
Adds wildcard detection to
resolve.pybefore resolving predictions:detect_wildcard(apex_domain)— probes 3 random subdomains (e.g.a8k2m9x4p1q7w3.example.com). If all resolve and share common IPs, a wildcard is present.get_registered_domains()— new optionalapex_domainparameter. When provided and a wildcard is detected, only returns predictions that resolve to at least one IP outside the wildcard set.Why IP-based filtering (not just "resolves = exists")
On wildcard domains, DNS resolution alone is meaningless — everything resolves. But real subdomains with explicit A records often point to different IPs than the wildcard. This approach catches those while filtering out the noise.
Changes
subwiz/resolve.pydetect_wildcard(),_resolve_ips()helper, updateget_registered_domains()with optionalapex_domainparamsubwiz/main.pyapex_domain=apextoget_registered_domains()in the recursive looptests/test_resolve.pyBackward Compatibility
apex_domaindefaults toNone— existing callers are unaffectedapex_domainis not provided, the original fast path (justis_registered()) is usedTest Plan
test_registered_domains— existing test passes (no regression)test_wildcard_detection_non_wildcard— confirmsdetect_wildcard()returnsNoneforhadrian.iotest_registered_domains_with_apex— confirms non-wildcard domains still resolve correctly whenapex_domainis provided*.narad.io→ CloudFront) — 2,199 false positives reduced to 0