We'd spent three weeks on a "neighborhood quality" feature. A junior modeler built it from permit records, school attendance zones, and a walkability score pulled from a vendor API. It tested beautifully. Held out against our 2023 sales file, it dropped error on the mid-tier suburban segment enough that someone started sketching out the changelog entry before the meeting even wrapped.
Then someone asked the question we'd all skipped: what's actually in this thing.
Turned out the walkability score was built off census tract boundaries that, in our metro, line up almost exactly with zip code splits. The school attendance zones did too. Permit density clusters by jurisdiction, which also tracks zip. We hadn't built a neighborhood quality signal. We'd built a very expensive zip code lookup with three extra steps between it and the model.
This is the trap that's easy to lecture about and hard to catch in practice: feature leakage through a proxy variable. Not the textbook kind, where you accidentally include the target itself. The quieter kind, where you build something that feels like new information but is really an old field wearing a costume. Zip code is already in most AVMs somewhere, directly or baked into the comps logic as a fixed effect. If your new feature is 90% explained by zip, you haven't added signal. You've added collinearity and a harder path to audit on the way to the same number.
The fair lending angle makes this worse than a plain modeling inefficiency. Zip code in a lot of metros correlates with protected-class composition closely enough that a reviewer looking at your feature importances is going to ask the same question we nearly didn't: why does this variable behave almost identically to geography, and what is geography standing in for here. "It improved RMSE" doesn't hold up well as an answer to that.
We caught it because someone ran a cheap test: regress the candidate feature on zip code fixed effects alone and check the R-squared. Ours came back north of 0.85, which doesn't leave room for a neighborhood quality story. It's zip code wearing a university logo.
What's frustrating is that the instinct behind the feature was right. Curb appeal, upkeep, how a block looks day to day, that's real signal and it's genuinely missing from most tabular models. The mistake was reaching for an administrative boundary as a shortcut to get there. A census tract or a zip code is a postal and legal convenience. It was never drawn to describe how a parcel or its block looks on the ground, and using it as a stand-in for "neighborhood character" is how you end up quietly rebuilding a variable you already have, at more expense, with worse paperwork.
On the next pass, we went and got the actual thing instead of a boundary that happened to correlate with it. Condition, landscaping, how dense the infill is, whether the lots around a parcel are kept up or sliding, those details live on the ground, not inside a zip code polygon drawn for mail routing decades ago. It's roughly the gap we've been filling by looking at what AVM Imagery does with it: a per-property vector built from a learned read of the parcel and its immediate surroundings, keyed to your own record IDs instead of a census or postal boundary file. It doesn't carry zip code's fingerprints because it isn't built from anything zip-shaped to start with.
We shipped a smaller version of the feature that quarter. It had less lift than the one that got everyone excited in the review, but no proxy problem, and an audit trail that didn't need anyone to explain away a suspicious R-squared.
Run the regression before your next feature review ships, not after. If a new field correlates with your geography fixed effects at 0.85, it is geography, whatever you called it in the sprint.
If you're scoping a real curb-appeal or context feature instead of a disguised one, that's the gap AVM Imagery was built to fill.
