Data Scientist & Quantitative Analyst | Model Validation, Experimentation & AI/LLM Evaluation | Relocating to the SF Bay Area

I work on whether models, metrics, and AI systems are reliable enough to support real decisions. That comes down to three things I keep returning to: credit-risk model validation, product experimentation, and applied LLM/AI evaluation. The common thread is checking when an output can be trusted, and being clear when it can't.

Right now I'm a Quantitative Analyst on Risk Analytics at CoStar. Before that I did federal analytics and RAG/LLM evaluation at Guidehouse, and research tooling at the Federal Reserve Board. I'm also pursuing a Master of Applied Science in Computer Science at Penn.

📍 Based in Boston · relocating to the San Francisco Bay Area

• Risk & model validation — validating model outputs before they inform a decision: stress testing, scenario and sensitivity analysis, model monitoring, credit-risk analytics.
• Experimentation & metrics — designing and reading A/B tests so a result reflects a real effect and not noise: power/MDE, CUPED, SRM checks, metric design.
• AI & LLM evaluation — checking whether LLM and agent workflows stay grounded and calibrated enough to act on: RAG evaluation, confidence calibration, human-in-the-loop review, failure-mode analysis.

• What it evaluates: Whether an LLM-based QA agent can review model-monitoring anomalies while staying grounded, calibrated, and reviewable by a person.
• Methods: 600 synthetic cases, 15 failure types, precision/recall, confidence calibration, unsupported-claim analysis, human-in-the-loop review.
• Why it matters: AI tools inside risk workflows have to be auditable and evidence-grounded before an analyst can rely on them.

• What it evaluates: Whether a simulated feed-ranking change improves 7-day retention without moving guardrail metrics the wrong way.
• Methods: 100K users, SRM check, power/MDE, CUPED, segmentation, and a ship / no-ship decision.
• Why it matters: Good product decisions need both a statistical result and judgment about which metrics actually matter.

• What it evaluates: How macro and market stress scenarios move commercial-real-estate risk indicators, using only public data.
• Methods: scenario analysis, stress testing, forecasting, model monitoring, reproducible pipeline.
• Why it matters: A risk model is only useful if its outputs can be tested across changing economic conditions.

• What it evaluates: Macro, trade, and commodity indicators built from FRED data in a reproducible R pipeline.
• Methods: FRED, ARIMA, distributed-lag regression, Quarto, GitHub Actions.
• Why it matters: Connects the macro-policy side of my background to versioned, reproducible tooling.

• Languages: Python, SQL, R
• AI / LLM evaluation: RAG evaluation, LLM evaluation, agent workflows, prompt experimentation, human-in-the-loop review, retrieval-quality analysis
• Analytics & experimentation: A/B testing, CUPED, SRM checks, power/MDE, metric design, forecasting
• Risk & model validation: stress testing, sensitivity analysis, model monitoring, credit-risk analytics
• Engineering: Git, GitHub Actions, Streamlit, Quarto, pytest, SQLAlchemy
• Visualization: Tableau, Power BI, Streamlit dashboards, Quarto

📫 yullieyang@gmail.com · Portfolio · LinkedIn

📍 Based in Boston · relocating to the San Francisco Bay Area

Currently looking at roles in AI evaluation, model risk, Trust & Safety analytics, Responsible AI, experimentation, and model validation.