fix(route): update Uber Engineering blog by zhsama · Pull Request #23174 · DIYgod/RSSHub

zhsama · 2026-09-01T11:25:36Z

Involved Issue / 该 PR 相关 Issue

Close #

Example for the Proposed Route(s) / 路由地址示例

/uber/blog
/uber/blog/uber-ai
/uber/blog/backend
/uber/blog/culture
/uber/blog/data
/uber/blog/mobile
/uber/blog/security
/uber/blog/web

New RSS Route Checklist / 新 RSS 路由检查表

New Route / 新的路由
- Follows Script Standard / 跟随路由规范
Anti-bot or rate limit / 反爬/频率限制
- If yes, do your code reflect this sign? / 如果有, 是否有对应的措施?
Date and time / 日期和时间
- Parsed / 可以解析
- Correct time zone / 时区正确
New package added / 添加了新的包
Puppeteer

Note / 说明

Uber's previously used Engineering RSS endpoint now redirects to a 404 page. This change switches the existing route to the current SSR Engineering overview and official category pages.

Supports all current Engineering categories through /uber/blog/:category?.
Preserves deprecated numeric maxPage paths as overview-feed aliases.
Uses semantic/test selectors rather than generated presentation classes.
Hydrates and caches each article detail for full content, publication date, authors, categories, and image.
Uses first-page results only and applies the common limit before detail requests.
Updates route metadata used by generated docs and Radar; no separate documentation file is required.

Validation:

pnpm typecheck
pnpm format:check
pnpm build:docs
Isolated Linux/Docker pnpm test: 403/403 tests passed
Production build and deployment with Redis: overview and all seven categories returned HTTP 200 with full content and metadata; repeated requests returned rsshub-cache-status: HIT
After rebasing onto the latest upstream/master, overview and backend live-route smoke tests passed

chatgpt-codex-connector · 2026-09-01T11:25:48Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-01T11:28:17.338384Z	`02b265f`	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

github-actions · 2026-09-01T11:31:12Z

Successfully generated as following:

http://localhost:1200/uber/blog - Success ✔️

<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
  <channel>
    <title>Uber Engineering Blog</title>
    <link>https://www.uber.com/us/en/blog/engineering/</link>
    <atom:link href="http://localhost:1200/uber/blog" rel="self" type="application/rss+xml"></atom:link>
    <description>The technology behind Uber Engineering. - Powered by RSSHub</description>
    <generator>RSSHub</generator>
    <webMaster>contact@rsshub.app (RSSHub)</webMaster>
    <language>en-us</language>
    <image>
      <url>https://tb-static.uber.com/prod/udam-assets/6b65f287-0bee-44e7-868a-e26b8722364e.png</url>
      <title>Uber Engineering Blog</title>
      <link>https://www.uber.com/us/en/blog/engineering/</link>
    </image>
    <lastBuildDate>Tue, 01 Sep 2026 11:31:00 GMT</lastBuildDate>
    <ttl>5</ttl>
    <item>
      <title>Running a Software Factory Efficiently at Uber Scale</title>
      <description>&lt;h1 class=&quot;rich-heading-h1&quot; id=&quot;introduction-1201&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Introduction&lt;/strong&gt;&lt;/h1&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;AI tools are now embedded in every phase of software development at Uber. More than 70% of pull requests are attributed to local or cloud agents. Engineers have built over 3,600 agent skills across the software development life cycle, and executed more than 30K agent skill executions per day.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;At the AI Engineer 2026 conference, we shared &lt;/span&gt;&lt;a href=&quot;https://youtu.be/17-YSUHo6Lk?si=EbqFAc2UwHX_3wSc&quot; class=&quot;rich-link&quot; dir=&quot;auto&quot;&gt;&lt;span class=&quot;rich-underline&quot; data-lexical-text=&quot;true&quot;&gt;our vision&lt;/span&gt;&lt;/a&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; for the Software Factory and the building blocks and managed agents we are building across the lifecycle. As we progress on that vision, a growing share of sessions aren’t initiated by humans, but by automated managed agents handling code review, self-healing CI failures, completing E2E PRs with visual validation, triaging on-call alerts, debugging incoming bugs, and handling a variety of code maintenance tasks with human reviews/escalations.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;As shown in Figure 1, from February to Aug 2026, weekly active users across all agentic offerings across all our employees (engineers &amp;amp; non-engineers) grew 7x, and weekly agentic requests grew 9.4x. Meanwhile, our total AI spend has relatively stabilized since April due to optimizations across the board.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Weekly active users and agent requests rose sharply from February to August 2026; costs fluctuated but increased overall.&quot; class=&quot;rich-image&quot; height=&quot;855&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9kZGExMDZkMy0wNDM5LTQyN2ItYjc4MC1jMzk5ZGM4NWM3YzgucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 1: Weekly active users, agent requests, and cost from Feb–Mid Aug 2026, with users deduplicated across tools. &lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Since adoption, workload mix, and model upgrades are all continuously changing, isolating our own optimization gains means holding one model fixed, since behavior shifts with every upgrade and model family. We did that from February to July: cost per 1,000 model requests is down almost 34% from its peak, and cost per session is down 52% from its June peak.&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Unit costs for 2026: cost per 1,000 requests drops 34%, cost per session drops 52% from peak to August.&quot; class=&quot;rich-image&quot; height=&quot;855&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9iMjVkMTc1Zi04MjFiLTRiYTYtYTE0OS0xNGU1Yzk1YTI1Y2IucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 2: Cost optimization impact with model held constant. *Cost/session data starts from end of May.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;This blog walks through how we think about our software factory: the four layers agent sessions run in, the cost equation we use to decompose spend, how we measure each term, and how we optimize those terms across every layer.&amp;nbsp;&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;All pricing and vendor metrics in this comparison are based on publicly available information, with cost efficiency gains driven by routing our internal Uber workloads more intelligently within standard tier-pricing. While specific cost reductions we measure are unique to our environment and your mileage may vary depending on your codebase, team size, and agent workflows, the methodology of benchmarking real work and optimizing for accuracy and cost is universally applicable.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;the-software-factory-and--1955&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;The Software Factory and Its Cost Equation&lt;/strong&gt;&lt;/h3&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;four-layers-of-agent-usag-1954&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Four Layers of Agent Usage&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We organize AI usage into four layers, from the most specialized to the most general. As shown in Figure 3, the higher the layer, the more control we have over cost, quality, and model selection.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;the-cost-equation-1991&quot; dir=&quot;auto&quot;&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Dashboard categorizing agent types for code generation, validation, deployment, observation, and maintenance tasks.&quot; class=&quot;rich-image&quot; height=&quot;1596&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/crop/smartcrop/udam/format=auto/width=2400/height=1596/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy82ZjZhMmY5ZC04ZTcwLTQ1YTgtOTYzNy05ZmZhMTE4YWEyM2IucG5n&quot; width=&quot;2400&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 3: The four layers where agent sessions run. &lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;The Cost Equation&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Across any of the layers above, we can decompose the cost of an agentic session into the following terms, which we could measure and optimize independently.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Formula for total spend: users × sessions/user × turns/session × requests/turn × tokens/request × price/token.&quot; class=&quot;rich-image&quot; height=&quot;654&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/crop/smartcrop/udam/format=auto/width=2280/height=654/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9jYWY5MWU3ZC0wODVkLTQwMTQtOTE4Zi1hZmVmN2Y3YjI5OWIucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 4: Total spend, decomposed into six terms that multiply.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The first two terms represent adoption &amp;amp; engagement, which we want to keep growing across our overall user base, whether users use it interactively or agents handle tasks on their behalf. The three middle terms provide opportunities for optimization: the work the agent does on its own behalf, on top of the request an engineer actually made. That is where most of our effort goes. This includes mechanisms that help agents plan faster, reduce unwanted turns or errors, optimize input tokens, and more.&amp;nbsp;&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;how-we-measure-1952&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;How We Measure&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Below is the full set of metrics we track weekly and monthly that enable us to forecast &amp;amp; plan our efforts short-term and long-term.&lt;/span&gt;&lt;/p&gt;&lt;table class=&quot;rich-table&quot;&gt;&lt;tbody&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Layer&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Metrics&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;What it answers&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Portfolio&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Total attributed cost&amp;nbsp;&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Distinct attributed users&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Per-tool/agent cost, users, and spend share&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Where the money is going, and which tool moved&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Unit economics, per tool&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost per user&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Requests per user&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost per 1,000 requests&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;4&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Input, output, and total tokens per request&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;5&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost per 1 million tokens&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;6&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost per 1,000 sessions&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;7&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost per active session hour&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;8&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Prompt Cache hit rate&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Whether the tool is getting cheaper or usage is just shifting&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Model economics&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;For every model&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost and cost share&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Requests and request share&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost per 1,000 requests&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;4&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost per 1 million tokens&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Which model releases actually changed the bill, at identical or different per-token prices&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Driver decomposition&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost change decomposed sequentially into&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;adoption (users)&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;engagement (requests per user)&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;input workload (input tokens per request)&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;4&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;output workload (output tokens per request)&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Why the number moved, stated exactly, with nothing left in an unexplained residual&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Managed agent outcomes&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;For each managed agent:&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Outcome-denominated cost (cost per merged PR, cost per review, cost per alert, cost per cleanup);&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Quality signal (revert rate, F1, MTTR);&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Volume (diffs landed, reviews posted, alerts triaged)&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Whether each managed agent is getting cheaper per unit of value delivered, and whether quality holds through model migrations&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;optimization-levers-1951&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Optimization Levers&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;In the following sections, we detail the key levers we used to optimize each part of the cost equation. Some of these levers affect one or more rows in the cost equation.&lt;/span&gt;&lt;/p&gt;&lt;table class=&quot;rich-table&quot;&gt;&lt;tbody&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Price / Token&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Benchmark-driven, Pareto-optimal model selection&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Model defaults&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Tokens / Request&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;400K context cap &amp;amp; Medium reasoning effort by default&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Prompt Caching&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Tool search and CLI-resolved MCP calls&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Code-mode batching of tool calls&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Gateway-routed SaaS MCPs&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Requests / Turn&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Graph-grounded context&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Continuous Skill Optimization&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Visibility &amp;amp; Education&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Live cost counter in the status line&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Visibility and spend tiers&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Session analysis dashboard&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;optimizing-price-token-1975&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Optimizing Price / Token&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The vendor sets the token price. We pick which model runs which workload. Across all our managed agents’ layers, we pick the model that’s most Pareto efficient for that workload. For us, Pareto efficient means cost/completed task, output quality, and model reliability.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;benchmark-driven-model-se-1949&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Benchmark-Driven Model Selection&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Model selection happens in four steps, the same for every managed agent we run.&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Build a benchmark out of the agent’s real work.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Run the agent on a harness that serves any model, frontier or open-weight, behind one interface.&amp;nbsp;&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Move to whatever is Pareto optimal, and keep moving. The frontier shifts every few weeks.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Looking ahead, we continually refine our workload performance by leveraging aggregated insights from our managed agents to test and deploy various model routing strategies.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;For example, we use uReview, which handles AI code review for all pull requests. We built its benchmark from real pull requests with known bugs and graded them easy, medium, and hard. We score precision, recall, and F1 against those bugs, plus cost per review, latency, timeouts, and noise. As shown in Figure 5, switching models improved our F1 while dramatically reducing cost/PR. In the figure, the dashed line is the Pareto frontier. Everything below and left of it is beaten by something cheaper or better.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Scatter plot comparing cost per review and F1 score for 10 model configurations, highlighting Pareto frontier.&quot; class=&quot;rich-image&quot; height=&quot;1125&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9hM2I2OGQ3OS00ZDFmLTQ3ZDUtOTUyNC01YzBjNDA0YmRiZDIucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 5: Every configuration we tested for uReview. &lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Using thousands of real-world PRs across our large monorepos, we internally also have an Uber SWE Benchmark that runs frontier and open-weight models across different task types. We use it to inform model selection across all our SDLC-managed agents.&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;default-model-selection-1948&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Default Model Selection&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;In the interactive interface, token unit costs remain fixed; however, you can strategically manage token distribution across models. Two default settings primarily govern this distribution: the initial session model and the subagent model.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The subagent default setting has proven to be the most impactful lever, and its significance continues to grow. The proportion of sessions initiating subagents has steadily increased as the latest model capabilities enable more effective multi-agent orchestration. Because subagents perform well-defined tasks with specified inputs that often do not require frontier-level reasoning, we default them to a weaker, more cost-effective model while still allowing manual overrides. The primary model handles task decomposition and evaluation while subagents execute the work.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;optimizing-tokens-request-1947&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Optimizing Tokens / Request&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Every turn re-sends the full conversation history, project context, and tool results. Anything that reduces the per-request payload compounds across the session.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;defaults-1946&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Defaults&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;All interactive harnesses use a unified wrapper for installation management, configuration, authentication, and cost visibility. Two standardized default configurations directly reduce token consumption per request:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Automatic compaction is triggered at 400k tokens even for 1M context window models:&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; This threshold balances model performance against cache bursts and repeated input token costs. Our measurements show a meaningful reduction in fleet-wide input tokens per request.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Reasoning effort defaulted to Medium&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: Output tokens, including internal reasoning tokens, are billed at multiples of the rate of input tokens on primary models; this policy adjustment directly reduces spend in the highest-cost token category. For a large category of tasks, Medium reasoning hits a good balance between cost vs quality.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;prompt-caching-strategy-1945&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Prompt Caching Strategy&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Our prompt caching strategy is driven by the economics of provider prompt cache reads and writes. Since each turn re-transmits the full conversation history, caching the preceding context avoids paying the full cost repeatedly, reducing subsequent reads to just 0.1x the standard input token rate. However, write premiums vary: 5-minute cache entries cost 1.25x, while 1-hour entries cost 2x. Choosing an optimal TTL (Time-to-Live) therefore depends on the duration of gaps between turns. Available TTL options include 5 minutes and 1 hour from Anthropic®, alongside 30 minutes from OpenAI®.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Comparison of 5-minute and 1-hour TTL caching costs for main thread and subagent scenarios, showing cost savings.&quot; class=&quot;rich-image&quot; height=&quot;1200&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9mNjEzNDE2MC02YmViLTQyNzItOGQ4OC0yYmVlNzkyZDA0MDMucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 6: A comparison of 5 turns under both TTL durations. &lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Because engineers often leave interactive sessions idle for more than 5 minutes, we transitioned from the default 5-minute TTL to a 1-hour window. These frequent idle gaps previously invalidated the prefix cache, forcing costly full-price context rebuilds. Sub-agents, by contrast, retain a 5-minute cache TTL because their execution focus is limited to single, short-lived tasks.&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;executing-mcp-tools-via-t-1944&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Executing MCP Tools via the Shell&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;At Uber, all MCP (Model Context Protocol) interactions are routed through a unified gateway. This single entry point encompasses more than 1,000 MCP servers across internal and third-party SaaS MCP, enabling centralized authentication and policy enforcement.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;However, standard MCP loads all tool schemas directly into every session, regardless of whether an engineer will ever invoke them in that session or not. For example, with over 100 tools installed, this pre-loading added approximately 50K-70K tokens of schema overhead to the initial prompt, which was subsequently re-sent on every context turn.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Comparison of token usage: installing each server uses ~50-70,000 tokens, tool search plus CLI uses near zero.&quot; class=&quot;rich-image&quot; height=&quot;930&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy83MWJkMGZiYi0yNDk0LTRmOTItYjE5NC02MTBlM2E5NDA3YmYucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 7: What an agent is already carrying at the start of a session, under 3 ways of reaching the same tools.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;To address this context bloat, we introduced two complementary optimization mechanisms:&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;CLI tool resolution&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: Replaces direct MCP integration by allowing the model to execute a shell command. The CLI resolves and invokes the required tool against the gateway dynamically at call time, eliminating Uber MCP schemas from the session context. All 1K+ MCP tools from our internal MCP gateway are projected as CLI commands.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Tool search&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: Scales to thousands of tools by allowing the model to search the tool catalog and load only required tools on demand. This approach mitigates context bloat, typically reducing token usage for tool definitions, and maintains high selection accuracy even as the available tool library expands, preventing degradation associated with large tool sets.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;code-mode-1943&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Code-Mode&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;When tools call functions directly as shell commands, models can batch multiple actions within a single script. This batching is particularly advantageous for chatty tool protocols. Under standard MCP workflows, each action requires a separate model turn to emit a request, load the raw response into the context window, and process the results sequentially. For instance, executing a single SQL query requires submitting the request, polling status 2 to 5 times, and retrieving the output. Code-mode streamlines this entire flow into an automated Python loop, keeping intermediate polling out of the model’s active context. As shown in Figure 8 on the left, the model participates in the polling loop, and every response lands in its context. On the right, the loop runs in a subprocess, and only the summary comes back.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Comparison of MCP tool use and code-mode workflows, highlighting model turns, token usage, and data flow steps.&quot; class=&quot;rich-image&quot; height=&quot;1140&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy8zMThiN2YyYy0zOGQ2LTQxNWYtYWViMy0xODU5Zjk2NWM1NzMucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 8: The same warehouse query, both ways. &lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We measured this by running 5 identical SQL queries through both paths in the same session:&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;table class=&quot;rich-table&quot;&gt;&lt;tbody&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Query&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;LLM tool-use&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Code-mode&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Savings&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;SELECT 1 (1 row)&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;903&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;402&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;55%&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;COUNT(*) (1 row)&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;954&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;403&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;58%&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;GROUP BY LIMIT 20 (20 rows)&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;1,600&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;457&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;71%&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;SHOW COLUMNS (175 rows)&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;2,200&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;900&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;59%&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;SELECT * wide table (50 rows)&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;1,431,594&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;900&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;~100%&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Tokens per query, measured in the same Claude Code session.&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;br&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;br&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;br&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The initial three rows highlight the main finding: even for minimal result sets far below response-size limits, code-mode reduces token usage by more than 50%. Rather than bypassing large data payloads, these efficiencies stem from eliminating unnecessary overhead, including schema initialization, multi-turn polling, and redundant step-by-step reasoning.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Bulk workflows compound the effect, because the loop that would have been N model turns becomes one script and the savings compound to more than 90%. By deploying more than 25 pre-built code-mode skills for our most-accessed MCP servers, we ensure standard workflows default to the most cost-effective path.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;saas-mcps-1942&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;SaaS MCPs&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Managing third-party software proved significantly more challenging than our internal servers. Vendors design MCP servers to expose full product capabilities because they can&#39;t anticipate specific customer usage. For instance, a workspace suite bundles 49 tools into a single server, requiring ~22K tokens of schema, while messaging and project tracking vendors ship 34 and 46 tools, respectively. Loading two or three vendor servers makes the agent carry more schema overhead than the file being edited before a user even enters a prompt.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;To address this, we route SaaS MCP servers through our MCP gateway using the same mechanism we do for our internal MCPs. We also expose all these MCPs as CLIs that any agentic surface can invoke. Additionally, we author dedicated skills within our code-mode plugin for each server to encapsulate common workflows. This unlocked efficient agentic workflows across many SaaS vendors.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;optimizing-requests-turn-1983&quot; dir=&quot;auto&quot;&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Single shell command accesses multiple SaaS tools via MCP Gateway, including workspace, project, messaging, and design.&quot; class=&quot;rich-image&quot; height=&quot;825&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy84NjlhNTcxMi0zNmZiLTQwMmItYWQ5ZS03MWFlNmE2MTA1Y2YucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 9: Every SaaS MCP server is exposed behind our MCP gateway to ensure a unified, efficient access pattern. &lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Optimizing Requests / Turn&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;An ungrounded agent fails slowly rather than cheaply, repeatedly sending an expanding context window to search one more location. Providing richer information upfront remains the single most powerful lever to reduce this search overhead.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;context-engineering-1940&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Context Engineering&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Across Uber’s vast codebase and data ecosystem, comprising hundreds of millions of code lines and thousands of tables, agents spend most of their turns locating information rather than generating code. To address this, we engineered the AI Context Graph: a unified network containing 24 million nodes and 80 million edges across 86 nodes and 117 edge types. It integrates data from over 30 internal systems, including services, engineering teams, incident logs, pull requests, architectural design docs, deployments, datasets, and historical table usage queries, and lets any agent query it in natural language.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Query with the graph took 38s and was correct; without the graph took 20m 09s and was wrong.&quot; class=&quot;rich-image&quot; height=&quot;870&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy8xYzZjNzM3MS1hNGU0LTQ3ZGQtOTQ5Ni0zMTViZWEyYmYxMDIucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 10: Comparing execution paths for the identical prompt submitted to the same model, with and without graph grounding.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The grounded agent queried historical usage, identified the specific table used by over 50 analysts, and delivered the answer in 38 seconds. Conversely, the ungrounded agent lacked visibility into that table; it spent 20 minutes inspecting service code, spawning 2 subagents, and hitting 3 errors before incorrectly concluding the dataset was unqueryable.&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;visibility-education-1939&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Visibility &amp;amp; Education&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The levers here are visibility and feedback loops that help engineers and agents converge faster.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;the-status-line-1938&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;The Status Line&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We put a live cost counter in the harness status line that tracks live spend per harness and across all harnesses for each user.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;visibility-and-spend-tier-1986&quot; dir=&quot;auto&quot;&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Terminal-style status bar with colorful metrics, token usage, costs, and a URL, labeled &#39;What an engineer sees, all the time&#39;.&quot; class=&quot;rich-image&quot; height=&quot;312&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9lMDg3OGU0ZC01NWI0LTQ2MjAtYmI5Zi1jZmM5OTc3MTg1NmMucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 11: The status line, with the session analyzer and efficiency guide that ship alongside it.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Visibility and Spend Tiers&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;To avoid imposing strict caps, we implemented real-time spend tracking and automated nudges:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Statusline live counter.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; Running session cost is always visible in the terminal.&amp;nbsp;&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Harness pool.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; One shared tier across all interactive harnesses, not per-tool budgets. And separate tiers for managed agents.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Slack nudges.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; Alerts at 50/80/100% of expected spend so engineers have time to plan.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;4&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Easy approval flows.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; Manager sign-off for tier upgrades with quick propagation.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;5&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Cost check skill and tips.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; A dashboard skill for on-demand cost breakdown and live status line coaching.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;These enable engineers to evaluate task ROI independently while mitigating runaway expenses.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;session-analysis-dashboar-1936&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Session Analysis Dashboard&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;While the status line highlights a session&#39;s total expenditure, it lacks visibility into cost drivers or actionable efficiency steps. General guidance provides high-level principles, but can’t evaluate individual developer workflows. The session analysis dashboard bridges this gap by inspecting session artifacts directly.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Built directly into the runtime, it requires zero setup or opt-in. Executing the &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;cost dashboard&lt;/em&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt; &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;skill analyzes all session traces for the user across local and remote cloud sandboxes across all harnesses they use. Rather than producing an aggregate metric, it flags 16 distinct anti-patterns across sessions, pairing each with its financial impact and a targeted remediation. Some of the categories include:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Suboptimal model routing: &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Executing simple multi-turn sessions on Opus that Sonnet could easily fulfill.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Context window bloat: &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Large MCP payloads (for example, 40KB responses) persisting in context and incurring repeated billing on subsequent turns.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Cache expiration inefficiencies: &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Resuming sessions after extended breaks where expired prompt caches force full-price prefix rebuilds.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;4&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Prompt initialization overhead: &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Pre-loading 100,000 tokens of system instructions and tool definitions before any user input is provided.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;what-s-next-1989&quot; dir=&quot;auto&quot;&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Dashboard showing $4162.07 total spend, 433 sessions, 95% cache hit rate, $1097.82 miss cost, $1213.87 savings.&quot; class=&quot;rich-image&quot; height=&quot;348&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9kNGNhYWMzYy1hOWNkLTRiYzItYmQyNS0wN2M2NDY1NThjZmYucG5n&quot; width=&quot;1290&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 12: Session-level cost dashboard identifies waste patterns and potential savings.. &lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;What’s Next?&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Current initiatives in progress include:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Growing the fleet of managed agents: &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;For every new agent, we follow a consistent roadmap: establish target outcome metrics, assemble evaluation benchmarks, and identify a Pareto-optimal model. This systematic approach aims to elevate each stage of the SDLC higher up the factory maturity model.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Dynamic Model Routing: &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We’re expanding benchmark coverage across diverse programming languages, code repositories, and agent modalities. Effective model routing relies heavily on comprehensive evaluation, given that model capabilities vary widely.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Deepening context-graph integration: &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We’re unlocking graph query capabilities across a wider selection of autonomous agents.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;4&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Evolving session analytics into real-time developer guidance: &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;By shifting from periodic batch detection of anti-patterns to continuous trace monitoring, we aim to deliver personalized, real-time efficiency recommendations directly to engineers.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;5&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Continuous Skill Improvement&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: We are working on an automated way to record papercuts from agent skill executions and auto-generate skill updates from the collected traces.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;h1 class=&quot;rich-heading-h1&quot; id=&quot;conclusion-1894&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Conclusion&lt;/strong&gt;&lt;/h1&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Managing and curbing rising AI coding expenses is also a tractable engineering challenge. By eliminating wasted, zero-value token consumption rather than relying solely on lower unit prices or downgrading tooling, we scaled usage 7x while simultaneously reducing unit costs across all metrics and improving/maintaining output quality.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The core strategic shift is moving from interactive developer workflows to fully managed agents. Transitioning SDLC workloads into managed environments grants complete control over model routing, execution harnesses, and operational spend. Optimizing a fleet of specialized managed agents, each paired with dedicated evaluation benchmarks and a Pareto-efficient model, is inherently more cost-effective and scalable than optimizing individual terminal sessions across thousands of engineers.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;acknowledgments-1934&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Acknowledgments&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;This is a collective effort by many engineers who are building the most efficient blocks to implement Software Factory at Uber scale, while ensuring we get ROI for every token we spend. We would like to thank our core team involved in various efforts across Software Factory listed here: Abhishek Bhatia, Adam Huda, Aditya Patel, Alok Srivastava, Ameya Ketkar, Anil Purohit, Atakan Kandemir, Ben Chou, Brandon Barker, Danielle Yim, Deepanshu Mehndiratta, Gaurav Gill, Israel Marban, Jason Varbedian, Karen Xu, Lei Shi, Mager Mager, Meghana Somasundara, Peng Liu, Preet Inder, Qiushen Wang, Rush Tehrani, Shesh Patel, Shiven Tripathi, Shubham Gupta, Stas Khalup, Ting Chen, Tse-Shi Wang, Ty Smith, Vikram Hullukunte, Viv Keswani, Weiqiang Wang, Will Bond.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Also would like to thank Johannes Gehrke, Mattie Toia, Sumanth Sukumar, and Praveen Neppalli Naga for their leadership.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cover Photo Attribution: Himer Romana&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;Anthropic® is a registered trademark of Anthropic PBC.&amp;nbsp;&lt;/em&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;Claude Code™ and Claude® are trademarks of Anthropic, PBC.&lt;/em&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot;

...

github-actions · 2026-09-01T11:31:13Z

http://localhost:1200/uber/blog/uber-ai - Success ✔️

<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
  <channel>
    <title>Uber Engineering Blog - AI / ML</title>
    <link>https://www.uber.com/us/en/blog/engineering/uber-ai/</link>
    <atom:link href="http://localhost:1200/uber/blog/uber-ai" rel="self" type="application/rss+xml"></atom:link>
    <description>Uber AI is at the heart of AI-powered innovation and technologies at Uber. AI research and its applications solve challenges across the whole of Uber. - Powered by RSSHub</description>
    <generator>RSSHub</generator>
    <webMaster>contact@rsshub.app (RSSHub)</webMaster>
    <language>en-us</language>
    <image>
      <url>https://tb-static.uber.com/prod/udam-assets/6b65f287-0bee-44e7-868a-e26b8722364e.png</url>
      <title>Uber Engineering Blog - AI / ML</title>
      <link>https://www.uber.com/us/en/blog/engineering/uber-ai/</link>
    </image>
    <lastBuildDate>Tue, 01 Sep 2026 11:31:02 GMT</lastBuildDate>
    <ttl>5</ttl>
    <item>
      <title>Running a Software Factory Efficiently at Uber Scale</title>
      <description>&lt;h1 class=&quot;rich-heading-h1&quot; id=&quot;introduction-1201&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Introduction&lt;/strong&gt;&lt;/h1&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;AI tools are now embedded in every phase of software development at Uber. More than 70% of pull requests are attributed to local or cloud agents. Engineers have built over 3,600 agent skills across the software development life cycle, and executed more than 30K agent skill executions per day.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;At the AI Engineer 2026 conference, we shared &lt;/span&gt;&lt;a href=&quot;https://youtu.be/17-YSUHo6Lk?si=EbqFAc2UwHX_3wSc&quot; class=&quot;rich-link&quot; dir=&quot;auto&quot;&gt;&lt;span class=&quot;rich-underline&quot; data-lexical-text=&quot;true&quot;&gt;our vision&lt;/span&gt;&lt;/a&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; for the Software Factory and the building blocks and managed agents we are building across the lifecycle. As we progress on that vision, a growing share of sessions aren’t initiated by humans, but by automated managed agents handling code review, self-healing CI failures, completing E2E PRs with visual validation, triaging on-call alerts, debugging incoming bugs, and handling a variety of code maintenance tasks with human reviews/escalations.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;As shown in Figure 1, from February to Aug 2026, weekly active users across all agentic offerings across all our employees (engineers &amp;amp; non-engineers) grew 7x, and weekly agentic requests grew 9.4x. Meanwhile, our total AI spend has relatively stabilized since April due to optimizations across the board.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Weekly active users and agent requests rose sharply from February to August 2026; costs fluctuated but increased overall.&quot; class=&quot;rich-image&quot; height=&quot;855&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9kZGExMDZkMy0wNDM5LTQyN2ItYjc4MC1jMzk5ZGM4NWM3YzgucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 1: Weekly active users, agent requests, and cost from Feb–Mid Aug 2026, with users deduplicated across tools. &lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Since adoption, workload mix, and model upgrades are all continuously changing, isolating our own optimization gains means holding one model fixed, since behavior shifts with every upgrade and model family. We did that from February to July: cost per 1,000 model requests is down almost 34% from its peak, and cost per session is down 52% from its June peak.&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Unit costs for 2026: cost per 1,000 requests drops 34%, cost per session drops 52% from peak to August.&quot; class=&quot;rich-image&quot; height=&quot;855&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9iMjVkMTc1Zi04MjFiLTRiYTYtYTE0OS0xNGU1Yzk1YTI1Y2IucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 2: Cost optimization impact with model held constant. *Cost/session data starts from end of May.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;This blog walks through how we think about our software factory: the four layers agent sessions run in, the cost equation we use to decompose spend, how we measure each term, and how we optimize those terms across every layer.&amp;nbsp;&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;All pricing and vendor metrics in this comparison are based on publicly available information, with cost efficiency gains driven by routing our internal Uber workloads more intelligently within standard tier-pricing. While specific cost reductions we measure are unique to our environment and your mileage may vary depending on your codebase, team size, and agent workflows, the methodology of benchmarking real work and optimizing for accuracy and cost is universally applicable.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;the-software-factory-and--1955&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;The Software Factory and Its Cost Equation&lt;/strong&gt;&lt;/h3&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;four-layers-of-agent-usag-1954&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Four Layers of Agent Usage&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We organize AI usage into four layers, from the most specialized to the most general. As shown in Figure 3, the higher the layer, the more control we have over cost, quality, and model selection.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;the-cost-equation-1991&quot; dir=&quot;auto&quot;&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Dashboard categorizing agent types for code generation, validation, deployment, observation, and maintenance tasks.&quot; class=&quot;rich-image&quot; height=&quot;1596&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/crop/smartcrop/udam/format=auto/width=2400/height=1596/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy82ZjZhMmY5ZC04ZTcwLTQ1YTgtOTYzNy05ZmZhMTE4YWEyM2IucG5n&quot; width=&quot;2400&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 3: The four layers where agent sessions run. &lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;The Cost Equation&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Across any of the layers above, we can decompose the cost of an agentic session into the following terms, which we could measure and optimize independently.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Formula for total spend: users × sessions/user × turns/session × requests/turn × tokens/request × price/token.&quot; class=&quot;rich-image&quot; height=&quot;654&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/crop/smartcrop/udam/format=auto/width=2280/height=654/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9jYWY5MWU3ZC0wODVkLTQwMTQtOTE4Zi1hZmVmN2Y3YjI5OWIucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 4: Total spend, decomposed into six terms that multiply.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The first two terms represent adoption &amp;amp; engagement, which we want to keep growing across our overall user base, whether users use it interactively or agents handle tasks on their behalf. The three middle terms provide opportunities for optimization: the work the agent does on its own behalf, on top of the request an engineer actually made. That is where most of our effort goes. This includes mechanisms that help agents plan faster, reduce unwanted turns or errors, optimize input tokens, and more.&amp;nbsp;&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;how-we-measure-1952&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;How We Measure&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Below is the full set of metrics we track weekly and monthly that enable us to forecast &amp;amp; plan our efforts short-term and long-term.&lt;/span&gt;&lt;/p&gt;&lt;table class=&quot;rich-table&quot;&gt;&lt;tbody&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Layer&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Metrics&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;What it answers&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Portfolio&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Total attributed cost&amp;nbsp;&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Distinct attributed users&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Per-tool/agent cost, users, and spend share&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Where the money is going, and which tool moved&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Unit economics, per tool&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost per user&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Requests per user&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost per 1,000 requests&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;4&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Input, output, and total tokens per request&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;5&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost per 1 million tokens&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;6&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost per 1,000 sessions&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;7&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost per active session hour&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;8&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Prompt Cache hit rate&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Whether the tool is getting cheaper or usage is just shifting&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Model economics&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;For every model&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost and cost share&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Requests and request share&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost per 1,000 requests&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;4&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost per 1 million tokens&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Which model releases actually changed the bill, at identical or different per-token prices&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Driver decomposition&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cost change decomposed sequentially into&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;adoption (users)&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;engagement (requests per user)&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;input workload (input tokens per request)&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;4&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;output workload (output tokens per request)&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Why the number moved, stated exactly, with nothing left in an unexplained residual&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Managed agent outcomes&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;For each managed agent:&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Outcome-denominated cost (cost per merged PR, cost per review, cost per alert, cost per cleanup);&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Quality signal (revert rate, F1, MTTR);&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Volume (diffs landed, reviews posted, alerts triaged)&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Whether each managed agent is getting cheaper per unit of value delivered, and whether quality holds through model migrations&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;optimization-levers-1951&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Optimization Levers&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;In the following sections, we detail the key levers we used to optimize each part of the cost equation. Some of these levers affect one or more rows in the cost equation.&lt;/span&gt;&lt;/p&gt;&lt;table class=&quot;rich-table&quot;&gt;&lt;tbody&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Price / Token&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Benchmark-driven, Pareto-optimal model selection&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Model defaults&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Tokens / Request&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;400K context cap &amp;amp; Medium reasoning effort by default&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Prompt Caching&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Tool search and CLI-resolved MCP calls&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Code-mode batching of tool calls&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Gateway-routed SaaS MCPs&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Requests / Turn&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Graph-grounded context&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Continuous Skill Optimization&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Visibility &amp;amp; Education&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Live cost counter in the status line&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Visibility and spend tiers&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Session analysis dashboard&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;optimizing-price-token-1975&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Optimizing Price / Token&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The vendor sets the token price. We pick which model runs which workload. Across all our managed agents’ layers, we pick the model that’s most Pareto efficient for that workload. For us, Pareto efficient means cost/completed task, output quality, and model reliability.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;benchmark-driven-model-se-1949&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Benchmark-Driven Model Selection&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Model selection happens in four steps, the same for every managed agent we run.&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Build a benchmark out of the agent’s real work.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Run the agent on a harness that serves any model, frontier or open-weight, behind one interface.&amp;nbsp;&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Move to whatever is Pareto optimal, and keep moving. The frontier shifts every few weeks.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Looking ahead, we continually refine our workload performance by leveraging aggregated insights from our managed agents to test and deploy various model routing strategies.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;For example, we use uReview, which handles AI code review for all pull requests. We built its benchmark from real pull requests with known bugs and graded them easy, medium, and hard. We score precision, recall, and F1 against those bugs, plus cost per review, latency, timeouts, and noise. As shown in Figure 5, switching models improved our F1 while dramatically reducing cost/PR. In the figure, the dashed line is the Pareto frontier. Everything below and left of it is beaten by something cheaper or better.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Scatter plot comparing cost per review and F1 score for 10 model configurations, highlighting Pareto frontier.&quot; class=&quot;rich-image&quot; height=&quot;1125&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9hM2I2OGQ3OS00ZDFmLTQ3ZDUtOTUyNC01YzBjNDA0YmRiZDIucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 5: Every configuration we tested for uReview. &lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Using thousands of real-world PRs across our large monorepos, we internally also have an Uber SWE Benchmark that runs frontier and open-weight models across different task types. We use it to inform model selection across all our SDLC-managed agents.&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;default-model-selection-1948&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Default Model Selection&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;In the interactive interface, token unit costs remain fixed; however, you can strategically manage token distribution across models. Two default settings primarily govern this distribution: the initial session model and the subagent model.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The subagent default setting has proven to be the most impactful lever, and its significance continues to grow. The proportion of sessions initiating subagents has steadily increased as the latest model capabilities enable more effective multi-agent orchestration. Because subagents perform well-defined tasks with specified inputs that often do not require frontier-level reasoning, we default them to a weaker, more cost-effective model while still allowing manual overrides. The primary model handles task decomposition and evaluation while subagents execute the work.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;optimizing-tokens-request-1947&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Optimizing Tokens / Request&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Every turn re-sends the full conversation history, project context, and tool results. Anything that reduces the per-request payload compounds across the session.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;defaults-1946&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Defaults&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;All interactive harnesses use a unified wrapper for installation management, configuration, authentication, and cost visibility. Two standardized default configurations directly reduce token consumption per request:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Automatic compaction is triggered at 400k tokens even for 1M context window models:&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; This threshold balances model performance against cache bursts and repeated input token costs. Our measurements show a meaningful reduction in fleet-wide input tokens per request.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Reasoning effort defaulted to Medium&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: Output tokens, including internal reasoning tokens, are billed at multiples of the rate of input tokens on primary models; this policy adjustment directly reduces spend in the highest-cost token category. For a large category of tasks, Medium reasoning hits a good balance between cost vs quality.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;prompt-caching-strategy-1945&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Prompt Caching Strategy&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Our prompt caching strategy is driven by the economics of provider prompt cache reads and writes. Since each turn re-transmits the full conversation history, caching the preceding context avoids paying the full cost repeatedly, reducing subsequent reads to just 0.1x the standard input token rate. However, write premiums vary: 5-minute cache entries cost 1.25x, while 1-hour entries cost 2x. Choosing an optimal TTL (Time-to-Live) therefore depends on the duration of gaps between turns. Available TTL options include 5 minutes and 1 hour from Anthropic®, alongside 30 minutes from OpenAI®.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Comparison of 5-minute and 1-hour TTL caching costs for main thread and subagent scenarios, showing cost savings.&quot; class=&quot;rich-image&quot; height=&quot;1200&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9mNjEzNDE2MC02YmViLTQyNzItOGQ4OC0yYmVlNzkyZDA0MDMucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 6: A comparison of 5 turns under both TTL durations. &lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Because engineers often leave interactive sessions idle for more than 5 minutes, we transitioned from the default 5-minute TTL to a 1-hour window. These frequent idle gaps previously invalidated the prefix cache, forcing costly full-price context rebuilds. Sub-agents, by contrast, retain a 5-minute cache TTL because their execution focus is limited to single, short-lived tasks.&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;executing-mcp-tools-via-t-1944&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Executing MCP Tools via the Shell&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;At Uber, all MCP (Model Context Protocol) interactions are routed through a unified gateway. This single entry point encompasses more than 1,000 MCP servers across internal and third-party SaaS MCP, enabling centralized authentication and policy enforcement.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;However, standard MCP loads all tool schemas directly into every session, regardless of whether an engineer will ever invoke them in that session or not. For example, with over 100 tools installed, this pre-loading added approximately 50K-70K tokens of schema overhead to the initial prompt, which was subsequently re-sent on every context turn.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Comparison of token usage: installing each server uses ~50-70,000 tokens, tool search plus CLI uses near zero.&quot; class=&quot;rich-image&quot; height=&quot;930&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy83MWJkMGZiYi0yNDk0LTRmOTItYjE5NC02MTBlM2E5NDA3YmYucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 7: What an agent is already carrying at the start of a session, under 3 ways of reaching the same tools.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;To address this context bloat, we introduced two complementary optimization mechanisms:&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;CLI tool resolution&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: Replaces direct MCP integration by allowing the model to execute a shell command. The CLI resolves and invokes the required tool against the gateway dynamically at call time, eliminating Uber MCP schemas from the session context. All 1K+ MCP tools from our internal MCP gateway are projected as CLI commands.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Tool search&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: Scales to thousands of tools by allowing the model to search the tool catalog and load only required tools on demand. This approach mitigates context bloat, typically reducing token usage for tool definitions, and maintains high selection accuracy even as the available tool library expands, preventing degradation associated with large tool sets.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;code-mode-1943&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Code-Mode&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;When tools call functions directly as shell commands, models can batch multiple actions within a single script. This batching is particularly advantageous for chatty tool protocols. Under standard MCP workflows, each action requires a separate model turn to emit a request, load the raw response into the context window, and process the results sequentially. For instance, executing a single SQL query requires submitting the request, polling status 2 to 5 times, and retrieving the output. Code-mode streamlines this entire flow into an automated Python loop, keeping intermediate polling out of the model’s active context. As shown in Figure 8 on the left, the model participates in the polling loop, and every response lands in its context. On the right, the loop runs in a subprocess, and only the summary comes back.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Comparison of MCP tool use and code-mode workflows, highlighting model turns, token usage, and data flow steps.&quot; class=&quot;rich-image&quot; height=&quot;1140&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy8zMThiN2YyYy0zOGQ2LTQxNWYtYWViMy0xODU5Zjk2NWM1NzMucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 8: The same warehouse query, both ways. &lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We measured this by running 5 identical SQL queries through both paths in the same session:&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;table class=&quot;rich-table&quot;&gt;&lt;tbody&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Query&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;LLM tool-use&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Code-mode&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Savings&lt;/strong&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;SELECT 1 (1 row)&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;903&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;402&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;55%&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;COUNT(*) (1 row)&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;954&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;403&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;58%&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;GROUP BY LIMIT 20 (20 rows)&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;1,600&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;457&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;71%&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;SHOW COLUMNS (175 rows)&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;2,200&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;900&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;59%&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;SELECT * wide table (50 rows)&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;1,431,594&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;900&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;~100%&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr class=&quot;rich-table-row&quot;&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Tokens per query, measured in the same Claude Code session.&lt;/span&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;br&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;br&gt;&lt;/p&gt;&lt;/td&gt;&lt;td class=&quot;rich-table-cell&quot;&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;br&gt;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The initial three rows highlight the main finding: even for minimal result sets far below response-size limits, code-mode reduces token usage by more than 50%. Rather than bypassing large data payloads, these efficiencies stem from eliminating unnecessary overhead, including schema initialization, multi-turn polling, and redundant step-by-step reasoning.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Bulk workflows compound the effect, because the loop that would have been N model turns becomes one script and the savings compound to more than 90%. By deploying more than 25 pre-built code-mode skills for our most-accessed MCP servers, we ensure standard workflows default to the most cost-effective path.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;saas-mcps-1942&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;SaaS MCPs&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Managing third-party software proved significantly more challenging than our internal servers. Vendors design MCP servers to expose full product capabilities because they can&#39;t anticipate specific customer usage. For instance, a workspace suite bundles 49 tools into a single server, requiring ~22K tokens of schema, while messaging and project tracking vendors ship 34 and 46 tools, respectively. Loading two or three vendor servers makes the agent carry more schema overhead than the file being edited before a user even enters a prompt.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;To address this, we route SaaS MCP servers through our MCP gateway using the same mechanism we do for our internal MCPs. We also expose all these MCPs as CLIs that any agentic surface can invoke. Additionally, we author dedicated skills within our code-mode plugin for each server to encapsulate common workflows. This unlocked efficient agentic workflows across many SaaS vendors.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;optimizing-requests-turn-1983&quot; dir=&quot;auto&quot;&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Single shell command accesses multiple SaaS tools via MCP Gateway, including workspace, project, messaging, and design.&quot; class=&quot;rich-image&quot; height=&quot;825&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy84NjlhNTcxMi0zNmZiLTQwMmItYWQ5ZS03MWFlNmE2MTA1Y2YucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 9: Every SaaS MCP server is exposed behind our MCP gateway to ensure a unified, efficient access pattern. &lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Optimizing Requests / Turn&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;An ungrounded agent fails slowly rather than cheaply, repeatedly sending an expanding context window to search one more location. Providing richer information upfront remains the single most powerful lever to reduce this search overhead.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;context-engineering-1940&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Context Engineering&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Across Uber’s vast codebase and data ecosystem, comprising hundreds of millions of code lines and thousands of tables, agents spend most of their turns locating information rather than generating code. To address this, we engineered the AI Context Graph: a unified network containing 24 million nodes and 80 million edges across 86 nodes and 117 edge types. It integrates data from over 30 internal systems, including services, engineering teams, incident logs, pull requests, architectural design docs, deployments, datasets, and historical table usage queries, and lets any agent query it in natural language.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Query with the graph took 38s and was correct; without the graph took 20m 09s and was wrong.&quot; class=&quot;rich-image&quot; height=&quot;870&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy8xYzZjNzM3MS1hNGU0LTQ3ZGQtOTQ5Ni0zMTViZWEyYmYxMDIucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 10: Comparing execution paths for the identical prompt submitted to the same model, with and without graph grounding.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The grounded agent queried historical usage, identified the specific table used by over 50 analysts, and delivered the answer in 38 seconds. Conversely, the ungrounded agent lacked visibility into that table; it spent 20 minutes inspecting service code, spawning 2 subagents, and hitting 3 errors before incorrectly concluding the dataset was unqueryable.&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;visibility-education-1939&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Visibility &amp;amp; Education&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The levers here are visibility and feedback loops that help engineers and agents converge faster.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;the-status-line-1938&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;The Status Line&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We put a live cost counter in the harness status line that tracks live spend per harness and across all harnesses for each user.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;visibility-and-spend-tier-1986&quot; dir=&quot;auto&quot;&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Terminal-style status bar with colorful metrics, token usage, costs, and a URL, labeled &#39;What an engineer sees, all the time&#39;.&quot; class=&quot;rich-image&quot; height=&quot;312&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9lMDg3OGU0ZC01NWI0LTQ2MjAtYmI5Zi1jZmM5OTc3MTg1NmMucG5n&quot; width=&quot;2280&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 11: The status line, with the session analyzer and efficiency guide that ship alongside it.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Visibility and Spend Tiers&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;To avoid imposing strict caps, we implemented real-time spend tracking and automated nudges:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Statusline live counter.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; Running session cost is always visible in the terminal.&amp;nbsp;&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Harness pool.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; One shared tier across all interactive harnesses, not per-tool budgets. And separate tiers for managed agents.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Slack nudges.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; Alerts at 50/80/100% of expected spend so engineers have time to plan.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;4&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Easy approval flows.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; Manager sign-off for tier upgrades with quick propagation.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;5&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Cost check skill and tips.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; A dashboard skill for on-demand cost breakdown and live status line coaching.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;These enable engineers to evaluate task ROI independently while mitigating runaway expenses.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;session-analysis-dashboar-1936&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Session Analysis Dashboard&lt;/strong&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;While the status line highlights a session&#39;s total expenditure, it lacks visibility into cost drivers or actionable efficiency steps. General guidance provides high-level principles, but can’t evaluate individual developer workflows. The session analysis dashboard bridges this gap by inspecting session artifacts directly.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Built directly into the runtime, it requires zero setup or opt-in. Executing the &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;cost dashboard&lt;/em&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt; &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;skill analyzes all session traces for the user across local and remote cloud sandboxes across all harnesses they use. Rather than producing an aggregate metric, it flags 16 distinct anti-patterns across sessions, pairing each with its financial impact and a targeted remediation. Some of the categories include:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Suboptimal model routing: &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Executing simple multi-turn sessions on Opus that Sonnet could easily fulfill.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Context window bloat: &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Large MCP payloads (for example, 40KB responses) persisting in context and incurring repeated billing on subsequent turns.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Cache expiration inefficiencies: &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Resuming sessions after extended breaks where expired prompt caches force full-price prefix rebuilds.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;4&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Prompt initialization overhead: &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Pre-loading 100,000 tokens of system instructions and tool definitions before any user input is provided.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;what-s-next-1989&quot; dir=&quot;auto&quot;&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Dashboard showing $4162.07 total spend, 433 sessions, 95% cache hit rate, $1097.82 miss cost, $1213.87 savings.&quot; class=&quot;rich-image&quot; height=&quot;348&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9kNGNhYWMzYy1hOWNkLTRiYzItYmQyNS0wN2M2NDY1NThjZmYucG5n&quot; width=&quot;1290&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 12: Session-level cost dashboard identifies waste patterns and potential savings.. &lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;What’s Next?&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Current initiatives in progress include:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Growing the fleet of managed agents: &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;For every new agent, we follow a consistent roadmap: establish target outcome metrics, assemble evaluation benchmarks, and identify a Pareto-optimal model. This systematic approach aims to elevate each stage of the SDLC higher up the factory maturity model.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Dynamic Model Routing: &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We’re expanding benchmark coverage across diverse programming languages, code repositories, and agent modalities. Effective model routing relies heavily on comprehensive evaluation, given that model capabilities vary widely.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Deepening context-graph integration: &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We’re unlocking graph query capabilities across a wider selection of autonomous agents.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;4&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Evolving session analytics into real-time developer guidance: &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;By shifting from periodic batch detection of anti-patterns to continuous trace monitoring, we aim to deliver personalized, real-time efficiency recommendations directly to engineers.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;5&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Continuous Skill Improvement&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: We are working on an automated way to record papercuts from agent skill executions and auto-generate skill updates from the collected traces.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;h1 class=&quot;rich-heading-h1&quot; id=&quot;conclusion-1894&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Conclusion&lt;/strong&gt;&lt;/h1&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Managing and curbing rising AI coding expenses is also a tractable engineering challenge. By eliminating wasted, zero-value token consumption rather than relying solely on lower unit prices or downgrading tooling, we scaled usage 7x while simultaneously reducing unit costs across all metrics and improving/maintaining output quality.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The core strategic shift is moving from interactive developer workflows to fully managed agents. Transitioning SDLC workloads into managed environments grants complete control over model routing, execution harnesses, and operational spend. Optimizing a fleet of specialized managed agents, each paired with dedicated evaluation benchmarks and a Pareto-efficient model, is inherently more cost-effective and scalable than optimizing individual terminal sessions across thousands of engineers.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;acknowledgments-1934&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Acknowledgments&lt;/strong&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;This is a collective effort by many engineers who are building the most efficient blocks to implement Software Factory at Uber scale, while ensuring we get ROI for every token we spend. We would like to thank our core team involved in various efforts across Software Factory listed here: Abhishek Bhatia, Adam Huda, Aditya Patel, Alok Srivastava, Ameya Ketkar, Anil Purohit, Atakan Kandemir, Ben Chou, Brandon Barker, Danielle Yim, Deepanshu Mehndiratta, Gaurav Gill, Israel Marban, Jason Varbedian, Karen Xu, Lei Shi, Mager Mager, Meghana Somasundara, Peng Liu, Preet Inder, Qiushen Wang, Rush Tehrani, Shesh Patel, Shiven Tripathi, Shubham Gupta, Stas Khalup, Ting Chen, Tse-Shi Wang, Ty Smith, Vikram Hullukunte, Viv Keswani, Weiqiang Wang, Will Bond.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Also would like to thank Johannes Gehrke, Mattie Toia, Sumanth Sukumar, and Praveen Neppalli Naga for their leadership.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cover Photo Attribution: Himer Romana&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;Anthropic® is a registered trademark of Anthropic PBC.&amp;nbsp;&lt;/em&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;Claude C

...

github-actions · 2026-09-01T11:31:14Z

http://localhost:1200/uber/blog/backend - Success ✔️

<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
  <channel>
    <title>Uber Engineering Blog - Backend</title>
    <link>https://www.uber.com/us/en/blog/engineering/backend/</link>
    <atom:link href="http://localhost:1200/uber/blog/backend" rel="self" type="application/rss+xml"></atom:link>
    <description>Backend Engineering - Powered by RSSHub</description>
    <generator>RSSHub</generator>
    <webMaster>contact@rsshub.app (RSSHub)</webMaster>
    <language>en-us</language>
    <image>
      <url>https://tb-static.uber.com/prod/udam-assets/6b65f287-0bee-44e7-868a-e26b8722364e.png</url>
      <title>Uber Engineering Blog - Backend</title>
      <link>https://www.uber.com/us/en/blog/engineering/backend/</link>
    </image>
    <lastBuildDate>Tue, 01 Sep 2026 11:31:03 GMT</lastBuildDate>
    <ttl>5</ttl>
    <item>
      <title>Zero Sum by Design 10 Years of Uber&#39;s Payments Platform</title>
      <description>&lt;h1 class=&quot;rich-heading-h1&quot; id=&quot;introduction-6&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Introduction&lt;/span&gt;&lt;/h1&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The journey of Uber’s Payments Platform (known internally as Gulfstream) began in 2016 in a single conference room, where a small team bootstrapped a foundational set of microservices. A decade later, many of those services still run in production, reflecting the durability of the system’s early design.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Over the same period, Uber scaled from $50 billion to $217 billion in annualized gross bookings. Supporting this growth required a platform that not only collects payments from customers, but also disburses funds to drivers, couriers, and merchants—processing nearly 2x gross bookings in total money movement and maintaining ledger balances for more than 1.2 billion entities.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;In this post, we share the core principles that shaped this journey and continue to underpin the system today.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;core-principles-270&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Core Principles&lt;/span&gt;&lt;/h3&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;immutable-money-order-271&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Immutable Money Order&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;One of the most critical data models within our architecture is called a Money Order. It is a series of debits and credits between the entities involved in the real-world commerce transaction (such as a trip, an Uber Eats order, a freight shipment, or a digital Uber Cash top-up). To make our system self-auditable, we made our money order immutable.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;One may ask: What happens when a trip is adjusted or a food item is missed in an Uber Eats order? We write additional money orders to capture the adjustment.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;But the principle stands; an order once written can’t be changed in any shape or form.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;zero-sum-principle-272&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Zero Sum Principle&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Before writing a money order down, we always run pre-commit validations. One of the validations is to ensure that all entries in any money order sum to zero. That is, no money can ever be created or destroyed.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;In addition, we also follow the &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;double-entry bookkeeping&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; principle, where every credit entry on the order has an equivalent debit entry, which in turn ensures that the zero-sum principle can never really be violated. Over the years, we have even pushed these principles to systems upstream of the Payments Platform. &lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;strongly-consistent-ledge-273&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Strongly Consistent Ledger Balances&amp;nbsp;&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Back in 2016, Uber was just coming off its monolith architecture, and while we had a homegrown sharded MySQL® offering, there was no perfect solution to guarantee strong consistency for ledger balances. We ended up building this specific data store on Amazon DynamoDB®. Each entity (such as a Spender, Earner, or Business) can have multiple sub-ledger accounts, all stored as a single row in DynamoDB. The typical entity size was 1KB, though over the years, we definitely had to manage the number of sub-ledger accounts within each entity. We had to ensure any sub-ledger account with a 0 balance was pruned to manage the row size.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;core-data-models-274&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Core Data Models&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Our early emphasis on defining core data models proved to be a strategic investment that has yielded significant returns, as these models remain foundational to our operations today.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The 3 data models that matter the most are:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Money order&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: Models money movements between 2 or more real-world entities, encapsulating a commerce transaction at Uber.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Ledger&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: Represents a real-world entity with 1 or more accounts, each account holding a balance.&amp;nbsp;&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Entity changelog&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: Captures balance updates on the ledger, providing a complete audit trail and the ability to recreate an entity’s ledger since inception&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;loosely-coupled-architect-309&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Loosely Coupled Architecture&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We designed the Payments Platform as a collection of independent, stateless microservices, each owning a discrete unit of the money movement lifecycle — money order creation, processing, collection, and disbursement — with Apache Kafka® as the underlying messaging bus. The two shared sources of truth across all services are the money order store and the entity balance store. This async pipeline processes the majority of Uber&#39;s money traffic today. For user-in-session flows, we used Uber’s internal workflow engine - Cadence, to support synchronous payments without compromising the async backbone.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;line-of-business-lob-agno-275&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Line of Business (LOB) Agnostic&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;When we were building the platform, Uber had one core business, Rides, and we were just getting started with Uber Eats. However, we wanted to build the platform so that if Uber ventured into new product lines, it would just work out of the box.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;To this effect, to model the typical real-world money movement, we had generic money orders like:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Collection Money Order&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Disbursement Money Order&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Refund Money Order, and so on&amp;nbsp;&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;These were pure platform concepts and had no association with any specific product line.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We also built a generic commerce Money Order to capture money movements between entities, representing any real-world commerce transaction between 2 or more entities.&amp;nbsp;&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Over the years, we onboarded, Uber Freight, 2-wheels, Transit, Hotels, Rental Cars, Grocery &amp;amp; Retail, Ads,&amp;nbsp; Memberships (Uber one, Costco etc.) and other lines of businesses that has needed very little to no changes within the core platform.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;payment-instrument-agnost-276&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Payment Instrument Agnostic&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;In our early days, we already supported half a dozen payment instruments, including some regional payment instruments, but ‌we knew that over time we’d want to add all of the top payment instruments for any specific country to delight our customers. To this end, we wanted our core platform to operate on a generic concept of a Payment Instrument, backed by a Payment integration interface with APIs such as charge, disburse, refund, and so on.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The core platform itself had no instrument-specific logic, which was abstracted into a payment integration implementation. Over the years, this has allowed us to scale the platform to a broad range of payment methods, including credit and debit cards, PayPal®, Paytm®, UPI®, Alipay®, Apple Pay®, Google Pay®, iDEAL®, PIX®, and dozens of other local/regional wallets, as well as Uber-native instruments like Uber Cash—each plugging into the platform through the same generic Payment integration interface with little to no change to the core platform. &lt;/span&gt;&lt;a href=&quot;https://uberpay.uber.com/#section/Introduction-to-Uber-Pay&quot; class=&quot;rich-link&quot; dir=&quot;auto&quot;&gt;&lt;span class=&quot;rich-underline&quot; data-lexical-text=&quot;true&quot;&gt;Uber Pay&lt;/span&gt;&lt;/a&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; alone, our reverse-integration platform for local and alternative payment methods, today manages 50+ payment methods globally.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;scaling-a-decade-long-jou-277&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Scaling: A Decade-Long Journey&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;While our core principles provided a strong foundation, we’ve continued to evolve the system as the business scaled. Some noteworthy architectural shifts include simplifying how fare components are recorded and utilized for money movements, addressing the high-throughput challenges of hot ledger entities driven by B2B growth, and supporting diverse internal use cases through a Ledger-as-a-Service model.&lt;/span&gt;&lt;/p&gt;&lt;h5 class=&quot;rich-heading-h5&quot; id=&quot;data-layer-293&quot; dir=&quot;auto&quot;&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Distributed payment platform architecture with West and East regions, async replication, and global table replication.&quot; class=&quot;rich-image&quot; height=&quot;2138&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/crop/smartcrop/udam/format=auto/width=3264/height=2138/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9kZTZmN2Q3Ni03YzgxLTRlMDEtODU5Ny04NjYwMjdkYzVmZjcuanBlZw==&quot; width=&quot;3264&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Current Architecture&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Data layer&lt;/span&gt;&lt;/h5&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;As any software solution matures, regulatory and compliance requirements evolve alongside the product. We initially launched with a simpler data architecture backed by Amazon DynamoDB® for orders and changelogs.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;As audit requirements deepened, we moved beyond off-the-shelf databases to build infrastructure tailored to our unique requirements. This led us to partner with Uber&#39;s Storage Platform team to architect and build &lt;/span&gt;&lt;a href=&quot;https://www.uber.com/us/en/blog/dynamodb-to-docstore-migration/&quot; class=&quot;rich-link&quot; dir=&quot;auto&quot;&gt;&lt;span class=&quot;rich-underline&quot; data-lexical-text=&quot;true&quot;&gt;LedgerStore&lt;/span&gt;&lt;/a&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; — a custom storage layer purpose-built for auditability and tamper-evident record-keeping.&lt;/span&gt;&lt;/p&gt;&lt;h5 class=&quot;rich-heading-h5&quot; id=&quot;entity-fares-294&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Entity Fares&lt;/span&gt;&lt;/h5&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;While we could model money movements across any number of parties in a single transaction, our upstream systems didn’t fully have the same flexibility nor enforce the same principles (like zero-sum). Any ambiguity in fare representation directly conflicted with the core principles we had established. To address this, we reimagined our approach by introducing Entity Fares. By explicitly recording fares for every participating entity and enforcing the zero-sum principle at the point of computation, we ensured that our foundational tenets remained intact from the very inception of a transaction.&lt;/span&gt;&lt;/p&gt;&lt;h5 class=&quot;rich-heading-h5&quot; id=&quot;hot-entity-problem-295&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Hot Entity Problem&lt;/span&gt;&lt;/h5&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;As Uber’s B2B ecosystem—including Uber for Business and Uber Direct—witnessed rapid expansion, a unique architectural challenge surfaced: the hot entity problem. Maintaining strong global consistency while frequently updating the same ledger row posed a significant hurdle to serialized write performance and system correctness. To overcome this, we had to &lt;/span&gt;&lt;a href=&quot;https://www.uber.com/us/en/blog/high-throughput-processing/&quot; class=&quot;rich-link&quot; dir=&quot;auto&quot;&gt;&lt;span class=&quot;rich-underline&quot; data-lexical-text=&quot;true&quot;&gt;innovate&lt;/span&gt;&lt;/a&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; with a specialized, serialized batch-write mechanism, ultimately enabling us to scale ledger mutations and achieve a 10x increase in throughput for these high-traffic entities.&lt;/span&gt;&lt;/p&gt;&lt;h5 class=&quot;rich-heading-h5&quot; id=&quot;ledger-as-a-service-296&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Ledger-as-a-Service&lt;/span&gt;&lt;/h5&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Beyond its core architecture, our Ledger primitives proved versatile enough to support a variety of critical stored value and money movement use cases across Uber, including:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Uber Cash and Uber Money&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: Managing both closed-loop and semi-open-loop Stored Value systems required a battle-tested ledger to ensure absolute financial integrity.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Uber for Business&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: To support our B2B invoice-based settlement flows, we leveraged the Ledger to track complex unbilled, billed, and settled balances throughout the entire enterprise billing lifecycle.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;By decoupling and generalizing our ledger components, we evolved our infrastructure into a multi-tenant solution. This Ledger-as-a-Service model empowered engineering teams across the company to iterate on specific business requirements without the overhead of building their own financial primitives. Today, several of these multi-tenant ledgers have been operating reliably for over 7 years.&lt;/span&gt;&lt;/p&gt;&lt;h1 class=&quot;rich-heading-h1&quot; id=&quot;conclusion-207&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Conclusion&lt;/span&gt;&lt;/h1&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Building a Payments Platform that moves hundreds of billions of dollars annually doesn’t happen by accident—it’s the result of principled decisions made early and defended consistently over a decade. Immutability kept our audit trail clean. Zero-sum accounting ensured money was never created or destroyed. Strong consistency gave us a ledger we could trust. And by designing around generic abstractions—LOB-agnostic money orders and instrument-agnostic payment integrations—we built a foundation that absorbed every new product line and payment instrument Uber threw at it, all while avoiding the need for any re-architecture and yet evolving to meet Uber’s needs.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;acknowledgments-278&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Acknowledgments&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;None of this would have been possible without the world-class team that built the system back in 2016 and the equally world-class team that has scaled and operationalized it over the years.&amp;nbsp;&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cover Photo Attribution: Generated with ChatGPT&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;Alipay is a registered trademark of Advanced New Technologies Co., Ltd.&lt;/em&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;Apache Kafka® is a registered trademark of the Apache Software Foundation&lt;/em&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;Apple Pay is a registered trademark of Apple Inc.&amp;nbsp;&lt;/em&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;DynamoDB is a registered trademark of Amazon Technologies, Inc.&amp;nbsp;&lt;/em&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;Google Pay is a registered trademark of Google LLC.&amp;nbsp;&lt;/em&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;iDEAL is a registered trademark of Currence iDEAL B.V.&amp;nbsp;&lt;/em&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;MySQL is a registered trademark of Oracle® and/or its affiliates.&lt;/em&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;PayPal is a registered trademark of PayPal, Inc.&lt;/em&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;Paytm is a registered trademark of One 97 Communications Limited.&lt;/em&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;PIX is a registered trademark of the Banco Central do Brasil (Central Bank of Brazil).&amp;nbsp;&lt;/em&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;UPI is a registered trademark of National Payments Corporation of India (NPCI).&lt;/em&gt;&lt;/p&gt;</description>
      <link>https://www.uber.com/us/en/blog/ubers-payments-platform/</link>
      <guid isPermaLink="false">https://www.uber.com/us/en/blog/ubers-payments-platform/</guid>
      <pubDate>Wed, 05 Aug 2026 16:00:00 GMT</pubDate>
      <author>Nimish Sheth, Manas Kelshikar, Dhirendra Kumar Singh, Rajan Jana, Wasim Raza</author>
      <enclosure url="https://tb-static.uber.com/prod/udam-assets/ceab4d4d-db89-47db-ba46-9717856c82a3.png" type="image/jpeg"></enclosure>
      <category>Engineering</category>
      <category>Backend</category>
    </item>
    <item>
      <title>GitFarm: Git® as a Service for Large-Scale Monorepos</title>
      <description>&lt;h1 class=&quot;rich-heading-h1&quot; id=&quot;introduction-2&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Introduction&lt;/span&gt;&lt;/h1&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;At Uber’s monorepo scale, traditional Git® workflows have become a fundamental bottleneck. Cloning multi-gigabyte repositories, maintaining local checkouts, periodically syncing from upstream, and executing repetitive fetch or push operations consume substantial compute and I/O across hundreds of automation systems. While CI systems such as Jenkins® and Buildkite® provide caching mechanisms to reduce clone times, in practice, these approaches incur significant infrastructure overhead, manual maintenance, and cold-start latencies of several minutes for large monorepos. Moreover, thousands of independent clone and fetch operations impose heavy loads on upstream Git servers, making them a shared scalability bottleneck.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;To address this, we built GitFarm—a platform that provides &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;Git as a Service&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; through a high-performance gRPC® API. GitFarm isn’t a Git SCM—it doesn’t store, replicate, or serve repositories. Instead, it acts as a centralized Git client in the cloud: it executes standard Git commands on behalf of other services, within secure, ephemeral sandboxes backed by pre-warmed repository and container pools. The system enforces identity-scoped authorization, supports multi-command workflows, and leverages specialized back-end clusters for workload isolation.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;For clients, this means no local clones, access to a full Git checkout in under 500 milliseconds, and significantly lower compute and I/O overhead. Services no longer suffer cold starts of 10-15 minutes due to initial clones on each host. In production, GitFarm has reduced client-side resource utilization by over 80% while preserving the flexibility of native Git semantics.&lt;/span&gt;&lt;/p&gt;&lt;h2 class=&quot;rich-heading-h2&quot; id=&quot;background-18&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Background&lt;/span&gt;&lt;/h2&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;At Uber, various automation systems and developer services invoke Git millions of times per day across multiple monorepos spanning Go, Java, Python, Web, Android, and iOS. Historically, each system maintained its own full repository checkout—even for lightweight operations like reading files or validating merges. This duplication consumed large amounts of compute and storage, and generated sustained loads on upstream Git servers as every system performed its own clone and fetch operations.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;the-cost-of-local-checkou-24&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The Cost of Local Checkouts&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cloning Uber’s Go monorepo takes roughly 15 minutes and consumes&amp;nbsp; around 6 CPU cores, 32 GB of memory, and over 40 GB of disk. A service that clones all major monorepos can easily require 16 CPU cores, 64 GB of memory, and 96 GB of storage—just to maintain repository state.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Techniques like shallow clones, partial clones, or single-branch clones reduce initial data transfer, but don’t address the fundamental problem: each request still forces the upstream Git server to enumerate objects, generate packfiles, and stream data. As the client count grows, this turns the Git server into a shared bottleneck. These optimizations also have functional limitations—operations like &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;merge-base&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; or &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;bisect&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; may fail on shallow clones.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;why-not-just-use-ci-38&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Why Not Just Use CI?&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;A natural alternative is to offload Git operations to CI platforms like Jenkins or Buildkite. Both provide standardized execution environments and handle repository checkouts as part of job execution. But this model has significant drawbacks at Uber’s scale:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Heavyweight for simple operations.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; Need to compute a merge-base and push a ref? With CI, you’re provisioning a worker, initializing a workspace, and syncing a repository—all for a few Git commands.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;No standalone Git API.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; All interactions must occur within the job. You can’t just call an API to run git log,&amp;nbsp; you need a full build job.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Redundant work.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; Even when multiple jobs target the same commit, each agent fetches independently.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;4&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Stale state.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; Long-lived agents accumulate outdated refs and packfiles, degrading performance over time.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;5&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Coupled to build a lifecycle.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; Repository operations can’t be triggered outside of a running job — no pre-warming, no async maintenance, no programmatic access.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;These limitations motivated GitFarm: a shared, service-based alternative purpose-built for executing Git operations efficiently at scale.&lt;/span&gt;&lt;/p&gt;&lt;h2 class=&quot;rich-heading-h2&quot; id=&quot;architecture-76&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Architecture&lt;/span&gt;&lt;/h2&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;When a request is submitted to GitFarm, it’s first received by the GitFarm Gateway, which forwards the request to the GitFarm Backend. Within the Backend, an available node is chosen to process the request. The request then executes in a sandbox on that node, where a pre-warmed repository checkout is mounted, as shown in Figure 1.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; style=&quot;text-align: center;&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot; contenteditable=&quot;false&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;GitFarm architecture flowchart showing gateway, backend, sandbox, and upstream Git server interactions via APIs and commands.&quot; class=&quot;rich-image&quot; height=&quot;365&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy84OWY2Mjg0ZS0zMTNmLTQ5NmUtOTUyMy00NWQxY2U1NWY0NmUucG5n&quot; width=&quot;1348&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 1: GitFarm architecture.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;br&gt;&lt;p&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;gateway-entrypoint-86&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Gateway–Entrypoint&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The Gateway is responsible for authenticating and authorizing incoming requests, and routing them to the appropriate back end. Upon receiving a request, the Gateway identifies the client and verifies that the client has permission to access the requested repository. Requests from clients lacking the required privileges are denied.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The Gateway also functions as a load balancer for the GitFarm back ends. It continuously tracks back-end node availability by monitoring periodic heartbeats and status updates that report the number of available sandboxes on each node for each repository. This state is maintained in a Redis® data store. For each incoming request, the Gateway selects the back-end node with the highest number of available repository checkouts for the requested repository, and marks the selected repository checkout as occupied. It then releases the sandbox upon completion by updating the Redis state. If no back-end nodes are available for a given repository, the Gateway rejects incoming requests, effectively throttling the workload due to insufficient resources.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;backend-request-processor-98&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Backend–Request Processor&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The GitFarm Backend forms the core execution layer of the system. It maintains a warm, up-to-date repository state by periodically synchronizing on-disk repositories with their upstream remotes (via git fetch). The Backend manages a pool of isolated execution environments (sandboxes) used to execute Git commands issued through API requests.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;For each incoming request, the Backend provisions the appropriate repository state, identity, and execution context, and mounts them into a sandbox to perform the requested operation, ensuring isolation and consistency across executions, as shown in Figure 2.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; style=&quot;text-align: center;&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot; contenteditable=&quot;false&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;GitFarm architecture diagram showing gateway, backend, sandboxes, repos on disk, and upstream Git server interactions.&quot; class=&quot;rich-image&quot; height=&quot;534&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy82YzdhZWMyYy0zNjE4LTQzNTItYjhlYi00YmMxYTE0YTg1M2EucG5n&quot; width=&quot;1360&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 2: GitFarm Backend architecture.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;br&gt;&lt;p&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Each GitFarm Backend node maintains a single on-disk bare clone for each repository. These clones are kept up to date through an event-driven synchronization strategy (push-based updates) that fetches changes as soon as they become available. We also perform periodic synchronizations with the upstream repository by executing git fetch every 5 minutes, to ensure the repository state remains fresh in the event of missed or delayed events.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;GitFarm doesn’t enforce a single global freshness guarantee across all workloads. Instead, it exposes the repository state that’s eventually consistent with the upstream repository, while allowing clients to explicitly control freshness on a per-request basis.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;For workloads that require the most up-to-date repository state, clients may invoke an explicit git fetch as part of their execution session before running any other Git commands. This ensures that all subsequent operations within the session observe the latest available commits from the upstream repository. Other workloads that tolerate bounded staleness may rely solely on the Backend’s repository synchronization mechanisms to minimize execution latency and avoid redundant fetches.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Another part of the architecture is sandboxes. A sandbox is an ephemeral execution environment (container)&lt;/span&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt; &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;where all Git operations for a given request are performed. Each request runs in complete isolation within its own sandbox and on its own Git checkout, ensuring separation between other requests executing in GitFarm. Sandboxes are pre-initialized to reduce latency, and are protected by strict security boundaries. Access privileges within a sandbox are scoped to the caller’s identity.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Creating a new repository checkout and provisioning a sandbox container on demand for each request is highly resource-intensive. Spinning up a sandbox container typically takes 1-2 seconds, and materializing a repository checkout from a local bare clone (periodically synchronized from upstream) can take up to 3 minutes. Such per-request latencies are unacceptable and don’t scale for high-volume workloads.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;To address this, GitFarm employs a pooling model for both repository checkouts and sandbox containers. The Backend maintains a fixed-size pool of repository checkouts on disk, each synchronized from a local bare clone of the repository. These pre-warmed checkouts are immediately available for request execution.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Similarly, the Backend pre-creates a fixed-size pool of sandbox containers, each initialized with an isolated execution environment and a dedicated mount point. When a request arrives, the Backend acquires an available sandbox from the pool and mounts a repository checkout from the repository pool into the sandbox. This design eliminates on-demand provisioning and allows multiple repositories to be efficiently served within a single cluster.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;With pooling in place, the overhead of providing a sandbox with a ready-to-use repository checkout is reduced to less than a second (as seen in Figure 3), substantially lowering request latency. This optimization enables the system to dedicate resources to executing Git operations rather than incurring repeated initialization costs associated with cold starts.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; style=&quot;text-align: center;&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot; contenteditable=&quot;false&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Line graph showing fluctuating milliseconds values from December 1 to December 19, peaking around 650 ms.&quot; class=&quot;rich-image&quot; height=&quot;534&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy8xMDI1ZjBhYS1kZGY4LTQxNzAtYTAwNS04N2ZkNWZiMmU1ZGIucG5n&quot; width=&quot;880&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 3: P95 Latency to acquire sandbox across Uber monorepos.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;br&gt;&lt;p&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;request-chaining-141&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Request Chaining&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Many automation workflows require executing multiple Git operations sequentially, where the output of one command is consumed by subsequent commands, all operating within the same repository checkout. For example, computing the merge base between 2 branches and publishing it under a derived reference requires capturing the output of git merge-base and using it to push a new Git ref.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;GitFarm supports such workflows through a bidirectional gRPC streaming API that allows clients to execute a sequence of Git commands within a single persistent session, with full access to stdin and stdout. This model preserves a consistent repository checkout across commands while enabling output-dependent command chaining, minimizing connection setup and environment initialization overhead.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;clustering-150&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Clustering&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Clustering in GitFarm refers to deploying multiple GitFarm Backend nodes grouped into logical clusters, each purpose-built to serve a specific, uniform use case, as shown in Figure 4. This model ensures predictable performance characteristics while avoiding noisy neighbor effects caused by heterogeneous workloads sharing the same back-end nodes.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;At Uber, we operate multiple specialized clusters in addition to a generic shared cluster. Specialized clusters are tailored for high-throughput or latency-sensitive workloads with well-defined access patterns, whereas the shared cluster supports lighter-weight use cases and provides a fast, low-friction integration path for new clients, while isolating heterogeneous workloads using cluster-level resource partitioning strategies.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;When a new use case arises, we conduct a review to determine whether the workload can be safely accommodated by an existing cluster or requires a dedicated one. For use cases that warrant isolation, we perform sizing analysis and apply cluster-specific configuration updates (like resource limits, sandbox pool sizing, synchronization policies) prior to onboarding.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Routing decisions are enforced by the GitFarm Gateway, which selects the appropriate cluster for each incoming request based on centrally-managed placement policies. These policies associate each client with a designated cluster, enabling fast and deterministic request routing while preserving workload isolation. This design allows GitFarm to support specialized clusters alongside a shared cluster, without exposing complexity to clients.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; style=&quot;text-align: center;&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot; contenteditable=&quot;false&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;GitFarm architecture diagram showing gateway, routing, backend clusters, and connection to an upstream Git server.&quot; class=&quot;rich-image&quot; height=&quot;640&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9hMWU1MzA5Yi03OTY4LTRjMjQtOTU2MS04ZjM2MDk2MWE4ZGUucG5n&quot; width=&quot;980&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 4: GitFarm clustering setup.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;br&gt;&lt;p&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;api-specification-172&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;API Specification&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;GitFarm exposes an execution interface (Figure 5) that enables clients to execute a sequence of Git commands within a single repository checkout. The interface is designed to support multi-step workflows while preserving repository consistency and minimizing execution overhead.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; style=&quot;text-align: center;&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot; contenteditable=&quot;false&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Function signature and data structure for Exec with Command and CommandResult definitions in code format.&quot; class=&quot;rich-image&quot; height=&quot;684&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy8yYjVjN2Q4OC0wMDJlLTRlMmEtOGQ5ZC02MWJiN2E0NzE5NmQucG5n&quot; width=&quot;1632&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 5: Pseudo interface for the GitFarm API.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;br&gt;&lt;p&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Each Exec invocation establishes a persistent session bound to a single logical repository checkout. Commands are executed sequentially in the order they are received, and the result of each command is returned to the client as a corresponding CommandResult. The alias field is used to correlate command invocations with their outputs.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;GitFarm provides the following guarantees for each execution session:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;&amp;nbsp;Isolation&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: commands execute within an isolated sandbox environment&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;&amp;nbsp;Consistency&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: all commands observe a consistent repository state throughout the session&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;&amp;nbsp;Determinism&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: command execution order matches the order of submission on the session&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Command failures are surfaced via non-zero exit codes and populated standard error output. Fatal errors terminate the session.&lt;/span&gt;&lt;/p&gt;&lt;h2 class=&quot;rich-heading-h2&quot; id=&quot;results-211&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Results&lt;/span&gt;&lt;/h2&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;GitFarm has been in production at Uber since early 2025. Here’s what we’ve seen across 3 representative workloads.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;reducing-80-of-compute-fo-217&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Reducing 80%+ of Compute for Read-Heavy Services&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;One of our production services determines code ownership by scanning CODEOWNERS files across hundreds of thousands of directories in Uber’s monorepos. Before GitFarm, it ran on 6 hosts, each maintaining full local checkouts of every monorepo, synced via cron every 15 minutes.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;After switching to GitFarm API calls, the service dropped its local checkouts entirely. The impact was dramatic:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;CPU&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: over 70 cores → 16 cores &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;(77% reduction)&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; as shown in Figure 6&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Memory&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: 400-600GB → 32 GB (90%+ reduction) as shown in Figure 7&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Startup time&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: 15-20 minutes → under 1 minute&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Beyond raw savings, removing local checkouts eliminated the need for per-instance disk management, repository repair, and bootstrap synchronization. The service now scales independently of monorepo size.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; style=&quot;text-align: center;&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot; contenteditable=&quot;false&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Bar chart showing monthly CPU core counts for 2025, peaking at 136 in June and dropping to 16 from September to December.&quot; class=&quot;rich-image&quot; height=&quot;1264&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy84MjU0NzZjMy00YWNkLTRkYzYtOGM5ZC00MmZhN2E3MDg2MTMucG5n&quot; width=&quot;2856&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 6: CPU cores usage over time.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;br&gt;&lt;p&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; style=&quot;text-align: center;&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot; contenteditable=&quot;false&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Monthly memory usage in GiB for 2025, peaking in June at 669 GiB, then dropping sharply from July onward.&quot; class=&quot;rich-image&quot; height=&quot;1282&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy8wYjdhOWU3OC00ZmJjLTQ0YWMtOTQ5Mi1kNjdkZjE2MDI1MTkucG5n&quot; width=&quot;2856&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 7: Memory usage over time.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;br&gt;&lt;p&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;short-lived-write-oriente-260&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Short-Lived Write-Oriented Git Workflows&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;A production service needs to fetch a branch, compute its merge base against &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;main&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;, and push the result as a new remote ref. With GitFarm, this entire workflow runs within a single execution session: a short sequence of Git commands executed in one sandbox, backed by a warm repository, with no local state required on the client.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The end-to-end p50 latency is about 25 seconds, driven primarily by the Git operations themselves. A &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;git fetch&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; takes roughly 5 seconds, and a &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;git push&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; about 12 seconds. We validated this by comparing against direct Git server interactions, where latencies were nearly identical, confirming that GitFarm introduces minimal overhead.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Without GitFarm, the same workflow would require cloning or syncing the repository on demand, adding minutes of startup time before any Git command can run.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;compliance-auditing-for-b-278&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Compliance Auditing for Bypassed Workflows&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;A compliance service audits commits that bypass standard review checks. It processes 10,000-20,000 events per hour across 9,000 repositories (Uber’s 6 primary monorepos plus thousands of smaller repos). For each flagged commit, it fetches the relevant refs and creates an audit pull request.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Previously, this ran on Buildkite. Each invocation provisioned a worker, set up a workspace, and synced the repository,&amp;nbsp; all before running a few Git commands. The result: &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;p50 latency of 110-160 seconds&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;, dominated by environment setup (as shown in Figure 8).&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;After switching to GitFarm: &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;p50 latency of 20-30 seconds&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;,&amp;nbsp; an over &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;80% reduction&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;. The savings come from eliminating 3 layers of overhead: Buildkite agent scheduling, workspace initialization, and per-invocation repository syncing.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; style=&quot;text-align: center;&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot; contenteditable=&quot;false&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Latency comparison: Buildkite shows higher and more variable p50 latency in seconds than GitFarm, which remains low and steady.&quot; class=&quot;rich-image&quot; height=&quot;484&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy84YmQ4ZmJkMC0xNGRkLTRmODEtYTQzOC0yOGQyMzRiMmExZjQucG5n&quot; width=&quot;806&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 8: P50 execution latency for compliance auditing over a week.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;br&gt;&lt;p&gt;&lt;/p&gt;&lt;h2 class=&quot;rich-heading-h2&quot; id=&quot;what-s-next-302&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;What’s Next?&lt;/span&gt;&lt;/h2&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;While the current GitFarm system is robust and capable, several enhancements are planned to improve efficiency and flexibility in handling diverse workloads.&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Streaming Git operation output.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;&amp;nbsp; Currently, GitFarm returns command results as strings through stdout and stderr. Introducing streaming output would enable real-time feedback for long-running operations and improve responsiveness for interactive use cases.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Support for alternative repository workspaces.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; GitFarm currently operates exclusively on full repository checkouts, which can be inefficient for large repositories. Future support for bare clones and sparse checkouts will allow faster, resource-efficient operations when full working trees are unnecessary.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Extended sessions.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; The existing gRPC connection is well-suited for short-lived sessions but limited by a 30-minute timeout to avoid resource hogging. But there are use cases where enabling long-lived sessions, lasting hours or even days—would benefit dedicated clusters optimized for persistent, single-use-case workloads that require sustained session continuity.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We’re also planning to onboard additional use cases to GitFarm such as:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Mirroring changes across Git upstreams. &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Backup and replication services will leverage GitFarm to continuously monitor repository push events and mirror new commits from a primary Git upstream to a secondary upstream. By centralizing repository state, these services will avoid maintaining local checkouts, while ensuring consistent and up-to-date backups across thousands of repositories.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Merge to trunk (or main).&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;&amp;nbsp; SubmitQueue (Uber’s merge queue system) will leverage GitFarm to validate incoming changes by applying them onto the latest trunk (e.g. main) branch within sandboxed environments. Validated commits will be pushed under a temporary reference for deferred merging. Once CI validation completes, SubmitQueue will invoke GitFarm to cherry-pick commits from this reference, perform the final merge, and push to trunk. This will enable efficient and isolated merge processing without requiring local repository state.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;h1 class=&quot;rich-heading-h1&quot; id=&quot;conclusion-341&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Conclusion&lt;/span&gt;&lt;/h1&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;GitFarm represents a major step forward in how Git operations are executed at Uber’s scale. By providing Git as a Service, it removes the burden of repository management from individual systems, offering a faster, more secure, and scalable alternative to traditional workflows. As the platform evolves with features like streaming output, flexible workspace types, and extended sessions, it’ll continue to enable seamless, high-performance Git operations that keep pac

...

github-actions · 2026-09-01T11:31:14Z

http://localhost:1200/uber/blog/culture - Success ✔️

<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
  <channel>
    <title>Uber Engineering Blog - Culture</title>
    <link>https://www.uber.com/us/en/blog/engineering/culture/</link>
    <atom:link href="http://localhost:1200/uber/blog/culture" rel="self" type="application/rss+xml"></atom:link>
    <description>Engineering Culture - Powered by RSSHub</description>
    <generator>RSSHub</generator>
    <webMaster>contact@rsshub.app (RSSHub)</webMaster>
    <language>en-us</language>
    <image>
      <url>https://tb-static.uber.com/prod/udam-assets/6b65f287-0bee-44e7-868a-e26b8722364e.png</url>
      <title>Uber Engineering Blog - Culture</title>
      <link>https://www.uber.com/us/en/blog/engineering/culture/</link>
    </image>
    <lastBuildDate>Tue, 01 Sep 2026 11:31:06 GMT</lastBuildDate>
    <ttl>5</ttl>
    <item>
      <title>The Transformative Power of Generative AI in Software Development: Lessons from Uber’s Tech-Wide Hackathon</title>
      <description>&lt;div&gt;&lt;h1 class=&quot;rich-heading-h1&quot;&gt;Introduction&lt;/h1&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;This blog examines how we tackled the challenge of &lt;strong class=&quot;rich-bold&quot;&gt;safely adopting generative AI&lt;/strong&gt; to simplify complex coding tasks and meet faster delivery demands.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Generative AI offers a lot of opportunities for &lt;strong class=&quot;rich-bold&quot;&gt;productivity gains &lt;/strong&gt;in the software development lifecycle, as our discipline deals with programming languages, frameworks, and tools that require communication and understanding of both natural and programming languages.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;By organizing an &lt;strong class=&quot;rich-bold&quot;&gt;Uber Tech-wide Hackdayz&lt;/strong&gt; event, we gained valuable insights into the strengths and weaknesses of &lt;strong class=&quot;rich-bold&quot;&gt;generative AI in software development.&lt;/strong&gt; This exercise helped us learn about challenges and opportunities generative AI posed.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Uber teams around the globe used this event to build generative-AI-powered proofs of concept to automate coding, generate tests, improve code quality, and reduce operational load. The event allowed us to &lt;strong class=&quot;rich-bold&quot;&gt;identify and prioritize&lt;/strong&gt; areas where generative AI could potentially&amp;nbsp;be used to measurably boost productivity and spur innovation. To ensure the safety and security of the tools and data used in the Uber Tech-wide Hackdayz, we partnered with a number of cross-functional teams, including legal, technical privacy, security, and open source teams. We also created guidelines for the tools and data that were used in the hackathon.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;hr class=&quot;wp-block-separator has-alpha-channel-opacity&quot; data-wp-block-name=&quot;core/separator&quot;&gt;&lt;/div&gt;&lt;div&gt;&lt;h2 class=&quot;rich-heading-h2&quot;&gt;How can generative AI impact developer productivity?&lt;/h2&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;At Uber and in the software engineering industry as a whole, developers face growing challenges: increasing complexity of software systems, growing demand for faster delivery, and managing diverse tasks like code development, testing, and maintenance–all while ensuring quality and adhering to best practices.&amp;nbsp;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Recent advances in generative AI and large language models have demonstrated advanced competencies in natural language understanding.&amp;nbsp;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;strong class=&quot;rich-bold&quot;&gt;Generative AI offers opportunities to automate and optimize significant aspects of the development process to varying degrees, easing these challenges and boosting developer productivity.&lt;/strong&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;The ability to understand and generate sophisticated text, analyze large datasets, identify patterns, and make predictions makes generative AI a perfect application for software development.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;hr class=&quot;wp-block-separator has-alpha-channel-opacity&quot; data-wp-block-name=&quot;core/separator&quot;&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;strong class=&quot;rich-bold&quot;&gt;“I think a lot of people obviously want to talk about the sexy kind of new consumer applications. I would tell you that I think that the earliest and most significant effect that AI is going to have on our company is actually going to be as it relates to our developer productivity. Some of the tools that we’re seeing are going to allow our devs to kind of be super devs and to be able to innovate more, build more faster, and that will essentially leverage and accelerate innovation across the platform.”&amp;nbsp;&lt;/strong&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;strong class=&quot;rich-bold&quot;&gt;Dara Khosrowshahi &lt;/strong&gt;&lt;strong class=&quot;rich-bold&quot;&gt;&lt;br&gt;&lt;/strong&gt;&lt;strong class=&quot;rich-bold&quot;&gt;CEO of Uber&lt;/strong&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;em class=&quot;rich-italic&quot;&gt;Remarks are from &lt;/em&gt;&lt;a class=&quot;rich-link&quot; href=&quot;https://s23.q4cdn.com/407969754/files/doc_financials/2023/q1/Uber-Q1-23-Prepared-Remarks.pdf&quot; rel=&quot;noreferrer noopener&quot; target=&quot;_blank&quot;&gt;&lt;em class=&quot;rich-italic&quot;&gt;Uber’s Q1’23 earnings call&lt;/em&gt;&lt;strong class=&quot;rich-bold&quot;&gt;&amp;nbsp;&lt;/strong&gt;&lt;/a&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;hr class=&quot;wp-block-separator has-alpha-channel-opacity&quot; data-wp-block-name=&quot;core/separator&quot;&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;&lt;strong class=&quot;rich-bold&quot;&gt;Generative AI can automate simple tasks&lt;/strong&gt;&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;By automating simpler, tedious tasks (generating boilerplate code, fixing linter errors, generating unit tests, etc.), generative AI can help engineers focus on more complex tasks.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;&lt;strong class=&quot;rich-bold&quot;&gt;Generative AI can improve quality &amp;amp; reliability&lt;/strong&gt;&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Since generative AI models are trained on large codebases, they have the potential to provide intelligent suggestions and recommendations based on the existing codebases. Generative AI can analyze the code semantics and could improve the reliability of software by identifying potential bugs, vulnerabilities, or performance issues early in the development cycle.&amp;nbsp;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;&lt;strong class=&quot;rich-bold&quot;&gt;Generative AI has the potential to improve communication&lt;/strong&gt;&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;With its natural language comprehension, generative AI also has the potential to enhance the communication of requirements, by better facilitating collaboration between stakeholders and minimizing misinterpretation.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;&lt;strong class=&quot;rich-bold&quot;&gt;Generative AI could allow for faster prototyping which might lead to quicker validation of ideas&lt;/strong&gt;&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Finally generative AI can more quickly than humans generate prototypes and working code snippets based on high-level specifications, thus enabling developers to validate ideas and fail fast without committing a lot of development time and resources that human development trial and error otherwise would.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;hr class=&quot;wp-block-separator has-alpha-channel-opacity&quot; data-wp-block-name=&quot;core/separator&quot;&gt;&lt;/div&gt;&lt;div&gt;&lt;h2 class=&quot;rich-heading-h2&quot;&gt;Organizing an Uber tech-wide hackathon to unleash innovation&lt;/h2&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;figure class=&quot;rich-figure&quot;&gt;&lt;img alt=&quot;&quot; class=&quot;wp-image-1054049&quot; decoding=&quot;async&quot; height=&quot;490&quot; loading=&quot;lazy&quot; sizes=&quot;auto, (max-width: 1024px) 100vw, 1024px&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/quality=0/width=2160/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9lMTlmNDFiYy03MzAxLTU4YzMtYWRjYi0wYmUzZDkzOWI5MWQucG5n&quot; width=&quot;1024&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;figcaption class=&quot;rich-figure-caption&quot;&gt;Figure 1: Uber Generative AI Tech-wide HackDayz overview&lt;/figcaption&gt;&lt;/figure&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;To explore the potential of generative AI, we recently organized an “&lt;strong class=&quot;rich-bold&quot;&gt;Uber Tech-Wide Hackdayz&lt;/strong&gt;”. This event was distributed across all of Uber’s sites, and brought together interdisciplinary teams from various departments (software engineers, TPMs, PMs, data analysts, QA engineers).&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;figure class=&quot;rich-figure&quot;&gt;&lt;img alt=&quot;&quot; class=&quot;wp-image-1054053&quot; decoding=&quot;async&quot; height=&quot;574&quot; loading=&quot;lazy&quot; sizes=&quot;auto, (max-width: 1024px) 100vw, 1024px&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/quality=0/width=2160/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9hNGI1YTJkYy05OGYxLTU0MjMtYTNjYy1iYjU1NjZjY2M3MzEucG5n&quot; width=&quot;1024&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;figcaption class=&quot;rich-figure-caption&quot;&gt;Figure 2: Uber Generative AI Tech-wide HackDayz in numbers&lt;/figcaption&gt;&lt;/figure&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Overall, the Uber Tech-wide HackDayz was a huge success. 713 engineers around the globe participated and submitted &lt;strong class=&quot;rich-bold&quot;&gt;98 impressive working demos&lt;/strong&gt; across 3 categories: Product Experience, Developer Productivity, and Business Operations.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;The quality and depth of the submitted projects were inspiring and showed the spirit of the company, teaching us ways we can move faster, be creative, take risks, and do more with less.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;hr class=&quot;wp-block-separator has-alpha-channel-opacity&quot; data-wp-block-name=&quot;core/separator&quot;&gt;&lt;/div&gt;&lt;div&gt;&lt;h2 class=&quot;rich-heading-h2&quot;&gt;Applying GenAI to Uber’s Software Development Lifecycle&lt;/h2&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Based on the demos, we found that all phases of the software development lifecycle have the potential to be improved by generative AI.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;Generative AI’s Potential to Improve Developer Productivity&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;figure class=&quot;rich-figure&quot;&gt;&lt;img alt=&quot;&quot; class=&quot;wp-image-1054059&quot; decoding=&quot;async&quot; height=&quot;451&quot; loading=&quot;lazy&quot; sizes=&quot;auto, (max-width: 1024px) 100vw, 1024px&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/quality=0/width=2160/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy8yMGZhMTBhNi1mNWM3LTVkMDEtOTNmYy01NzRmZGI3YmJjMzcucG5n&quot; width=&quot;1024&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;figcaption class=&quot;rich-figure-caption&quot;&gt;Figure 3: Potential use of generative AI in software development lifecycle&lt;/figcaption&gt;&lt;/figure&gt;&lt;/div&gt;&lt;div&gt;&lt;hr class=&quot;wp-block-separator has-alpha-channel-opacity&quot; data-wp-block-name=&quot;core/separator&quot;&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Here are some of the interesting ways generative AI helps, as demonstrated by the prototypes we built during the three-day HackDayz activity:&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;Design document creation, review, and risk assessment&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;h4 class=&quot;rich-heading-h4&quot;&gt;Creating requirements &amp;amp; design documents&lt;/h4&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Although there are many operational requirements that may be common across multiple services and applications, requirements for a new feature may still be incomplete. Generative AI can assist product managers and engineers in capturing complete specifications faster.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;Code &amp;amp; test generation&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Generative AI can generate new text, which can be helpful with the right prompt. This makes it a great fit for improving code by suggesting new features or fixing bugs. It can also be used to generate code that is more efficient or easier to read.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;The diverse set of innovative HackDayz projects in the code and test generation space can&amp;nbsp; be summarized into the following areas:&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h4 class=&quot;rich-heading-h4&quot;&gt;Explain existing code&lt;/h4&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Engineers spend a lot of time reading code every day. It can take months or even years to deprecate older codebases because it is difficult to understand legacy code. Questions about context in other codebases take time and effort to answer through support channels. Furthermore, it can take months or years to build deep domain knowledge in some codebases. Generative AI can be used to explain existing codebases and significantly reduce the time it takes for engineers to be effective on unfamiliar code.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h4 class=&quot;rich-heading-h4&quot;&gt;Generating UI code&lt;/h4&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;One of the pragmatic applications of generative AI was a tool that allows users to create user interfaces by describing them in natural language. This tool generates a code snippet using Uber’s BaseWeb component library, which can be easily copied and pasted into any Uber web application.&amp;nbsp;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;This generated code can also be tested and tailored iteratively within the tool itself, allowing users to make changes and interact with the UI directly. The tool has a memory feature that allows users to build on previous queries and ask for improvements or modifications to the existing code snippet, such as adding or removing components.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h4 class=&quot;rich-heading-h4&quot;&gt;Automated refactoring &amp;amp; centralized migrations&lt;/h4&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Generative AI can automate code refactoring, which is time-consuming and error-prone. This could improve codebases and make it easier to upgrade dependencies or write new code. Open source code and existing tools can be used to achieve this.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Generative AI can help identify migration patterns and automate the refactoring of large codebases. Platform teams spend a lot of time on manual centralized migrations. For example, the Java platform team spent significant effort on migrating the Java framework over the last two years. Automating this process could have saved them a lot of engineering time and allowed them to focus on other business-critical work.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;hr class=&quot;wp-block-separator has-alpha-channel-opacity&quot; data-wp-block-name=&quot;core/separator&quot;&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;Automated fixes&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;h4 class=&quot;rich-heading-h4&quot;&gt;Review &amp;amp; improve existing code&lt;/h4&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Rule-based approaches for static analysis have been around for a while, and they are widely used to provide developers with syntactic corrections and suggestions. We see that generative AI has the potential to take this step to the next level by recommending improvements and automating the follow-up actions.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Traditional static analysis tools, such as &lt;a class=&quot;rich-link&quot; href=&quot;https://github.com/google/error-prone&quot; rel=&quot;noreferrer noopener&quot; target=&quot;_blank&quot;&gt;Errorprone&lt;/a&gt;, are effective for early detection of bugs in Java code. Such tools often flag bugs that are difficult to fix and can impact software reliability.&amp;nbsp;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Current automatic program repair focuses on standard benchmarks, and neglects evaluation on real production code. One of the hackathon projects introduced a novel approach to fix static analysis bugs at scale in an industrial context using generative AI.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;Test generation&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Generating unit tests is one of the tasks where developers shortcut the most. The ability to generate test cases with ease would increase overall test effectiveness and coverage, and consequently system quality.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;One of the most interesting HackDayz projects is the automatic generation of end-to-end tests for iOS and Android apps. The project team used large language models (LLMs) and mobile app images to automatically generate a test script, eliminating the need to write or update any code. The prototype first extracts the sequence of actions required to navigate between two screens specified by the user. It then inputs the action sequence into the LLM to generate the precise code for running the automated tests. Auto-generating end-to-end tests can help improve testing efficiency, reduce costs, and prevent outages.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;Debugging code&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Another innovative project was a debugging tool that used generative AI to create call stacks to&amp;nbsp; provide a clear understanding of the code execution flow. The tool generates all possible call stacks leading to a selected line of code and provides a graphical visualization of the code map within IDE to help developers gain a deeper understanding of their codebase. This approach reduces the time spent on debugging, enhances code understanding and maintainability, and leads to faster resolution of issues and bugs.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;Code review&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Automating code reviews is one of the more impactful areas to apply generative AI. On average our developers spend ~60 minutes per PR in addressing review comments. Several HackDazy projects automated the code review process by leveraging generative AI to resolve review comments, and save developers time and effort.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;Failure classification&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;It is critical for engineering velocity to understand the reasons for CI failures and respond to them quickly. CI build logs frequently include infrastructure-specific records as well as user-specific build output and errors. Understanding the actual failure often requires scrolling through the build output and navigating to specific error details. Furthermore, CI failures that are specific to user errors, such as programmatic errors in a source code, should be quickly exposed to the user, while infrastructure errors are frequently resolved through automatic retries. One of the HackDayz projects used the build logs to automatically categorize these failures and take the next set of actions to speed up the process.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;Localization&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Currently, it can take over a week to localize content using human translators. We believe that using generative AI can speed up the process while improving accuracy and efficiency. One hackathon project used generative AI to translate content into multiple languages and then verified the translations with human agents. This project proved that generative AI can be used to enhance the efficiency, accuracy, and speed of localization.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;Knowledge base&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;There were several projects to improve the collaboration and reduce the support load for our developers.&amp;nbsp;&amp;nbsp;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;One of the project teams built a tool that can automate level-1 support queries in Slack channels. They built a Slack bot that can answer support questions and retrieve real-time information from the associated services. Developers interacting with the bot would get instant responses rather than waiting for support from the team members in different time zones.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;The lack of meaningful documentation in large codebases can be supplemented by auto-generated documentation, making it easier for engineers to onboard and understand new libraries or projects.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Another impactful project focusing on the knowledge base space had the ambitious goal of revolutionizing software documentation by using generative AI. This project team built a prototype tool to automatically generate up-to-date documentation based on PRs, capturing dependencies, and providing high-level overviews. The approach involves using generative AI to analyze and create summaries for each document, recursively summarizing them at higher levels for the entire project.&amp;nbsp;This tool attempts to streamline documentation, thus empowering developers to focus on impactful work while ensuring accurate documentation that reflects the current codebase.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;hr class=&quot;wp-block-separator has-alpha-channel-opacity&quot; data-wp-block-name=&quot;core/separator&quot;&gt;&lt;/div&gt;&lt;div&gt;&lt;h2 class=&quot;rich-heading-h2&quot;&gt;Risks when using Generative AI&lt;/h2&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;Quality of the output generated&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;From our experience LLMs may generate buggy code, as well as spreading error-prone code patterns. LLMs also need a lot of data to learn how to use language correctly. If this data is biased or flawed, the LLMs can replicate these problems and give wrong or unfair suggestions. This means that the quality and accuracy of the data used to train LLMs is very important, as mistakes can get repeated and cause further issues. For example, LLMs used to help with coding can also carry over security problems from the training data. So, it’s crucial to be cautious with the data used for training LLMs.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;Explainability of the decisions&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;It is important to be able to understand and explain the reasoning behind the AI-generated output, especially when these outputs are used in business-critical settings. Not being able to trace the AI-generated code to its source can create trust issues.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;hr class=&quot;wp-block-separator has-alpha-channel-opacity&quot; data-wp-block-name=&quot;core/separator&quot;&gt;&lt;/div&gt;&lt;div&gt;&lt;h2 class=&quot;rich-heading-h2&quot;&gt;What’s Next&amp;nbsp;&lt;/h2&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;Integrating generative AI into our development processes&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;As we thought about addressing some of the challenges we discussed in the previous section, we decided to build a single API gateway designed to provide guardrails, help address PII redaction, detect hallucination, rate-limit, load balance, and capture audit logs.&amp;nbsp;&amp;nbsp;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;strong class=&quot;rich-bold&quot;&gt;Uber Generative AI API Gateway&lt;/strong&gt; is the unified gateway for LLM access at Uber. It provides one standard API for external and internal LLMs. Uber Generative AI API Gateway provides assistance with PII redaction, safety and Uber policy guardrails, hallucination detection, and other common platform functionalities that are essential for safely and efficiently using generative AI at Uber.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;figure class=&quot;rich-figure&quot;&gt;&lt;img alt=&quot;&quot; class=&quot;wp-image-1054076&quot; decoding=&quot;async&quot; height=&quot;853&quot; loading=&quot;lazy&quot; sizes=&quot;auto, (max-width: 1024px) 100vw, 1024px&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/quality=0/width=2160/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy83Mjg0ZGNhMS1mNDQ0LTVkMDItODA0Yy05ZWY4NTM0Yjc0MDIucG5n&quot; width=&quot;1024&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;figcaption class=&quot;rich-figure-caption&quot;&gt;Figure 4: Uber’s Generative AI API Gateway&lt;/figcaption&gt;&lt;/figure&gt;&lt;/div&gt;&lt;div&gt;&lt;hr class=&quot;wp-block-separator has-alpha-channel-opacity&quot; data-wp-block-name=&quot;core/separator&quot;&gt;&lt;/div&gt;&lt;div&gt;&lt;h1 class=&quot;rich-heading-h1&quot;&gt;Conclusion&lt;/h1&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;By organizing the Uber Tech-wide HackDayz, our engineers got hands-on experience and training with generative AI. Uber employees collaborated with their peers across time zones and disciplines for this event. It encouraged employees to think creatively and explore ground-breaking ideas of how generative AI could be used to create business impact at Uber.&amp;nbsp; As a result, HackDayz resulted in a large number of high-quality and practical ideas that we’ll explore and invest in further this year.&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;We look forward to applying our learnings and insights in ways that may boost developer productivity, create new customer experiences, and improve Uber’s business in new ways.&lt;/p&gt;&lt;/div&gt;</description>
      <link>https://www.uber.com/us/en/blog/the-transformative-power-of-generative-ai/</link>
      <guid isPermaLink="false">https://www.uber.com/us/en/blog/the-transformative-power-of-generative-ai/</guid>
      <pubDate>Wed, 02 Aug 2023 16:00:00 GMT</pubDate>
      <author>Adam Huda, Anshu Chadha, Serdar Badem, Ali-Reza Adl-Tabatabai, Brandon Lico</author>
      <enclosure url="https://tb-static.uber.com/prod/udam-assets/92a2fe2a-f5aa-59d3-a976-b9d823d69e90.jpg" type="image/jpeg"></enclosure>
      <category>Engineering</category>
      <category>Backend</category>
      <category>Culture</category>
      <category>AI / ML</category>
    </item>
    <item>
      <title>How My Uber Internship Expanded My Horizons</title>
      <description>&lt;p class=&quot;rich-paragraph&quot;&gt;Prior to this internship, my university’s professors, fellow students, and career counselors gave me a dystopian picture of developing software at a large company: a silent, gray room with no windows populated by rows of desks stretching to unseen walls, engineers spending eight uninterrupted hours poring over tedious work.&amp;nbsp;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Fortunately, my experience at Uber was nothing like that.&amp;nbsp;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;From my first day in the office, I’ve found nothing but friendly folks and colorful, sunny spaces. This internship overturned my assumptions about the industry, as I was free to pursue projects that interested me without guilt, supported by a strong and welcoming community.&amp;nbsp;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;This summer, I’ve had the pleasure of being an intern on the Productivity Applications team. For my summer project, I’ve been working on a Slackbot to assist in searching Whober (Uber’s internal employee directory) and group/channel edits.&amp;nbsp;&amp;nbsp;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;My first assignment was to learn &lt;a class=&quot;rich-link&quot; href=&quot;https://go.dev/&quot; rel=&quot;noreferrer noopener&quot; target=&quot;_blank&quot;&gt;Golang&lt;/a&gt;. In my college experience, we tend to explore the programming language in theory, following a series of projects that align with using specific theoretical concepts from the language in a particular context. The pressure to complete assignments on time limits our ability to explore further. Here, on the other hand, instead of a series of strict modules oriented toward Uber’s Go platforms, I had the freedom to explore the Go documentation and use the available materials in the OneLogin portal, and my mentor let me figure out how to learn the language in the way and timeframe that worked best for me. I even got to build a small-scale social media platform to practice my Go skills. After a couple of weeks, I was comfortable enough with Go that I was set up on the Go monorepo to work on testing suites and get a feel for what developing my summer project would be like.&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;For my first ticket, the Whober team had set up a new widget to display the chief of staff for employees that have one, and I got to design its unit tests. My mentor walked me through how the Go monorepo worked, how to generate coverage reports, and what the best practices were for setting up my environment and pushing my changes. Then I was let loose. I was allowed to figure out how to navigate the monorepo on my own, which helped me understand how the controllers, classes, and APIs all fit together to make development clean. After that, my new testing class was committed, approved by my reviewers, and sent to production. The moment my change landed was exhilarating. Even though I had only created a small testing class in the Go repository, I felt that I had made use of the skills I had only used theoretically or as part of my curriculum in college. Concepts such as testing, notation, and program structure were ideas that I had learned previously but never been able to apply independently, and my first few weeks allowed me to do so.&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;From that point onwards, I received more tickets. I was able to work within the web monorepo where I was able to change the chief of staff and assistant widgets order and fix a bug involving phone number entry. I learned the differences in development and production flow between the Go and web monorepos, how to develop components that would operate independently of one another so that the features on the site could grow, and how to read and understand the work of others before me.&amp;nbsp;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Once I had a firm understanding of Go and how the monorepos worked, I was able to start working on my summer project: the Whober Slackbot (Whobot for short). The Whobot currently interfaces with Whober to pull profiles for search by email and Slack mention and display them. It is capable of creating interactive message blocks and formatting message content.&amp;nbsp;&amp;nbsp;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;The first stage of my project was research, so I began by reading through the Slack API and looking at a similar implementation of a Slackbot built in Go, Urate, under the guidance of my mentor. Reading the documentation helped me develop a more full sense of a Slackbot’s capabilities and how to implement those features. I found the Go monorepo especially helpful in finding similar implementations. From these readings, I developed a sense of what it would take to send out messages from the Slackbot. I set up functions on how to format different messages and documented how to process the return values in an API for later use. With that knowledge, I was able to set up a series of Slack commands to figure out how to create interactive messages and formatting. Once I had figured out how to set those up, my team taught me how to make a gateway for requests to Whober, and then finally, I finished the function for searching and displaying employee information by request in Slack. Working on this project taught me a lot about how project development at a company works, how to learn freely and without fear of failure, and how important it is to work in a team.&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Working with my team is what changed my worldview the most. I found constant support whenever I encountered an issue or had a question, meetings were engaging and fun, and they really helped me feel part of a group here at Uber. From them, I learned that I could offer suggestions, be part of the conversation, and ask questions about what’s going on or how something works. They also gave me a much clearer and wider view than I had ever received at college about how diverse a person’s professional career could be.&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;Working here at Uber has shown me that sometimes the mental image I have couldn’t be further from the truth. I came into Uber with a rather limited view of the industry but discovered an engaging environment full of friendly people, work that broadened the horizons of software I could build, and normalization of exploration and failure that I lacked at my university. Thank you so much to my team, my friends, and everyone else who made my experience so wonderful!&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;If you want to work on open, interesting projects in a welcoming environment, then please consider joining Uber as an intern this coming year! More information regarding the internship program can be found &lt;a class=&quot;rich-link&quot; href=&quot;https://www.uber.com/us/en/careers/teams/university/&quot; rel=&quot;noreferrer noopener&quot; target=&quot;_blank&quot;&gt;here&lt;/a&gt;!&amp;nbsp;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;“Traveling Light” by Ardenswayoflife is licensed under CC BY-SA 2.0.&lt;/p&gt;</description>
      <link>https://www.uber.com/us/en/blog/how-my-uber-internship-expanded-my-horizons/</link>
      <guid isPermaLink="false">https://www.uber.com/us/en/blog/how-my-uber-internship-expanded-my-horizons/</guid>
      <pubDate>Wed, 16 Nov 2022 16:00:00 GMT</pubDate>
      <author>Gabby Franklin</author>
      <enclosure url="https://tb-static.uber.com/prod/udam-assets/ce9aa40b-dc36-530d-8c81-e52f21bdc3d3.jpg" type="image/jpeg"></enclosure>
      <category>Engineering</category>
      <category>Culture</category>
    </item>
    <item>
      <title>ML Education at Uber: Program Design and Outcomes</title>
      <description>&lt;div&gt;&lt;h1 class=&quot;rich-heading-h1&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Introduction&lt;/span&gt;&lt;/h1&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;If you have read our previous article, &lt;/span&gt;&lt;a class=&quot;rich-link&quot; href=&quot;https://www.uber.com/us/en/blog/ml-education-at-uber/&quot;&gt;&lt;i class=&quot;rich-italic&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;ML Education at Uber: Frameworks Inspired by Engineering Principles&lt;/span&gt;&lt;/i&gt;&lt;/a&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;, you have seen several examples of how Uber benefits from applying Engineering Principles to drive the ML Education Program’s content design and program frameworks.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;In this follow-up, we will dig deeper into what we believe to be other unique aspects of ML Education at Uber: our approach to &lt;/span&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Content Components, Content Delivery, Observability, and Marketing &amp;amp; Reach&lt;/span&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h2 class=&quot;rich-heading-h2&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Our Library of Content Modules&amp;nbsp;&lt;/span&gt;&lt;/h2&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;As mentioned in our first post, Modularity and Extensibility were huge motivations in building our learning content out of unique components. Having a uniform library of content modules provides 2 major benefits:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;ol class=&quot;rich-list-ol&quot;&gt; &lt;li class=&quot;rich-list-item&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Ability to tailor the inclusion or exclusion of component types based on the unique scope of a learning topic&lt;/span&gt;&lt;/li&gt; &lt;li class=&quot;rich-list-item&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Consistent delivery of component types across course offerings, which reduces cognitive load for users&lt;/span&gt;&lt;/li&gt; &lt;/ol&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Our library is always evolving, but 4 types of core modules have been with our program since the beginning: &lt;/span&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Prerequisites, Theory, Hands-on Codelabs, and “Getting Started” Packs&lt;/span&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;&lt;strong class=&quot;rich-bold&quot;&gt;Prerequisites&lt;/strong&gt;&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Courses may enforce 2 primary types of prerequisites: knowledge and action-based.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;An example of a knowledge prerequisite is: “&lt;/span&gt;&lt;i class=&quot;rich-italic&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;understanding of basic ML concepts and familiarity with Uber’s ML development workflow.&lt;/span&gt;&lt;/i&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;” Action-based prerequisites vary from things like requesting permissions and access to tooling or completing a unique development setup prior to the course.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;To complete most of the hands-on modules in our training courses, users are required to build a unique development setup on their local machine. We streamlined this setup process by repurposing the same development setup across most of our course offerings and storing all technical assets (like datasets) in a central repository within Uber’s ML monorepo. This enables users to complete one setup once, rather than a new setup each time they attend an ML Education course.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Prerequisite development setups are facilitated by Uber’s internal tool called “codelabs.”&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;&lt;b class=&quot;rich-bold&quot;&gt;&lt;br&gt; &lt;/b&gt;&lt;b class=&quot;rich-bold&quot;&gt;Theory and Q&amp;amp;A&lt;/b&gt;&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Visual learners were welcomed to ML Education offerings with slides, diagrams, and graphs, while auditory learners were welcomed with recordings and voice overs. While we often see in our post-course surveys that participants rate the hands-on components as being most helpful, theory remains critical to our courses. We feel that theory compliments our hands-on activities in 3 key ways:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;ul class=&quot;rich-list-ul&quot;&gt; &lt;li aria-level=&quot;1&quot; class=&quot;rich-list-item&quot; style=&quot;font-weight: 400;&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Introduces the functionality of the tools or services&lt;/span&gt;&lt;/li&gt; &lt;li aria-level=&quot;1&quot; class=&quot;rich-list-item&quot; style=&quot;font-weight: 400;&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Ties the specific subject back to Uber’s broader ML ecosystem&lt;/span&gt;&lt;/li&gt; &lt;li aria-level=&quot;1&quot; class=&quot;rich-list-item&quot; style=&quot;font-weight: 400;&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Details how the specific subject should be applied in the context of Uber use cases&lt;/span&gt;&lt;/li&gt; &lt;/ul&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Mindful that long stints of theory can be overwhelming or disengaging to some learner types, we built our theory modules to incorporate:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;ul class=&quot;rich-list-ul&quot;&gt; &lt;li aria-level=&quot;1&quot; class=&quot;rich-list-item&quot; style=&quot;font-weight: 400;&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Pre-recorded instructor videos, rather than support real-time instructor facilitation&lt;/span&gt;&lt;/li&gt; &lt;li aria-level=&quot;1&quot; class=&quot;rich-list-item&quot; style=&quot;font-weight: 400;&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Checkpoint questions to recap sections and inspire open Q&amp;amp;A&lt;/span&gt;&lt;/li&gt; &lt;/ul&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;&lt;b class=&quot;rich-bold&quot;&gt;Hands-on Codelabs&lt;/b&gt;&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Specifically for kinesthetic learners, hands-on resources are essential. We embedded interactive codelabs within each course, and prompted students with questions to think critically about what they were learning. We also included case studies of current Uber applications that grounded the learnings back to reality. Across all courses facilitated live in H1 2022, we found that on average 88% of attendees felt the hands-on codelab module was the most helpful component of the course.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;These hands-on exercises contain some or all of the following:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;ul class=&quot;rich-list-ul&quot;&gt; &lt;li aria-level=&quot;1&quot; class=&quot;rich-list-item&quot; style=&quot;font-weight: 400;&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Interactive Jupyter notebooks&lt;/span&gt;&lt;/li&gt; &lt;li aria-level=&quot;1&quot; class=&quot;rich-list-item&quot; style=&quot;font-weight: 400;&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Uber-specific datasets&amp;nbsp;&lt;/span&gt;&lt;/li&gt; &lt;li aria-level=&quot;1&quot; class=&quot;rich-list-item&quot; style=&quot;font-weight: 400;&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;ML project scaffolds or model templates for students to replicate&lt;/span&gt;&lt;/li&gt; &lt;li aria-level=&quot;1&quot; class=&quot;rich-list-item&quot; style=&quot;font-weight: 400;&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Assets stored in Uber’s ML monorepo to streamline environment setup and access to course assets that participants need to use locally&lt;/span&gt;&lt;/li&gt; &lt;/ul&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;&lt;b class=&quot;rich-bold&quot;&gt;“Getting Started” Packs&lt;/b&gt;&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;After obtaining knowledge on a new subject, it is sometimes difficult to know exactly how to apply it to a real-world use case. Our “Getting Started” packs aim to reduce this uncertainty for users post-course.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;We know our hands-on codelabs do an excellent job reinforcing what a user has just learned by allowing them to explore a tool or service in a controlled environment. However, in order to teach a diverse range of skill sets at scale, some exercises must use pre-determined datasets and existing projects are cloned. This removes an element of decision-making from the ML development workflow for users, even in our most complex courses.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;This may cause a user to be left with a feeling of “great, now what?” when preparing to dive headfirst into using our tools/services to build their own ML solution from scratch. To aid the users who need help taking their first pass at “real world” application of the course’s content, we equip each class of participants with a starter pack.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;The structure of each starter pack varies per topic, but we commonly provide things like direction to relevant project templates, high-level guidance for building their own project, and which resources or helpdesk channels to leverage if they need additional help. Providing this stepping stone also gives us a clear view into the impact metrics we establish for a given course (see the “Observability” section for more).&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h2 class=&quot;rich-heading-h2&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Content Delivery&amp;nbsp;&lt;/span&gt;&lt;/h2&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Once a new course’s components are determined, the ML Education team decides what delivery format best suits the course subject matter.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;When ML Education’s inaugural pilot courses launched, we delivered courses using the only delivery format we had available at the time: Instructor-led Live Facilitation. Today, we have 3 delivery formats, each of which evolved from a combination of needs of different learning topics, participants, and instructors.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;figure aria-describedby=&quot;caption-attachment-9685&quot; class=&quot;rich-figure&quot; style=&quot;width: 656px&quot;&gt;&lt;img alt=&quot;&quot; class=&quot;wp-image-9685&quot; decoding=&quot;async&quot; height=&quot;346&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/quality=0/width=2160/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9jYTYyOWM2Mi01N2M1LTUzZDItOTVmOC04OTNjNTU2MjVmN2QucG5n&quot; width=&quot;656&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;figcaption class=&quot;rich-figure-caption&quot;&gt;Figure 1: Delivery methods for ML Education content.&lt;/figcaption&gt;&lt;/figure&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;&lt;b class=&quot;rich-bold&quot;&gt;Live&lt;/b&gt;&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Our live delivery method encapsulates a full learning experience that commonly includes theory, applications, and hands-on activity components. Each theory module contains a combination of pre-recorded and live content, and our hands-on activities are facilitated through Uber codelabs.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;figure aria-describedby=&quot;caption-attachment-9686&quot; class=&quot;rich-figure&quot; style=&quot;width: 3292px&quot;&gt;&lt;img alt=&quot;&quot; class=&quot;size-full wp-image-9686&quot; decoding=&quot;async&quot; height=&quot;738&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/quality=0/width=2160/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy83MGZiYWU1MS04YzhkLTU3NTUtYTAyYi00MTMwMzhjOTMxYjUucG5n&quot; width=&quot;3292&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;figcaption class=&quot;rich-figure-caption&quot;&gt;Figure 2: Live content delivery method explained.&lt;/figcaption&gt;&lt;/figure&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Using pre-recorded material in a “live” session might sound counterintuitive, but we leverage recordings to meet time constraints and to minimize variance in content delivery. Where does that leave our on-call instructors? Their primary responsibility is to facilitate the Q&amp;amp;A that follows sections of theory and further elaborate on any important concepts in real time.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;To facilitate kinesthetic learning, we include an instructor-guided lab that applies theory content to a more tangible example, giving students the chance to get hands-on experience with the content and tooling just demonstrated.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;We commonly use live delivery for high-priority topics that range from intermediate to advanced level of difficulty. For instance, adoption of Canvas (Uber’s internal tool that brings reproducible code to the ML development process) was highly prioritized by UberAI in 2021. The ML Education program developed curriculum in response to the desired behavior change, and facilitated several live Canvas training instances in the US and India.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;So far, user feedback has overwhelmingly indicated that live content delivery is the most effective method for topics where we want to drive behavior changes like tool adoption. In live sessions, users can work closely with our ML experts in real time to fully understand theory concepts and troubleshoot throughout the hands-on course module. By the time they leave the session, users have had the opportunity to clarify anything they need to know in order to immediately implement the tools, services, and/or practices they just learned into a production use case.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;&lt;b class=&quot;rich-bold&quot;&gt;Semi-Guided&lt;/b&gt;&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;This is the ML Education Program’s blended learning approach to content delivery. It’s not quite as involved as a live course, but not as hands-off as an online course.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;In a semi-guided course, participants assume most of the ownership of absorbing theory content (see figure below).&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;figure aria-describedby=&quot;caption-attachment-9687&quot; class=&quot;rich-figure&quot; style=&quot;width: 3267px&quot;&gt;&lt;img alt=&quot;&quot; class=&quot;size-full wp-image-9687&quot; decoding=&quot;async&quot; height=&quot;658&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/quality=0/width=2160/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy81NGRhMjM2ZC0yY2IyLTU5YzgtYTNlNy05YjAxNjY1OGM4NGEucG5n&quot; width=&quot;3267&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;figcaption class=&quot;rich-figure-caption&quot;&gt;Figure 3: Semi-guided content delivery method explained.&lt;/figcaption&gt;&lt;/figure&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;This format allows users to learn theory content independently rather than with the entire group. Instructors are utilized only where and when they are needed, which is largely during the course’s codelab (hands-on) components.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;We have found that semi-guided delivery works well for topics where:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;ol class=&quot;rich-list-ol&quot;&gt; &lt;li aria-level=&quot;1&quot; class=&quot;rich-list-item&quot; style=&quot;font-weight: 400;&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Theory content is still important, but high-level enough where users can confidently consume key takeaways on their own&lt;/span&gt;&lt;/li&gt; &lt;li aria-level=&quot;1&quot; class=&quot;rich-list-item&quot; style=&quot;font-weight: 400;&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;A thorough explanation from a subject matter expert is not required to understand key concepts (independent reading is sufficient)&amp;nbsp;&amp;nbsp;&lt;/span&gt;&lt;/li&gt; &lt;/ol&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;It is more likely that beginner-level topics will fit the requirements listed above. Our &lt;/span&gt;&lt;i class=&quot;rich-italic&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Semi-Guided Intro to Regression&lt;/span&gt;&lt;/i&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt; and &lt;/span&gt;&lt;i class=&quot;rich-italic&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Intro to Classification&lt;/span&gt;&lt;/i&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt; courses have obtained overwhelmingly positive feedback to date, both earning 100% CSAT scores in H1 2022.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h3 class=&quot;rich-heading-h3&quot;&gt;&lt;b class=&quot;rich-bold&quot;&gt;Online&lt;/b&gt;&lt;/h3&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Our commitment to &lt;/span&gt;&lt;i class=&quot;rich-italic&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;reproducibility&lt;/span&gt;&lt;/i&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt; is exemplified by online courses. For every live course that matures, an online version is created, stored, and advertised to users. The online course versions contain all the same components as a topic’s live version, but available at a time more convenient to the user.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;An element of engagement is inevitably lost when leveraging online delivery. We combat this by providing Q&amp;amp;A support through dedicated Slack channels for online course participants.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;We have found extreme value in offering a near 1:1 mapping of live to online course offerings. Online delivery dramatically increases accessibility, which is critical for a large-scale global organization. At Uber, our engineers implement ML solutions around the globe and we are committed to ensuring all engineers have access to our unified library of learning resources regardless of location. Scheduling time-zone-inclusive live course instances can be challenging, so having an online offering available to anyone at any time is necessary in order for us to fulfill our commitment to global accessibility.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;While each of our 3 delivery formats are uniquely their own, we strive to bring consistencies across each method where applicable. For example, we provide users with a consistent approach to troubleshooting and Q&amp;amp;A regardless of delivery format.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Each course is allocated a unique Slack channel where participants submit questions or request help troubleshooting as they complete the course. We like using Slack because it allows for continued engagement between participants and instructors post-course. All questions are captured and archived which enables users to search questions asked in previous live sessions.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Additionally, each delivery method contains the same branded templates and artifacts. Course introductions and conclusions are organized and facilitated in a similar manner regardless of delivery in an attempt to reduce cognitive load on users, as well as provide an indistinguishable “look/feel” for all ML Education learning resources.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Which delivery method works best? There’s no right answer for that. They all satisfy different needs (whether it be needs of the content, audience, or both).&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;h2 class=&quot;rich-heading-h2&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Observability&lt;/span&gt;&lt;/h2&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;In the early days, when ML Education was an informally established program of 2 courses, the initiative was considered to be a pilot until value proved otherwise.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;How were the ML Education program creators able to capture and communicate this value so that the program could scale to what it is today? By weaving a disciplined&lt;/span&gt;&lt;i class=&quot;rich-italic&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt; observability&lt;/span&gt;&lt;/i&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt; strategy into the program’s foundation.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;When we think of observability, we do so in the context of our tooling as well as our individual courses. The capabilities embedded in Uber’s ML tooling provides ML Education with the opportunity to observe the positive business impact our courses provide to the organization.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Thankfully, Uber’s internal ML infrastructure is deeply integrated with Uber’s existing comprehensive &lt;/span&gt;&lt;a class=&quot;rich-link&quot; href=&quot;https://www.uber.com/us/en/blog/observability-at-scale/&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;observability toolstack&lt;/span&gt;&lt;/a&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt; (e.g., &lt;/span&gt;&lt;a class=&quot;rich-link&quot; href=&quot;https://m3db.io/&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;M3&lt;/span&gt;&lt;/a&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;, &lt;/span&gt;&lt;a class=&quot;rich-link&quot; href=&quot;https://www.uber.com/us/en/blog/elk/&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;ELK&lt;/span&gt;&lt;/a&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;). Each step of the ML workflow is checkpointed and logged, so we can easily identify the points of friction and dropoff of users as they are going through our codelabs. To reduce friction for users and drive towards completion, we made sure every step in the codelab is resumable or bypassable if a user so chooses. Instructors can also easily use our logged metrics to build dashboards with &lt;/span&gt;&lt;a class=&quot;rich-link&quot; href=&quot;https://grafana.com/&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Grafana&lt;/span&gt;&lt;/a&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt; to track engagement or to set up &lt;/span&gt;&lt;a class=&quot;rich-link&quot; href=&quot;https://www.uber.com/us/en/blog/observability-at-scale/&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;alerts&lt;/span&gt;&lt;/a&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt; to check if a component used in the hands-on portion is broken. One of our favorite use cases for this is monitoring the spike in new users during and following our annual internal machine learning conference, UberML.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Customer testimonials are a less systematic, but incredibly powerful approach to observability. At any given time we have former course attendees reach out to share their ML accomplishments following the training courses they’ve taken. For example, an engineer shared that following his attendance of &lt;/span&gt;&lt;i class=&quot;rich-italic&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Intro to Deep Learning&lt;/span&gt;&lt;/i&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt; he applied the skills he learned to productionize a phishing/fraud detection model that realized substantial cost savings for Uber. To more concretely establish our attribution models, we are also considering publishing specific docker images as part of ML Education and to track their usages as base images.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;We understand that in general, measuring shifts in behavior is really difficult. But having excellent observability capabilities in our tooling get us one step closer to accurately capturing a user’s motivation for adopting a new tool and appropriately attributing it to the investment we make in ML educational resources.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;Knowing that business impact can be a less concrete area of observability, we couple it with several other KPIs to enable observability for overall program health and success.&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span style=&quot;font-weight: 400;&quot;&gt;ML Education defines “health” from 4 major points of view:&lt;/span&gt;&lt;/p&gt;&lt;/div&gt;&lt;div&gt;&lt;ul class=&quot;rich-list

...

github-actions · 2026-09-01T11:31:15Z

http://localhost:1200/uber/blog/data - Success ✔️

<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
  <channel>
    <title>Uber Engineering Blog - Data</title>
    <link>https://www.uber.com/us/en/blog/engineering/data/</link>
    <atom:link href="http://localhost:1200/uber/blog/data" rel="self" type="application/rss+xml"></atom:link>
    <description>Data Engineering - Powered by RSSHub</description>
    <generator>RSSHub</generator>
    <webMaster>contact@rsshub.app (RSSHub)</webMaster>
    <language>en-us</language>
    <image>
      <url>https://tb-static.uber.com/prod/udam-assets/6b65f287-0bee-44e7-868a-e26b8722364e.png</url>
      <title>Uber Engineering Blog - Data</title>
      <link>https://www.uber.com/us/en/blog/engineering/data/</link>
    </image>
    <lastBuildDate>Tue, 01 Sep 2026 11:31:06 GMT</lastBuildDate>
    <ttl>5</ttl>
    <item>
      <title>Running Cost-Efficient Export Workloads at Uber</title>
      <description>&lt;h1 class=&quot;rich-heading-h1&quot; id=&quot;introduction-2&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Introduction&lt;/span&gt;&lt;/h1&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;At Uber, several offline export workloads retrieve small record sets from large historical datasets. Today, these selective queries can still trigger full-table scans. On a Google Cloud Storage™-backed lakehouse like that at Uber, repeated reads keep large table regions hot, increasing storage, retrieval, metadata-operation, GCS egress, and latency costs. This blog shows how Apache Hudi™ column stats in conjunction with sorting/clustering can reduce scan volume and make these workloads more cost-efficient in that lakehouse environment.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;what-are-export-workloads-433&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;What Are Export Workloads?&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Export workloads are common in compliance, privacy, and operational export flows, where the result set is small but the search space spans the entire history of data.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Export workloads typically have the following characteristics:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;They issue point queries or narrow predicate queries over large transaction or fact tables stored in Hudi and partitioned by date&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;They run repeatedly, often multiple times per week.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;They repeat the same query pattern, but with different parameters each time.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;4&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The target record set is only a small fraction of the overall dataset&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;5&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Despite their selectivity, they perform full-table scans to find the relevant records&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;One concrete example of this workload pattern is a DSAR request. A DSAR (Data Subject Access Request) is a legal request by an individual to access the personal data an organization holds about them. In practice, fulfilling a DSAR often translates into a narrow query pattern like the one shown in Figure 1.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;SQL query selects trip_id, from_city, to_city from trip_table where trip_date is on or before today.&quot; class=&quot;rich-image&quot; height=&quot;338&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9kOTJmZTlkOS1lMGJmLTRhNmYtYTZiMS00OThmNmJmMTI1NDYucG5n&quot; width=&quot;1958&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 1: Narrow query pattern.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;br&gt;&lt;p&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;gcs-auto-class-tiering-434&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;GCS Auto-Class Tiering&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Uber uses GCS (Google Cloud Storage) as the storage layer for its lakehouse. GCS offers multiple storage classes that trade off storage cost, retrieval cost, and access latency. Understanding these tiers is important because the cost efficiency of export workloads depends not only on how much data is stored, but also on how often that data is touched.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Comparison of Standard, Nearline/Coldline, and Archive storage for data access frequency, cost, and retention.&quot; class=&quot;rich-image&quot; height=&quot;474&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy8wNWZmM2QxNS03NTFjLTQ4ZGMtOTdkNS03NDIwZDA5YTY1ODIucG5n&quot; width=&quot;1366&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 2: Storage tier distinctions in GCS.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;GCS also provides auto-class tiering, which automatically moves objects between storage classes based on observed access patterns. In principle, this allows colder data to move into cheaper storage tiers over time. However, this mechanism is undermined by workloads that repeatedly touch all the files preventing the autoclass tiering feature to kick-in. Repeatedly scanning file metadata or footers can also keep objects hot and prevent them from transitioning into lower-cost storage classes. A desired&amp;nbsp; distribution based on typical access patterns&amp;nbsp; is approximately 60% Standard, 15% Nearline, 10% Coldline, and 15% Archive. When repeated reads on all the files keep colder data hot, 100 percent of data ends up in Standard tier and stays there. The result is you pay the highest storage cost. &lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;export-workloads-break-au-435&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Export Workloads Break Auto-Class Tiering&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Export workloads disrupt auto-class tiering because they combine high selectivity with a large historical search space. A query may return only a few records while still touching all the partitions and files.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Analytics workloads often process more data but over recent partitions, letting older data go cold. Export workloads do the opposite: they repeatedly search across history for a small target set. This repeated needle in a haystack pattern where the haystack is the full table will keep data hot, prevent auto-class transitions, and increase storage and retrieval costs.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Consider the example of the &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;trip_table &lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;again.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Figure 3 shows what a typical analytics query might look like.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;SQL query counts trips between cities in the last two days, grouped by departure and destination cities.&quot; class=&quot;rich-image&quot; height=&quot;250&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy81MDY0MGUzMS0wNDFiLTRiZTUtODQxZC04NjEzNTZjOTIxNjkucG5n&quot; width=&quot;1302&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 3: A typical analytics query.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;This query is broad in the data it aggregates, but narrow in time range because it focuses on recent partitions.&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;In contrast, Figure 4 shows what an export query looks like.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;SQL query selects trip_id, from_city, to_city for a specific user_id and trips up to today from trip_table.&quot; class=&quot;rich-image&quot; height=&quot;280&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy84NTYzOWU4NS00NDFmLTQzZTEtOTY0OS0yNjljZWQ1NTFmNjkucG5n&quot; width=&quot;1434&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 4: A typical export query.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;This query is narrow in the records it’s looking for, but broad in the historical range it may need to scan. Even though the result set is small, the engine may still need to examine files across many partitions to determine where the relevant records exist.&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The result is a much larger storage access footprint than ‌output size alone would suggest. Repeated execution of this pattern increases storage cost, retrieval cost, and metadata-heavy Class B operations such as &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;objects.get&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; and &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;objects.list&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;. It can also degrade query performance by forcing repeated scans over files that’d otherwise remain cold.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;A visualization of the scan surface area in a Hudi table makes this contrast clear.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Analytics workload accesses recent parquet files; DSAR workload accesses all files across all partitions.&quot; class=&quot;rich-image&quot; height=&quot;708&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy82ZTgzNmY3MS0xZGQwLTRjMjItOTJhMi0wNjk0M2JjNmQ0NGIucG5n&quot; width=&quot;1448&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 5: Diagram contrasting typical analytics versus an export workload’s file scanning patterns.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Full-table scanning patterns further drive up compute and GCS egress costs. For massive datasets, this execution model triggers significant resource overhead and risks exceeding peak &lt;/span&gt;&lt;a href=&quot;https://docs.cloud.google.com/storage/quotas#bandwidth&quot; class=&quot;rich-link&quot; dir=&quot;auto&quot;&gt;&lt;span class=&quot;rich-underline&quot; data-lexical-text=&quot;true&quot;&gt;bandwidth limits&lt;/span&gt;&lt;/a&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;.&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;secondary-index-doesn-t-h-436&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Secondary Index Doesn’t Help&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;To achieve our optimization and efficiency goals, we focused on minimizing the number of data files touched during query execution. We evaluated the &lt;/span&gt;&lt;a href=&quot;https://hudi.apache.org/blog/2025/04/02/secondary-index/&quot; class=&quot;rich-link&quot; dir=&quot;auto&quot;&gt;&lt;span class=&quot;rich-underline&quot; data-lexical-text=&quot;true&quot;&gt;Hudi Secondary Index&lt;/span&gt;&lt;/a&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; as a potential solution. Secondary indexes in Hudi are designed to accelerate predicates on non-primary-key columns by helping query engines prune files before scanning data. Hudi supports these indexes through its metadata-table indexing framework, and secondary indexes can be created with SQL when the table has the required record-index setup.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;For our export workload, the key question wasn’t whether a secondary index could prune files, but whether it’d prune enough files to justify its additional metadata and operational cost. On our target tables, records for the same subject were spread across most files. A secondary index helps locate candidate rows, but it can still map to a large file set. The engine therefore still pays a high file-scan and metadata cost.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;what-works-437&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;What Works ?&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;To achieve the goal of reducing the number of file scans, we combine two ideas:&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;a href=&quot;https://hudi.apache.org/docs/metadata/#types-of-table-metadata&quot; class=&quot;rich-link&quot; dir=&quot;auto&quot;&gt;&lt;span class=&quot;rich-underline&quot; data-lexical-text=&quot;true&quot;&gt;Hudi column stats&lt;/span&gt;&lt;/a&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; to identify which files are relevant for a predicate without touching the underlying Parquet data files&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Table sorting on the predicate column so that matching records are clustered into a much smaller subset of files, making file pruning significantly more effective since the query remains constant and filters on a consistent predicate.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;HUDI architecture diagram showing metadata and data tables, ingestion, query engine, and sorting jobs interactions.&quot; class=&quot;rich-image&quot; height=&quot;856&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy8wZWVkMzU2OC1hYWRkLTQyOWYtOTk4YS1jNjg0ODZhMGY4NzEucG5n&quot; width=&quot;1310&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 6: Architecture diagram.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;br&gt;&lt;p&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;hudi-column-stats-438&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Hudi Column Stats&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Hudi maintains &lt;/span&gt;&lt;a href=&quot;https://hudi.apache.org/docs/metadata/#types-of-table-metadata&quot; class=&quot;rich-link&quot; dir=&quot;auto&quot;&gt;&lt;span class=&quot;rich-underline&quot; data-lexical-text=&quot;true&quot;&gt;column-level statistics&lt;/span&gt;&lt;/a&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; in its metadata table, including values such as min, max, null count, and total count for tracked columns. Because this metadata is stored separately from the underlying Parquet data files, the query engine can use it to prune irrelevant files without directly reading them.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;For export workloads that repeatedly run selective predicates over large historical datasets, this avoids touching much of the table and reduces scan surface area.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Parquet stores useful column statistics, but they live in the file footer. In our environment, relying on those footers isn’t sufficient, because reading them still touches the data files. Over time, those repeated reads can keep files hot and prevent auto-class from moving them into colder, cheaper tiers.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Hudi addresses the footer reads problem by maintaining file-pruning metadata at the table level, rather than requiring the engine to inspect each Parquet file individually. Instead of opening many files to discover their statistics, the engine can query Hudi’s metadata table and determine which files are relevant before it begins scanning data.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;This makes file pruning both faster and more storage-efficient for export workloads. It reduces the need to touch cold files, preserves the benefits of auto-class tiering, and lowers the operational cost of repeated historical lookups.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;table-sorting-439&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Table sorting&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Export workload queries are usually driven by subject identifiers such as rider ID, driver ID, or courier ID. If a subject’s records are scattered across many files, even column stats may still leave the engine inspecting much of the table.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Sorting on the predicate column clusters similar keys together, tightens file-level min/max ranges, and makes pruning more selective. Column stats provide the pruning mechanism; sorting improves its effectiveness. Together, they turn broad file scans into narrow file selection, reducing reads, shuffle, and execution time for row-level queries.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;benchmark-440&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Benchmark&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We sorted a real table partition and&amp;nbsp; took 100, 200, 500, 1,000, and 5,000 predicates (user IDs), sorted the partition, compared the number of Parquet files where the target rows are spread across.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Bar chart comparing target file percentages for unsorted and sorted data across different user counts, showing unsorted dominance.&quot; class=&quot;rich-image&quot; height=&quot;732&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy83MmRhNGY1MS03YzBhLTQzMTctOWNjMS00ZDAxYzdkYmU0ZTUucG5n&quot; width=&quot;1282&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 7: Target files percentage across different number of predicates in non-sorted versus sorted partitions.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The next benchmarking we conducted is we sorted a one partition on the predicate column as sorting key and recorded the efficiency gains across files pruned (never touched during scanning), GCS egress and compute reduction during the scan.&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Table shows disk reduction of 24.8%, with files pruned and reductions for low, mid, and high predicate values.&quot; class=&quot;rich-image&quot; height=&quot;356&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9mMzFjZTRhNi1jY2FiLTQ4YjMtODQxZi04OGRlMjNjMTU5NDkucG5n&quot; width=&quot;1084&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 8:&amp;nbsp; Benchmark results for files pruned, egress, compute reduction and storage reduction.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The solution reduces the number of partitions and files touched per export query. For export workloads the main inefficiency isn’t output size, but the scan surface the engine must inspect. Minimizing file scans lets older data stay cold for GCS auto-class and reduces repeated data access, metadata operations, GCS egress, and query overhead. Furthermore, table sorting optimizes storage by enhancing compression efficiency, as clustering identical values physically together results in a smaller total data footprint.&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;why-not-sorting-paired-wi-441&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Why Not Sorting Paired With Secondary Index?&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Sorting makes pruning much more effective. And at this point Hudi column stats already provide effective file pruning with a smaller metadata footprint than a secondary index. Column stats scale primarily with file count and tracked columns, while a secondary index must maintain lookup mappings for indexed values.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;In our workload, sorting provides the physical clustering, and column stats index provides a lightweight pruning mechanism vs a much heavier secondary index.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;comparison-matrix-442&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Comparison Matrix&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Comparison table of Sorted + Hudi Secondary Index vs. Sorted + Hudi Column stats for index size and compute overhead.&quot; class=&quot;rich-image&quot; height=&quot;1068&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy8zYjcyYTQzZi04ZDQ1LTQ0ZGQtOTE4MC03NjUyZGMxZDQyODEucG5n&quot; width=&quot;1424&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 8: Target files in a sorted table.&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;For our export workloads, that makes the tradeoff clear: if the table is unsorted, a secondary index doesn’t prune enough files to materially change scan cost; if the table is sorted, Hudi column stats provide a simpler and more storage-efficient solution. For that reason, we chose sorted tables plus Hudi column stats over a secondary-index-based design.&lt;/span&gt;&lt;p&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;applicability-in-other-in-443&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Applicability in Other Industry Use Cases&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The underlying export access pattern appears in many domains where systems need to retrieve a small set of records from very large historical datasets, often under strict compliance, operational, or investigative requirements. The common pattern is a narrow, subject-oriented lookup over a long retention window.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Examples include:&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Finance&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: Regulatory, audit, and dispute workflows often require multi-year histories for specific accounts or entities&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Healthcare and life sciences&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: Compliance reviews, patient-history access, and audit workflows often require selective lookups over long-retained records&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;E-commerce and retail&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: Fraud investigations, customer support workflows, and order-history exports frequently retrieve records for a small set of users across long time ranges&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;4&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Ad tech and measurement&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: Timeline reconstruction for attribution, personalization, or compliance often requires subject-level lookups across historical event data&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;5&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Security and identity systems&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;: Investigations, access reviews, and compliance workflows often require full login or activity histories for specific users or principals&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;next-steps-444&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Next Steps&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We’re focused on expanding these optimizations across Uber’s broader infrastructure.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Our first objective is tackling multi-column lookup scenarios. For datasets where various clients use different predicate columns, single-column sorting often fails to provide uniform benefits. To overcome the limitations of standard lexicographic sorting as column counts increase, we’re exploring &lt;/span&gt;&lt;a href=&quot;https://hudi.apache.org/blog/2021/12/29/hudi-zorder-and-hilbert-space-filling-curves/&quot; class=&quot;rich-link&quot; dir=&quot;auto&quot;&gt;&lt;span class=&quot;rich-underline&quot; data-lexical-text=&quot;true&quot;&gt;Z-order space-filling curves&lt;/span&gt;&lt;/a&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; for more effective data layout optimization.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Additionally, we plan to develop an automated framework to detect these kinds of needle-in-a-haystack access patterns. This system will weigh the costs of indexing and sorting against potential query savings, triggering data layout optimization pipelines only when the ROI is justified.&lt;/span&gt;&lt;/p&gt;&lt;h1 class=&quot;rich-heading-h1&quot; id=&quot;conclusion-308&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Conclusion&lt;/span&gt;&lt;/h1&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Here are some of the lessons we learned from this effort:&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;ul class=&quot;rich-list-ul rich-ul1&quot;&gt;&lt;li value=&quot;1&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Access pattern drives storage cost&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;. Narrow, history-spanning lookups behave very differently from typical analytics queries, and naive table layouts can create disproportionate storage and egress costs.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;2&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;File layout matters as much as tiering.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; GCS auto-class only delivers savings when files are left untouched long enough to go cold. Sorting plus Hudi metadata helps minimize unnecessary file touches.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;3&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Metadata-driven pruning is critical.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; Relying on Parquet footers works against tiering because footer reads still touch the files. Hudi column stats enable pruning without repeatedly accessing data files.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;4&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;A one-time rewrite can unlock long-term savings.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; Sorting and rewriting tables introduces upfront cost, but it reduces scan surface area and enables sustained storage savings over time.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;5&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;The best results come from an end-to-end design.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; Storage layout, metadata, query execution, and orchestration all need to work together; otherwise, cost is simply shifted from one layer to another.&lt;/span&gt;&lt;/li&gt;&lt;li value=&quot;6&quot; class=&quot;rich-list-item&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Hudi Table Service as the primary enabler.&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; The Hudi Table Service plays a pivotal role in enabling these access-pattern-driven optimizations, serving as the central engine required to manage and execute all backfill sorting, index construction, and incremental sorting operations.&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Overall, optimizing export workloads at Uber shows that cloud storage costs depend heavily on access patterns and file layout. By combining Hudi column stats with predicate-column sorting, we transformed broad historical scans into selective, low-touch lookups that sustain storage and egress savings.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;acknowledgments-445&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Acknowledgments&amp;nbsp;&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;a href=&quot;mailto:meenalb@uber.com&quot; class=&quot;rich-link&quot; dir=&quot;auto&quot;&gt;&lt;span class=&quot;rich-underline&quot; data-lexical-text=&quot;true&quot;&gt;Meenal Binwade&lt;/span&gt;&lt;/a&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; &lt;/span&gt;&lt;a href=&quot;mailto:vamshi@uber.com&quot; class=&quot;rich-link&quot; dir=&quot;auto&quot;&gt;&lt;span class=&quot;rich-underline&quot; data-lexical-text=&quot;true&quot;&gt;Vamshi Pasunuru&lt;/span&gt;&lt;/a&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; &lt;/span&gt;&lt;a href=&quot;mailto:jiashenz@uber.com&quot; class=&quot;rich-link&quot; dir=&quot;auto&quot;&gt;&lt;span class=&quot;rich-underline&quot; data-lexical-text=&quot;true&quot;&gt;Jiashen Zhang&lt;/span&gt;&lt;/a&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; Uber&#39;s Hudi team has been instrumental in productionizing the column stats at Uber.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cover Photo Attribution: Generated using Gemini&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;Apache®, Apache Hudi™, and the star logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.&lt;/em&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;Google Cloud Storage™ is a trademark of Google LLC and this blog post is not endorsed by or affiliated with Google in any way.&lt;/em&gt;&lt;/p&gt;</description>
      <link>https://www.uber.com/us/en/blog/running-cost-efficient-export/</link>
      <guid isPermaLink="false">https://www.uber.com/us/en/blog/running-cost-efficient-export/</guid>
      <pubDate>Tue, 11 Aug 2026 16:00:00 GMT</pubDate>
      <author>Pankaj Mohapatra, Arun Mahadeva Iyer, Balajee Nagasubramaniam, Prashant Wason, Alon Levy</author>
      <enclosure url="https://tb-static.uber.com/prod/udam-assets/0bb7fb40-6a03-4872-a96a-fae249fa1a7f.png" type="image/jpeg"></enclosure>
      <category>Engineering</category>
      <category>AI / ML</category>
      <category>Data</category>
    </item>
    <item>
      <title>Scaling Exact COUNT(DISTINCT) for High-Cardinality Non-Rollup Metrics in Distributed Data Pipelines</title>
      <description>&lt;h1 class=&quot;rich-heading-h1&quot; id=&quot;introduction-2&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Introduction&lt;/span&gt;&lt;/h1&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;How many unique users did Uber serve this quarter? For mission-critical metrics like these, the standard engineering answer is approximate. We needed an exact count, but no prior solution existed at our scale.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;At Uber, a class of metrics such as monthly active users, quarterly retention, and cross-window engagement must be computed exactly. These are non-rollup metrics: their values can’t be derived from coarser pre-aggregates, and no composition of daily or monthly counts yields a correct quarterly result. Every unique entity must be tracked simultaneously across the full aggregation window. At a quarterly scale, that means holding over 3.6 billion UUID-valued identifiers in memory concurrently.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;RoaringBitmap is the industry standard for this class of problems: compact, fast, and exact. But at high enough cardinality, it hits a hard constraint baked into the JVM: Java’s internal array indexing is bounded by &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;Integer.MAX_VALUE&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; (of around 2 GB). When the fully merged bitmap at the reduce stage exceeds this, the job terminates with &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;OutOfMemoryError&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; deterministically, on every run, with no configuration workaround. The community’s answer has been to switch to approximate counting. We couldn’t.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;To solve this, we built a chunked aggregation buffer strategy: a redesign that partitions the bitmap aggregation buffer across a map, keyed on the top 16 bits of each hashed identifier. Because each chunk serializes independently, we bound peak memory to the largest individual chunk and eliminated the 2 GB JVM constraint entirely. Deployed across 75 metric families at Uber, it reduced two-year backfill time by 65% on average and eliminated all out-of-memory failures post-deployment.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;If you run JVM-based data pipelines computing exact distinct counts at scale on Apache Hive™, Apache Spark&lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;™&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;, or any framework where aggregation state must be returned as a single &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;byte[]&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;, this problem is one cardinality threshold away from being yours. This blog walks through how we solved this problem and why the solution is generalizable beyond Uber’s stack.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;challenges-with-non-rollu-34&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Challenges with Non-Rollup Metrics&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Aggregate metrics in large-scale data platforms fall into two categories. Rollup metrics like completed trip counts compose cleanly across time windows: a weekly total is simply the sum of daily totals. These can be pre-aggregated at finer granularities, and combined later. Non-rollup metrics don&#39;t work this way. MAU (Monthly Active Users) is a defining example, a user active on both Monday and Wednesday must be counted once in the weekly total, not twice. No combination of daily active user counts produces the correct result. The only way to compute it correctly is to track every unique user simultaneously across the full window,&amp;nbsp; and at a quarterly scale, which means holding 3.6 billion UUID-valued identifiers in memory at once.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Comparison of rollup metric (trips summed weekly) vs. non-rollup metric (unique users tracked across days).&quot; class=&quot;rich-image&quot; height=&quot;772&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9iZjM4Y2FmNi01ODYxLTRkMWUtODJiZC03MWRkMGU5ZDA3NjcucG5n&quot; width=&quot;1950&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 1: Rollup vs. non-rollup metric aggregation.&amp;nbsp;&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;br&gt;&lt;p&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;As Figure 1 shows, rollup metrics (completed trips) compose additively across granularities and&amp;nbsp; non-rollup metrics (MAU) require full identity preservation across the aggregation window. Naive summation of daily counts overcounts users appearing on multiple days.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Compressed bitmap indexes represent the standard architectural solution: encoding set membership as individual bits and using high-performance bitwise OR for set unions to eliminate the prohibitive cost of shuffling raw identifiers across the network. The challenge is that at this specific cardinality, every conventional approach fails deterministically, each breaking against a different technical constraint.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Table showing distinct identifiers by aggregation window: daily 40M, monthly 1.2B, quarterly 3.6B, two-year 28.8B.&quot; class=&quot;rich-image&quot; height=&quot;311&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy8yY2NjYjQxZS1jMmIzLTQxMTctOTBkYy01ZDI0OTg0MWRkYmIucG5n&quot; width=&quot;559&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 2: Scale of non-rollup metric computation (Uber production figures at time of evaluation).&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;br&gt;&lt;p&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;bitmap-32-and-the-global--59&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Bitmap-32 and the Global Dictionary&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;RoaringBitmap processes 32-bit integers, requiring a mapping from UUIDs to unique integer ranks through a shared global dictionary. This architecture creates a critical single point of failure: any corruption or replication lag in the dictionary invalidates all in-flight jobs. Furthermore, concurrent jobs generate overlapping rank assignments, mandating that backfill jobs execute sequentially. At our scale, this pushed two-year historical data backfills to over 20 days.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;bitmap-64-monolithic-seri-66&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Bitmap-64, Monolithic Serialization&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Transitioning to Roaring64Bitmap removes the dictionary requirement by hashing UUIDs directly to 64-bit integers with negligible collision risk. However, the Hive and Spark UDAF contract mandates that aggregation state be returned as a single Java &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;byte[]&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;, which is capped at around 2 GB by the JVM. For high-cardinality UUID workloads, this limit is breached at approximately 179 million unique identifiers.&lt;/span&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt; &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;At a quarterly scale of 3.6 billion, this results in deterministic job failure on every run.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;approximate-counting-77&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Approximate Counting&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;HyperLogLog bypasses these constraints but introduces a 1-5% error, unacceptable for metrics like MAU that underpin financial reporting. We needed a solution that was exact, dictionary-free, and never materialized into a single serialized object. None existed.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;breaking-the-jvm-barrier-83&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Breaking the JVM Barrier&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;Sometimes the solution isn’t found by thinking outside the box, it&#39;s found by utilizing more boxes.&lt;/em&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The JVM architecture imposes a hard limit, refusing to allocate any single array exceeding 2 GB. Instead of a monolithic Roaring64Bitmap, we implemented a &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;Map&amp;lt;Integer, Roaring64Bitmap&amp;gt;&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; where each entry handles a disjoint segment of the 64-bit value space. Because these slices are non-overlapping, the exact total is derived simply by summing per-slice cardinalities. This is the chunked aggregation buffer strategy, as shown in Figure 3.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Comparison of monolithic Roaring64Bitmap exceeding 2GB with chunked buffer strategy using safe-sized chunks.&quot; class=&quot;rich-image&quot; height=&quot;870&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy8yYTJkNTU1OC0wYjc2LTRkM2QtOWZhNC1hYmU5ZDBmMjdmZWUucG5n&quot; width=&quot;1980&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 3: 16-bit chunk partitioning of 64-bit hashed values.&amp;nbsp;&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;br&gt;&lt;p&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The top 16 bits dictate the routing to a specific chunk identifier via &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;chunkId = (int)(hash &amp;gt;&amp;gt;&amp;gt; 48)&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;. The full 64-bit value is then persisted within the corresponding chunk’s Roaring64Bitmap buffer.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Each incoming UUID is transformed into a 64-bit integer via &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;xxHash64&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;, the native Hive hash function. The hash’s top 16 bits then dictate the routing to a specific chunk:&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot; style=&quot;text-align: center;&quot;&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;chunkId = (int)(hash &amp;gt;&amp;gt;&amp;gt; 48)&lt;/em&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We chose &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;xxHash64&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; for two reasons: as a Hive built-in, it requires no external dependencies. Its avalanche properties also guarantee uniform distribution across all 65,536 chunk IDs for production UUID workloads.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;why-16-bits-123&quot; dir=&quot;auto&quot;&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt;Why 16 Bits?&lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;&amp;nbsp;&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Using 16 bits partitions the 64-bit value space into 65,536 distinct chunks. At a quarterly scale of 3.6 billion identifiers, the robust distribution properties of our hashing function yield an average of 55,000 values per chunk. Statistical analysis via the Chernoff bound confirms that the probability of any single chunk reaching the 179 million-value OOM threshold is effectively zero.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Shorter bit depths offer insufficient headroom: 8 bits yields 256 chunks averaging around 14 million values each, with no guarantee against worst-case skew. Longer bit depths (24 bits) create over 16 million possible chunks, inflating map overhead and GC pressure disproportionately.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;unlocking-concurrent-exec-135&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Unlocking Concurrent Execution&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The Bitmap-32 global dictionary forced backfill jobs to run sequentially to prevent rank assignment collisions. By hashing directly to 64-bit integers, we eliminate all shared state between jobs; multiple backfill windows now execute concurrently by default.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;self-describing-partial-r-142&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Self-Describing Partial Results&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We prefix every chunked partial result with a 4-byte magic number: 0xDEADBEEF. Upon reading a partial, the evaluator inspects the initial 4 bytes; a match routes to the chunked merge path, a mismatch to the legacy deserialization path. The choice of 0xDEADBEEF is deliberate: Roaring64Bitmap&#39;s internal cookie header can never collide with this value, making false positives structurally impossible.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;streaming-serialization-149&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Streaming Serialization&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;During partial aggregation, each chunk serializes independently into its own buffer and writes sequentially to the output stream. By bounding the size of the partial result to the largest individual chunk rather than the union of all chunks, we effectively bypass the 2 GB JVM constraint at all intermediate stages.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;cardinality-only-final-ou-156&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Cardinality-Only Final Output&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;In the final reduce stage, the &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;terminate() &lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;function avoids bitmap serialization entirely. It simply aggregates the cardinalities across populated chunks and returns an 8-byte long. This ensures the final output remains a constant 8 bytes regardless of the total distinct count, eliminating OOM risks at the &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;reduce&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; stage.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;In aggregate, these mechanisms expand our safe cardinality threshold from around 179 million to approximately 11.7 trillion, representing a 65,000× scalability improvement.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot;&gt;&lt;/p&gt;&lt;div data-lexical-decorator=&quot;true&quot;&gt;&lt;div class=&quot;rich-float-clear&quot;&gt;&lt;/div&gt;&lt;div class=&quot;rich-image-wrapper rich-image-center&quot;&gt;&lt;img alt=&quot;Bar chart comparing safe cardinality thresholds: Monolithic (~179M) vs. Chunked (~11.7T), showing 65,000x increase.&quot; class=&quot;rich-image&quot; height=&quot;1024&quot; loading=&quot;lazy&quot; src=&quot;https://cn-geo1.uber.com/image-proc/resize/udam/format=auto/srcb64=aHR0cHM6Ly90Yi1zdGF0aWMudWJlci5jb20vcHJvZC91ZGFtLWFzc2V0cy9hZWU3MzZmZC1kODU4LTQzODYtYjFhMi0zZGI2ZGMzNWFiZTYuanBlZw==&quot; width=&quot;995&quot; referrerpolicy=&quot;no-referrer&quot;&gt;&lt;p data-baseweb=&quot;typo-paragraphsmall&quot; class=&quot;rich-image-caption css-hWVzCq css-lnLvkz&quot;&gt;Figure 4: Effective safe cardinality threshold before and after the chunked strategy; dotted lines show actual production workload cardinalities.&amp;nbsp;&lt;/p&gt;&lt;/div&gt;&lt;/div&gt;&lt;br&gt;&lt;p&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;zero-disruption-migration-176&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Zero-Disruption Migration at Scale&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;We took proactive measures to implement this solution in production without any disruption.&amp;nbsp;&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;seamless-integration-with-182&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Seamless Integration with the Hive UDAF Lifecycle&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The Chunked Bitmap UDAF maps cleanly onto Hive&#39;s four lifecycle phases: &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;PARTIAL1 &lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;(per-mapper aggregation), &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;PARTIAL2 &lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;(combiner merge), &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;FINAL&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; (reducer), and &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;terminate()&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;. During the map and combine phases, each node serializes only its local partition, a fraction of the full value space. At the &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;FINAL&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; stage, &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;terminate() &lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;bypasses bitmap serialization entirely, returning a constant 8-byte long.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;streaming-serialization-w-201&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Streaming Serialization Without Object Materialization&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Each chunk serializes independently into its own &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;ByteArrayOutputStream&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; and writes sequentially to the output stream. During deserialization,&lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt; mergeFromStream() &lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;reads one chunk at a time from a &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;DataInputStream&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;, allocates a &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;byte[] &lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;exactly matching the chunk size, merges it, and releases it before processing the next, bounding peak memory to the largest individual chunk, not the total payload.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;seamless-migration-across-216&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Seamless Migration Across Dual Paths&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Our transition addressed two legacy architectures. Bitmap-32 pipelines, tethered to a global dictionary that forced sequential execution and introduced corruption risks, underwent a fresh migration retiring the dictionary enabled concurrent backfill execution with no cross-job coordination required.&lt;/span&gt;&lt;/p&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Conversely, some pipelines were already using monolithic Bitmap-64, as their cardinalities remained below the 179 million OOM threshold; valid &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;Roaring64Bitmap&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; partial results were already persisted in HDFS. For these workloads, the self-describing magic-number router facilitated seamless interoperability. Legacy partials are automatically detected and routed through the original deserialization path to be merged into the chunked buffer, while native chunked partials are processed via the optimized merge path. This structural guarantee ensured that no pipeline, regardless of its legacy category, necessitated a historical recompute window.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;rigorous-correctness-vali-229&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Rigorous Correctness Validation&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Prior to full production rollout, distinct counts generated by the chunked UDAF were cross-referenced against raw &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;COUNT(DISTINCT identifier)&lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt; results across 10 sampled metric families. These benchmarks covered daily, monthly, and quarterly pipelines to ensure accuracy at varying scales. Across all 30 validation runs, results matched exactly with zero deviation. We also stress-tested the router against mixed partial types within a single job, confirming deterministic routing.&lt;/span&gt;&lt;/p&gt;&lt;h3 class=&quot;rich-heading-h3&quot; id=&quot;use-cases-at-uber-238&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Use Cases at Uber&lt;/span&gt;&lt;/h3&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;The chunked aggregation buffer strategy&lt;/span&gt;&lt;strong class=&quot;rich-bold&quot; data-lexical-text=&quot;true&quot;&gt; &lt;/strong&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;is now the platform-standard implementation for exact &lt;/span&gt;&lt;em class=&quot;rich-italic&quot; data-lexical-text=&quot;true&quot;&gt;COUNT(DISTINCT) &lt;/em&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;across Uber&#39;s Mobility, Delivery, and Platform verticals. Since deployment across 75 metric families, zero OOM failures have been observed in production.&lt;/span&gt;&lt;/p&gt;&lt;h4 class=&quot;rich-heading-h4&quot; id=&quot;backfill-duration-249&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Backfill Duration&lt;/span&gt;&lt;/h4&gt;&lt;p class=&quot;rich-paragraph&quot; dir=&quot;auto&quot;&gt;&lt;span data-lexical-text=&quot;true&quot;&gt;Across historical data reprocess

...

fix(route): update Uber Engineering blog

02b265f

github-actions Bot added route auto: ready to review Human review will come in after lint issues and merge conflicts are fixed labels Sep 1, 2026

Navigation Menu

fix(route): update Uber Engineering blog - #23174

fix(route): update Uber Engineering blog#23174
zhsama wants to merge 1 commit into
DIYgod:masterfrom
zhsama:fix/uber-engineering-blog

zhsama commented Sep 1, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

zhsama commented Sep 1, 2026

Involved Issue / 该 PR 相关 Issue

Example for the Proposed Route(s) / 路由地址示例

New RSS Route Checklist / 新 RSS 路由检查表

Note / 说明

Uh oh!

chatgpt-codex-connector Bot commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

chatgpt-codex-connector Bot commented Sep 1, 2026 •

edited

Loading