Unify token estimation interface in optimiser

README.md

@@ -4,7 +4,7 @@ RTK token-optimisation proxy adapter for [PapiAI](https://papi-ai.org).
 
 Wraps the [RTK](https://github.com/rtk-ai/rtk) CLI (a proxy that compresses verbose developer
 output before it reaches an LLM) behind PapiAI's `LLMTokenOptimisationProxyInterface`, so agents
-and tools can shrink tool/command output — often by 60–90% — before it enters the context window.
+and tools can shrink command output, often by 60-90%, before it enters the context window.
 
 ## Install
 
@@ -30,6 +30,10 @@ echo $result->savingsPercent();      // e.g. 62.5
 // Or run a read-only command through RTK's specialised filter and measure the saving
 $result = $rtk->optimiseCommand('git status');
 echo $result->tokensBefore, ' -> ', $result->tokensAfter;
+
+// Measuring runs the command twice (raw, then via RTK). Skip the baseline when you only
+// want the output: one execution, `tokensBefore` and `savingsPercent()` are then null.
+$result = $rtk->optimiseCommand('git status', ['measure' => false]);
 ```
 
 Use it inside a tool so a command's output is compressed before it re-enters the agent's context:
@@ -45,21 +49,52 @@ $tool = Tool::make(
 );
 ```
 
+## What to send through it (and what never to)
+
+Optimisation is lossy by design. RTK rewrites the text, and its named filters drop detail they
+consider incidental (the `find` filter, for example, discards file names).
+
+**Only compress text the model needs to read.** Command output, logs, test runs, diffs: that is
+where the savings are, and losing formatting costs nothing.
+
+**Never compress text the model must reproduce verbatim.** Source files it is about to edit,
+whole-file payloads, anything round-tripped back to disk. Measured on a 15.7KB PHP file: an
+unfiltered `rtk pipe` returned it byte-identical (safe, but zero saving), while the named filters
+that do save tokens mangled it. There is nothing to win and a file to lose.
+
+Text that is already summarised at the source has nothing left to squeeze either. If your tool
+already returns a structured report rather than raw runner output, you solved the same problem
+upstream.
+
 ## API
 
-`RtkProxy implements LLMTokenOptimisationProxyInterface`:
+`RtkProxy implements LLMTokenOptimisationProxyInterface` (which extends `TokenEstimatorInterface`):
 
 | Method | Purpose |
 |---|---|
 | `optimise(string $content, array $options = [])` | Pipe text through `rtk pipe` (`filter`, `ultraCompact` options) |
-| `optimiseCommand(string $command, array $options = [])` | Run a read-only command through RTK and measure the saving |
-| `estimateTokens(string $content)` | Cheap ~4-bytes-per-token estimate |
+| `optimiseCommand(string $command, array $options = [])` | Run a read-only command through RTK (`measure`, `ultraCompact` options) |
+| `estimateTokens(string $content)` | Cheap byte-based estimate, delegated to an injectable estimator |
+
+Both optimise methods return an `OptimisationResult` (`optimised`, `tokensBefore`, `tokensAfter`,
+`isMeasured()`, `tokensSaved()`, `savingsPercent()`, `strategy`). Token counts are estimates, not a
+real tokenizer: by default `PapiAI\Core\HeuristicTokenEstimator` (bytes divided by 4, so multibyte
+text estimates high). Inject any `TokenEstimatorInterface` to change that:
+
+```php
+$rtk = new RtkProxy('rtk', new MyTokenizerBackedEstimator());
+```
+
+> `optimiseCommand()` executes the command (raw, then via RTK) to measure the saving, so pass only
+> side-effect-free commands (git status, grep, ls, test runners). `['measure' => false]` runs it
+> once, through RTK only.
 
-All return an `OptimisationResult` (`optimised`, `tokensBefore`, `tokensAfter`, `tokensSaved()`,
-`savingsPercent()`, `strategy`). Token counts are estimates, not a real tokenizer.
+## Where it fits
 
-> `optimiseCommand()` executes the command (raw, then via RTK) to measure the saving — only pass
-> side-effect-free commands (git status, grep, ls, test runners).
+Nothing in PapiAI wires an optimiser in for you, by design. Agent middleware only sees the request
+prompt (tool results are resolved inside the run loop), so a middleware could never reach the text
+RTK is good at. The integration point is your own tool handler, as in the example above: run the
+command, compress its output, return the compressed text.
 
 ## License
 

composer.json

@@ -11,7 +11,7 @@
     ],
     "require": {
         "php": "^8.2",
-        "papi-ai/papi-core": "^0.11"
+        "papi-ai/papi-core": ">=0.14 <1.0"
     },
     "require-dev": {
         "pestphp/pest": "^3.0",

src/RtkProxy.php

@@ -15,6 +15,8 @@
 namespace PapiAI\Rtk;
 
 use PapiAI\Core\Contracts\LLMTokenOptimisationProxyInterface;
+use PapiAI\Core\Contracts\TokenEstimatorInterface;
+use PapiAI\Core\HeuristicTokenEstimator;
 use PapiAI\Core\OptimisationResult;
 use RuntimeException;
 
@@ -29,15 +31,23 @@
  *   - optimiseCommand(): run a read-only command through RTK's specialised filter and report
  *                        the saving against its raw output.
  *
- * Token counts are cheap byte-based estimates (~4 bytes per token), not a real tokenizer.
+ * Both are lossy: RTK rewrites the text and its named filters discard detail (the `find` filter,
+ * for instance, drops file names). Compress output the model only needs to read. Never pass
+ * content the model must reproduce verbatim, such as a source file it is about to edit; RTK
+ * either returns it unchanged, for no saving, or mangles it.
+ *
+ * Token counts come from an injectable {@see TokenEstimatorInterface}, by default the byte-based
+ * {@see HeuristicTokenEstimator}, not a real tokenizer.
  */
 class RtkProxy implements LLMTokenOptimisationProxyInterface
 {
     /**
-     * @param string $binary Path to the rtk executable (defaults to "rtk" on PATH)
+     * @param string                 $binary    Path to the rtk executable (defaults to "rtk" on PATH)
+     * @param TokenEstimatorInterface $estimator Sizes the before/after payloads
      */
     public function __construct(
         private readonly string $binary = 'rtk',
+        private readonly TokenEstimatorInterface $estimator = new HeuristicTokenEstimator(),
     ) {
     }
 
@@ -66,15 +76,24 @@ public function optimise(string $content, array $options = []): OptimisationResu
         );
     }
 
-    /** {@inheritDoc} */
+    /**
+     * {@inheritDoc}
+     *
+     * Measuring the saving costs a second execution of the command, unfiltered. Pass
+     * `measure: false` to run it once, through RTK only: `tokensBefore` is then null and no
+     * saving is reported.
+     */
     public function optimiseCommand(string $command, array $options = []): OptimisationResult
     {
-        $raw = $this->execute(['sh', '-c', $command]);
-        $optimised = $this->execute(['sh', '-c', $this->binary . ' ' . $command]);
+        $tokensBefore = ($options['measure'] ?? true)
+            ? $this->estimateTokens($this->execute(['sh', '-c', $command]))
+            : null;
+
+        $optimised = $this->execute(['sh', '-c', $this->binary . ' ' . $this->withUltraCompact($command, $options)]);
 
         return new OptimisationResult(
             $optimised,
-            $this->estimateTokens($raw),
+            $tokensBefore,
             $this->estimateTokens($optimised),
             'rtk:command',
         );
@@ -83,7 +102,7 @@ public function optimiseCommand(string $command, array $options = []): Optimisat
     /** {@inheritDoc} */
     public function estimateTokens(string $content): int
     {
-        return (int) ceil(strlen($content) / 4);
+        return $this->estimator->estimateTokens($content);
     }
 
     /**
@@ -126,4 +145,24 @@ protected function execute(array $argv, ?string $stdin = null): string
 
         return $stdout === false ? '' : $stdout;
     }
+
+    /**
+     * Insert RTK's ultra-compact flag where RTK expects it: after the filter name, before the
+     * command's own arguments (`rtk git --ultra-compact status`).
+     *
+     * @param string              $command The command line being proxied
+     * @param array<string, mixed> $options The caller's options
+     *
+     * @return string The command line, with the flag inserted when requested
+     */
+    private function withUltraCompact(string $command, array $options): string
+    {
+        if (empty($options['ultraCompact'])) {
+            return $command;
+        }
+
+        $parts = explode(' ', $command, 2);
+
+        return rtrim($parts[0] . ' --ultra-compact ' . ($parts[1] ?? ''));
+    }
 }

tests/Unit/RtkProxyTest.php

@@ -13,6 +13,8 @@
 declare(strict_types=1);
 
 use PapiAI\Core\Contracts\LLMTokenOptimisationProxyInterface;
+use PapiAI\Core\Contracts\TokenEstimatorInterface;
+use PapiAI\Core\HeuristicTokenEstimator;
 use PapiAI\Core\OptimisationResult;
 use PapiAI\Rtk\RtkProxy;
 
@@ -52,6 +54,10 @@ protected function execute(array $argv, ?string $stdin = null): string
         expect($this->proxy)->toBeInstanceOf(LLMTokenOptimisationProxyInterface::class);
     });
 
+    it('is usable as a plain token estimator', function () {
+        expect($this->proxy)->toBeInstanceOf(TokenEstimatorInterface::class);
+    });
+
     describe('estimateTokens', function () {
         it('estimates ~4 bytes per token', function () {
             expect($this->proxy->estimateTokens('abcd'))->toBe(1);
@@ -137,6 +143,7 @@ public function run(array $argv, ?string $stdin = null): string
 
             expect($result->optimised)->toBe('SHORT');
             expect($result->strategy)->toBe('rtk:command');
+            expect($result->isMeasured())->toBeTrue();
             expect($result->tokensBefore)->toBeGreaterThan($result->tokensAfter);
             expect($result->savingsPercent())->toBeGreaterThan(0.0);
 
@@ -145,5 +152,42 @@ public function run(array $argv, ?string $stdin = null): string
             expect($this->proxy->calls[0]['argv'])->toBe(['sh', '-c', 'git status']);
             expect($this->proxy->calls[1]['argv'])->toBe(['sh', '-c', 'rtk git status']);
         });
+
+        it('skips the baseline execution when measure is false', function () {
+            $result = $this->proxy->optimiseCommand('git status', ['measure' => false]);
+
+            expect($result->optimised)->toBe('SHORT');
+            expect($result->isMeasured())->toBeFalse();
+            expect($result->tokensBefore)->toBeNull();
+            expect($result->savingsPercent())->toBeNull();
+
+            // one execution: the rtk-filtered run only
+            expect($this->proxy->calls)->toHaveCount(1);
+            expect($this->proxy->calls[0]['argv'])->toBe(['sh', '-c', 'rtk git status']);
+        });
+
+        it('inserts the ultra-compact flag after the filter name', function () {
+            $this->proxy->optimiseCommand('git status --short', ['measure' => false, 'ultraCompact' => true]);
+
+            expect($this->proxy->calls[0]['argv'])->toBe(['sh', '-c', 'rtk git --ultra-compact status --short']);
+        });
+
+        it('appends the ultra-compact flag to a bare filter name', function () {
+            $this->proxy->optimiseCommand('ls', ['measure' => false, 'ultraCompact' => true]);
+
+            expect($this->proxy->calls[0]['argv'])->toBe(['sh', '-c', 'rtk ls --ultra-compact']);
+        });
+    });
+
+    describe('token estimation', function () {
+        it('defaults to the core byte-based heuristic', function () {
+            expect($this->proxy->estimateTokens('abcd'))->toBe(1);
+        });
+
+        it('accepts an injected estimator', function () {
+            $proxy = new TestableRtkProxy('rtk', new HeuristicTokenEstimator(2));
+
+            expect($proxy->estimateTokens('abcd'))->toBe(2);
+        });
     });
 });