support remote datasets

README.md

@@ -54,13 +54,64 @@ dotnet add package Braintrust.Sdk.AgentFramework
 </ItemGroup>
 ```
 
+## Datasets
+
+Evals can read their cases straight from a Braintrust dataset instead of being written out by
+hand:
+
+```csharp
+using Braintrust.Sdk.Eval;
+
+var braintrust = Braintrust.Get();
+
+// Reads from the configured default project. Resolves the dataset id now; rows are fetched a
+// page at a time as the eval reads them.
+var dataset = await braintrust.FetchDatasetAsync<string, string>("my-dataset");
+
+using var eval = await braintrust.EvalBuilder<string, string>()
+    .Name("my-eval")
+    .Dataset(dataset)
+    .TaskFunction(input => Classify(input))
+    .Scorers(new FunctionScorer<string, string>("exact", (expected, actual) => expected == actual ? 1.0 : 0.0))
+    .BuildAsync();
+
+await eval.RunAsync();
+```
+
+Disposing the eval closes the HTTP client it opened for itself. An API client you pass to the
+builder yourself is left alone - it stays yours to dispose.
+
+Each case's `input` and `expected` are deserialized into the type arguments you pick, so
+`FetchDatasetAsync<Question, Answer>("my-dataset")` gives you typed cases.
+
+`expected` is optional in a dataset, and the type arguments are not nullable, so rows the default
+deserializer cannot handle need an `inputConverter`/`expectedConverter` to say what a missing or
+oddly-shaped field means:
+
+```csharp
+var dataset = await braintrust.FetchDatasetAsync<string, string>(
+    "my-dataset",
+    expectedConverter: e => e.ValueKind == JsonValueKind.Null ? "" : e.GetString()!);
+```
+
+Passing `version` pins the read to a transaction id. Leaving it null - the default - resolves the
+latest version when enumeration starts and reads every page as of that one version, so a dataset
+written to mid-run still produces a consistent eval. Either way the experiment records which
+dataset and version it ran against, and each eval row links back to the dataset record it came
+from.
+
+For a dataset outside the configured project, or one you already have the id of, use
+`Dataset.FetchFromBraintrustAsync<...>(apiClient, projectName, datasetName)` and
+`Dataset.FromId<...>(apiClient, datasetId)`. These factories require a caller-owned API client and
+never dispose it.
+
 ## Low-level API client
 
 Beyond evals and tracing, the SDK ships a client for the full [Braintrust REST API](https://api.braintrust.dev),
 generated from Braintrust's public OpenAPI spec:
 
 ```csharp
-using var client = DefaultBraintrustApiClient.Of(BraintrustConfig.FromEnvironment());
+using var client = BraintrustOpenApiClient.Of(BraintrustConfig.FromEnvironment());
 var api = client.Api;   // every Braintrust REST endpoint
 ```
 

docs/api-client.md

@@ -21,15 +21,15 @@ because `Braintrust.Sdk` references it with `PrivateAssets="all"`. See
 
 ## Basic usage
 
-`DefaultBraintrustApiClient` is the SDK's own client; its `Api` property is the generated
+`BraintrustOpenApiClient` is the SDK's own client; its `Api` property is the generated
 client, already wired up with the base URL, bearer auth and timeout from your config.
 
 ```csharp
 using Braintrust.Sdk.Api;
 using Braintrust.Sdk.Config;
 using Generated = Braintrust.Sdk.Api.Generated;
 
-using var client = DefaultBraintrustApiClient.Of(BraintrustConfig.FromEnvironment());
+using var client = BraintrustOpenApiClient.Of(BraintrustConfig.FromEnvironment());
 Generated.IBraintrustGeneratedApiClient api = client.Api;
 
 // Create a project (POST /v1/project upserts by name).
@@ -42,7 +42,7 @@ Generated.Project project = await api.PostProjectAsync(new Generated.CreateProje
 Console.WriteLine($"{project.Id} {project.Name}");
 ```
 
-The generated client borrows the `HttpClient` owned by `DefaultBraintrustApiClient`, so keep
+The generated client borrows the `HttpClient` owned by `BraintrustOpenApiClient`, so keep
 that instance alive for as long as you use `Api`.
 
 ### Runnable example
@@ -165,9 +165,9 @@ serialize that field yourself if you need to.
 
 ### Errors
 
-Generated calls throw `Generated.ApiException` (or `Generated.ApiException<T>`), not the
-SDK's `Braintrust.Sdk.Api.ApiException` - only the SDK's own wrapper methods translate. The
-server's message is in `ex.Response`:
+Calls handled by the generated OpenAPI client throw `Generated.ApiException` (or
+`Generated.ApiException<T>`), including calls made by the SDK's wrapper methods. The server's
+message is in `ex.Response`:
 
 ```csharp
 try
@@ -191,14 +191,13 @@ The spec is generated from the API's own types, but a few declared filters are r
 runtime - `project_automation_name` on `GET /v1/project_automation`, for instance, comes back
 `400 Extraneous key`. Filter client-side when that happens.
 
-`/api/apikey/login` is not in the spec at all. `DefaultBraintrustApiClient` issues it by hand
-and exposes the result through `GetProjectAndOrgInfo`.
+`POST /btql` is absent from the spec, so `BraintrustOpenApiClient` implements that operation
+itself. Project and organization lookup use the generated endpoints.
 
 ## Compatibility
 
 The generated surface tracks whatever spec ref the build pinned, so it is **not** covered by
-the SDK's own compatibility promises: bumping the ref can rename a class or change a
-signature. Prefer `IBraintrustApiClient`'s methods where they already cover what you need.
+the SDK's own compatibility promises: bumping the ref can rename a class or change a signature.
 
 ## Bumping the spec
 

examples/AgentFrameworkInstrumentation/Program.cs

@@ -62,7 +62,8 @@ static async Task Main(string[] args)
                 Console.WriteLine($"Messages: {response.Messages.Count}");
 
                 // Print Braintrust link
-                var url = await braintrust.GetProjectUriAsync()
+                var projectUri = await braintrust.GetProjectUriAsync();
+                var url = projectUri.AbsoluteUri
                     + $"/logs?r={rootActivity.TraceId}&s={rootActivity.SpanId}";
                 Console.WriteLine($"\n  View your trace in Braintrust: {url}\n");
             }

examples/AnthropicInstrumentation/Program.cs

@@ -32,7 +32,8 @@ static async Task Main()
             {
                 await MessageCompletionExample(instrumentedClient);
                 await MessageStreamingExample(instrumentedClient);
-                var url = await braintrust.GetProjectUriAsync()
+                var projectUri = await braintrust.GetProjectUriAsync();
+                var url = projectUri.AbsoluteUri
                     + $"/logs?r={rootActivity.TraceId}&s={rootActivity.SpanId}";
                 Console.WriteLine($"\n\n  Example complete! View your data in Braintrust: {url}\n");
             }

examples/ApiClientExample/Program.cs

@@ -32,10 +32,10 @@ class Program
 
     static async Task Main(string[] args)
     {
-        // DefaultBraintrustApiClient is the SDK's own client; its Api property is the
+        // BraintrustOpenApiClient is the SDK's own client; its Api property is the
         // generated client, already wired up with the base URL, bearer auth and timeout
         // from the config.
-        using var client = DefaultBraintrustApiClient.Of(BraintrustConfig.FromEnvironment());
+        using var client = BraintrustOpenApiClient.Of(BraintrustConfig.FromEnvironment());
         Generated.IBraintrustGeneratedApiClient api = client.Api;
 
         // Pick the project to read from, and resolve its org for the banner.

examples/AzureOpenAIInstrumentation/Program.cs

@@ -56,7 +56,8 @@ static async Task Main(string[] args)
             if (rootActivity != null)
             {
                 await ChatCompletionsExample(instrumentedClient, deploymentName);
-                var url = await braintrust.GetProjectUriAsync()
+                var projectUri = await braintrust.GetProjectUriAsync();
+                var url = projectUri.AbsoluteUri
                     + $"/logs?r={rootActivity.TraceId}&s={rootActivity.SpanId}";
                 Console.WriteLine($"\n\n  Example complete! View your data in Braintrust: {url}\n");
             }

examples/ClassifiersExample/Program.cs

@@ -137,7 +137,7 @@ static async Task Main()
             .Select(m => DatasetCase.Of(m.Input, m.Expected))
             .ToArray();
 
-        var eval = await braintrust
+        using var eval = await braintrust
             .EvalBuilder<string, string>()
             .Name($"dotnet-classifiers-example-{DateTimeOffset.UtcNow.ToUnixTimeMilliseconds()}")
             .Tags("classifiers-example", "dotnet-sdk")

examples/EvalExample/Program.cs

@@ -65,7 +65,7 @@ async Task<string> GetFoodType(string food)
         }
 
         // Create and run the evaluation
-        var eval = await braintrust
+        using var eval = await braintrust
             .EvalBuilder<string, string>()
             .Name($"dotnet-eval-x-{DateTimeOffset.UtcNow.ToUnixTimeMilliseconds()}")
             // Experiment-level tags and metadata (shown in the Braintrust UI for the experiment)
@@ -75,6 +75,8 @@ async Task<string> GetFoodType(string food)
                 { "model", "gpt-4o-mini" },
                 { "description", "Classifies food items as fruit or vegetable" }
             })
+            // instead of Cases, you can use a dataset from Braintrust
+            // .Dataset(await braintrust.FetchDatasetAsync<string, string>("food"))
             .Cases(
                 DatasetCase.Of("strawberry", "fruit"),
                 DatasetCase.Of("asparagus", "vegetable"),

examples/OpenAIInstrumentation/Program.cs

@@ -31,7 +31,8 @@ static async Task Main(string[] args)
             if (rootActivity != null)
             {
                 await ChatCompletionsExample(instrumentedClient);
-                var url = await braintrust.GetProjectUriAsync()
+                var projectUri = await braintrust.GetProjectUriAsync();
+                var url = projectUri.AbsoluteUri
                     + $"/logs?r={rootActivity.TraceId}&s={rootActivity.SpanId}";
                 Console.WriteLine($"\n\n  Example complete! View your data in Braintrust: {url}\n");
             }

examples/SimpleOpenTelemetry/Program.cs

@@ -13,7 +13,8 @@ static async Task Main(string[] args)
             ArgumentNullException.ThrowIfNull(activity);
             Console.WriteLine("Performing simple operation...");
             activity.SetTag("some boolean attribute", true);
-            url = await braintrust.GetProjectUriAsync() + $"/logs?r={activity.TraceId}&s={activity.SpanId}";
+            var projectUri = await braintrust.GetProjectUriAsync();
+            url = projectUri.AbsoluteUri + $"/logs?r={activity.TraceId}&s={activity.SpanId}";
         }
         Console.WriteLine($"\n\n  Example complete! View your data in Braintrust: {url}");
     }

examples/TraceScoring/Program.cs

@@ -65,7 +65,7 @@ Dictionary<string, int> CountFruits(string fruitList)
         // Scorer: uses the trace to verify each LLM call returned a numeric string
         var traceScorer = new FruitTraceScorer();
 
-        var eval = await braintrust
+        using var eval = await braintrust
             .EvalBuilder<string, Dictionary<string, int>>()
             .Name($"trace-scoring-{DateTimeOffset.UtcNow.ToUnixTimeMilliseconds()}")
             .Tags("trace-scoring", "dotnet-sdk", "multi-call")

src/Braintrust.Sdk.Anthropic/InstrumentedMessageService.cs

@@ -5,7 +5,6 @@
 using Anthropic.Models.Messages;
 using Anthropic.Services;
 using Anthropic.Services.Messages;
-using OpenTelemetry.Trace;
 
 namespace Braintrust.Sdk.Anthropic;
 
@@ -62,7 +61,7 @@ public async Task<Message> Create(
             if (activity != null)
             {
                 activity.SetStatus(ActivityStatusCode.Error, ex.Message);
-                activity.RecordException(ex);
+                activity.AddException(ex);
             }
             throw;
         }
@@ -115,7 +114,7 @@ public async IAsyncEnumerable<RawMessageStreamEvent> CreateStreaming(
                 if (activity != null)
                 {
                     activity.SetStatus(ActivityStatusCode.Error, ex.Message);
-                    activity.RecordException(ex);
+                    activity.AddException(ex);
                 }
                 throw;
             }

src/Braintrust.Sdk.Api.Generated/SerializerSettings.cs

@@ -14,8 +14,8 @@ static partial void UpdateJsonSerializerSettings(JsonSerializerOptions settings)
     {
         // The generator writes every unset optional member as an explicit null, and the API
         // validates nullability strictly: creating an automation whose action never mentions
-        // formatting_prompt was rejected with "Expected string, received null". Omitting
-        // nulls is also what the SDK's hand-rolled client did.
+        // formatting_prompt was rejected with "Expected string, received null", so unset
+        // members are omitted instead.
         //
         // The trade-off is that an explicit null can no longer be sent to clear a field on
         // PATCH. Braintrust's PATCH endpoints treat an absent field as "leave alone" and

src/Braintrust.Sdk.OpenAI/InstrumentedChatClient.cs

@@ -3,7 +3,6 @@
 using System.Text.Json;
 using System.Text.Json.Nodes;
 using OpenAI.Chat;
-using OpenTelemetry.Trace;
 
 namespace Braintrust.Sdk.OpenAI;
 
@@ -71,7 +70,7 @@ public override ClientResult<ChatCompletion> CompleteChat(IEnumerable<ChatMessag
             if (activity != null)
             {
                 activity.SetStatus(ActivityStatusCode.Error, ex.Message);
-                activity.RecordException(ex);
+                activity.AddException(ex);
             }
             // intentionally re-throwing original exception
             throw;
@@ -114,7 +113,7 @@ public override async Task<ClientResult<ChatCompletion>> CompleteChatAsync(IEnum
             if (activity != null)
             {
                 activity.SetStatus(ActivityStatusCode.Error, ex.Message);
-                activity.RecordException(ex);
+                activity.AddException(ex);
             }
             // intentionally re-throwing original exception
             throw;