Skip to content

Add DetectFromBytes(ReadOnlySpan<byte>) overload with comprehensive tests#211

Draft
304NotModified with Copilot wants to merge 7 commits into
masterfrom
copilot/update-unit-tests-for-overload
Draft

Add DetectFromBytes(ReadOnlySpan<byte>) overload with comprehensive tests#211
304NotModified with Copilot wants to merge 7 commits into
masterfrom
copilot/update-unit-tests-for-overload

Conversation

Copilot AI commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Applies PR #204 changes to add a ReadOnlySpan<byte> overload for DetectFromBytes, allowing callers to detect encoding from a span or slice without copying to a byte[]. Also addresses review feedback and bugs found during integration.

Source changes

  • New overload: CharsetDetector.DetectFromBytes(ReadOnlySpan<byte>) — existing byte[] overloads now delegate to it
  • Internal signature updates: Feed, HandleData, FindInputState, IsStartsWithBom, FindCharSetByBom, RunProber, all probers, and analyzers updated to accept ReadOnlySpan<byte> instead of byte[] + offset/len
  • WriteSpanToStream compile guard: changed from NETSTANDARD2_1_OR_GREATER || NETCOREAPP2_1_OR_GREATERNET8_0_OR_GREATER per review comment (targets are netstandard2.0 and net8.0)
  • JapaneseContextAnalyser.HandleData: removed stale offset/len parameters, now span-only for consistency
  • Empty span guard: added early return in Feed() when buf.Length == 0; several probers (e.g. EUCKRProber, SJISProber) unconditionally access buf[max - 1] and would throw IndexOutOfRangeException on empty input

Tests

Added 16 new tests covering the ReadOnlySpan<byte> overload:

// Empty span — should return no detection, not throw
CharsetDetector.DetectFromBytes(ReadOnlySpan<byte>.Empty); // Detected == null

// Key use case: detect from a slice without allocating a copy
ReadOnlySpan<byte> slice = largeBuffer.AsSpan(offset, len);
var result = CharsetDetector.DetectFromBytes(slice);

Covers: empty span, ASCII, UTF-8 (with and without BOM), sliced span, UTF-7/GB18030/UTF-16 BE+LE/UTF-32 BE+LE BOM detection.

Comment on lines +922 to +925
if (buf[1] >= 0xA1)
return 157 * (buf[0] - 0xA4) + buf[1] - 0xA1 + 63;
else
return 157 * (buf[offset] - 0xA4) + buf[offset+1] - 0x40;
return 157 * (buf[0] - 0xA4) + buf[1] - 0x40;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants