bytearray.take_bytes() bytes object may have incorrect hash

[cmaloney]

Bug report

Bug description:

bytearray in 3.15+ tries to avoid runtime copying by "adopting" bytes when possible. One case where this happens is constructing from a temporary produced by encoding text: bytearray("text", encoding="utf-8"). If the encoder returns a uniquely-referenced exact bytes object the bytearray "adopts" it as storage avoiding an allocation and copy.

cpython/Objects/bytearrayobject.c

Lines 991 to 998 in 2168416

	if (_PyObject_IsUniquelyReferenced(encoded)
	&& PyBytes_CheckExact(encoded))
	{
	Py_ssize_t size = Py_SIZE(encoded);
	self->ob_bytes_object = encoded;
	bytearray_reinit_from_bytes(self, size, size);
	return 0;
	}

The contents can then be mutated directly via the bytearray API. Those mutations do not change the bytes object hash. That means if a uniquely referenced bytes with a hash set is "adopted" then retrieved via .take_bytes() the bad hash will persist causing issues with dict, set, etc.

For 3.15 the only case with this "adoption" is text encoding in bytearray construction. All other codepaths get new unhashed bytes. Every built-in codec constructs its output fresh and directly returns it not setting ob_shash. Custom codecs could trigger this but it is unlikely so I think a latent bug that should be fixed but not a release blocker.

A custom codec which reproduces the issue in 3.15:

import codecs

def encode(s, errors='strict'):
    b = s.encode('utf-8')
    hash(b)                   # a codec may hash its own output, e.g. for a cache
    return b, len(s)

codecs.register(lambda name: codecs.CodecInfo(encode, None, name=name)
                             if name == 'hashy' else None)

ba = bytearray('hello', 'hashy')     # adopts the pre-hashed bytes as its buffer
ba[0] = ord('H')                     # mutates that bytes object in place
b = ba.take_bytes()                  # hands the very same object back out

print(b)                             # b'Hello'
print(hash(b) == hash(b'Hello'))     # False  -- stale hash
print({b: 1}.get(b'Hello', 'MISS'))  # MISS   -- dict lookup is broken

CPython versions tested on:

3.15

Operating systems tested on:

Linux

Linked PRs

• gh-158219: Clear the cached hash when bytearray adopts a bytes buffer #158222