Clarify validation semantics for omitted dataset dtype (e.g., VectorData) across APIs

[ehennestad]

Summary

For datasets whose schema omits dtype, it is not clear what validation behavior should be considered normative across APIs.

A concrete example is hdmf_common/VectorData, which omits dtype in the schema. In Python HDMF, omitted dtype still has practical runtime restrictions based on build-time inference and backend handling, but those rules do not appear to be documented as schema-level validation requirements.

For another implementation such as MatNWB, this creates ambiguity about what should be accepted or rejected during validation.

Concrete problem in MatNWB

At the moment, the following is possible in MatNWB:

types.hdmf_common.VectorData(data, types.hdmf_common.VectorData(data, 1))

This seems like it should be invalid, because the payload of a dataset is itself another HDMF container object rather than dataset data.

However, because MatNWB currently does not perform type validation when dtype is omitted, this is accepted.

Question

What is the intended normative validation behavior for datasets with omitted dtype, especially concrete types such as VectorData?

More specifically:

• Should omitted dtype be interpreted as "no fixed primitive dtype is prescribed by the schema", while still requiring the dataset payload to be valid leaf/storable data?
• Should implementations reject HDMF container objects as direct dataset payloads unless the schema explicitly declares a reference dtype?
• Which parts of Python HDMFs current behavior for omitted dtype are intentional cross-API semantics, and which parts are just implementation details of that API?

Why this matters

As a maintainer of another HDMF-based API, I need to know whether validation for omitted dtype should be:

• schema-level permissive: no explicit dtype constraint, but still reject obviously non-dataset objects
• Python-HDMF-compatible: mirror current inference/restriction behavior
• something else

Right now, the schema appears intentionally generic, but the practical validation semantics are underspecified for alternative implementations.

Request

Could the intended semantics for omitted dataset dtype be clarified, especially for VectorData and similar concrete types?

If there is already an intended rule, documenting it in the schema language docs and/or hdmf-common docs would help other implementations validate consistently.

This issue text was drafted with the help of Codex (GPT-5.4).

[rly]

We clarified this at our team meeting today. Omitted dtype means that any dtype, including any of the basic dtype spec values, any reference dtype, and any compound dtype, are allowed. I created hdmf-dev/hdmf-schema-language#49 to clarify the docs.

[rly]

I think we can close this and track the issue over in hdmf-dev/hdmf-schema-language#49.

Note that HDMF does not yet support storing compound dtype data when dtype is unspecified in the spec. #655 tracks this. #1265 explicitly disallows it with a clear error message. In the future, we may want to add support for storing compound dtype data when the field names and dtypes can be fully determined from the data. I think it should be supported if possible, but we should evaluate the use cases - separate VectorData or TimeSeries objects may be better. If we eventually remove compound dtypes from the schema and the schema language because they are a pain to support (NeurodataWithoutBorders/nwb-schema#588), then this will no longer be an issue.

HDMF does support storing object references when dtype is unspecified in the spec. I tested that with VectorData containing pointers to other containers. References are properly created. I don't know why someone would create an NWB TimeSeries with object references to other containers as data, but it is allowed.