USE 531 - Add "fulltexts" data type

[ghukill]

Purpose and background context

This PR adds a new "fulltexts" data type to the TIMDEX dataset, sitting along pre-existing data types "records" and "embeddings".

This data type add leverages the new data type parity, requiring basically just a schema + metadata definition, relying on shared code for all data types.

This "fulltext" rows capture fulltext associated with a TIMDEX record. At this time, only DSpace theses will have fulltext records, but we may explore including LibGuides and MITLib websites fulltext in here as well (currently they are added directly to the Opensearch document by Transmogrifier, making them searchable, but not available in the dataset for any additional uses). We may also explore fulltext for other sources, e.g. archival materials, Dome materials, etc.

Unlike embeddings, there is not much metadata about fulltexts at this time. We do include an MD5 checksum of the fulltext bytes to avoid re-harvesting / re-writing fulltext in the future, but that is not currently utilized (though CLI app dspace-fulltext-harvester may use it). The primary data column is simply fulltext with the fulltext of the record as bytes.

The first planned use of this new data type is this:

1. For source dspace, run the dspace-fulltext-harvester as part of the TIMDEX ETL StepFunction
2. Write data back to this new data type
3. Update TIM to update Opensearch records with fulltext stored here, very similar to how we did embeddings

How can a reviewer manually see the effects of these changes?

1- Set Dev TimdexManagers Credentials

2- Set env vars:

TDA_LOG_LEVEL=DEBUG
WARNING_ONLY_LOGGERS=asyncio,botocore,urllib3,s3transfer,boto3,MARKDOWN
TIMDEX_DATASET_LOCATION=s3://timdex-extract-dev-222053980223/dataset

3- Load dataset:

import os

from timdex_dataset_api import TIMDEXDataset
from timdex_dataset_api.config import configure_dev_logger

configure_dev_logger()

td = TIMDEXDataset(os.environ["TIMDEX_DATASET_LOCATION"])

4- Write batch of fulltexts records, using an arbitrary researchdatabases ETL run as the source TIMDEX records:

from tests.utils import generate_sample_fulltexts_for_run

# utility to generate some fulltexts
fulltexts = generate_sample_fulltexts_for_run(
    timdex_dataset=td,
    run_id="18cbec87-ce25-4355-a526-caaa3eb1ac1e",  # run for 'researchdatabases'
)

# write to dataset, using recently normalized 'td.<source>.write(...)'
td.fulltexts.write(fulltexts)
# INFO:timdex_dataset_api.data_type:Dataset write complete - elapsed: 12.0s, total files: 1, total rows: 901, total size: 101180

5- Confirm our write worked and we can see records:

# get all fulltexts for 'researchdatabases' source
td.fulltexts.read_dataframe(table='current_fulltexts')

# get a single 'fulltexts' record (where the 'xxxxx...' is just filler text)
next(td.fulltexts.read_dicts_iter(table='current_fulltexts'))
"""
Out[12]: 
{'timdex_record_id': 'researchdatabases:az-65257807',
 'source': 'researchdatabases',
 'run_date': datetime.datetime(2026, 5, 1, 0, 0),
 'run_type': 'full',
 'action': 'index',
 'run_id': '18cbec87-ce25-4355-a526-caaa3eb1ac1e',
 'run_record_offset': 0,
 'run_timestamp': datetime.datetime(2026, 5, 1, 15, 24, 31, 248765, tzinfo=<DstTzInfo 'America/Detroit' EDT-1 day, 20:00:00 DST>),
 'filename': 's3://timdex-extract-dev-222053980223/dataset/data/fulltexts/year=2026/month=05/day=01/ab02e407-b62d-4807-8f3b-7dd1c3a4d6ad-0.parquet',
 'fulltext_timestamp': datetime.datetime(2026, 5, 1, 15, 24, 31, 248765, tzinfo=<DstTzInfo 'America/Detroit' EDT-1 day, 20:00:00 DST>),
 'fulltext_md5': '48dac3804fbe73d3aeb1e272e0460045',
 'fulltext': b'Sample fulltext content for researchdatabases:az-65257807.  Content xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'}
"""

Note that the record returned, and as we'll see below in step 6, contain additional metadata about the TIMDEX record like [source, run_date, action, ...] etc. This is a byproduct of the v5 refactoring, making all data types in the dataset see very similar "base" metadata for each row, specifically the record it's associated with.

6- Confirm the fulltext_md5 is a queryable / filterable metadata field, not a data field:

td.conn.query("""select * from metadata.fulltexts where fulltext_md5 = '967d0a0bf60829bee5dfc05476cc0f0f';""")
# NOTE: there may be multiple rows for this MD5 checksum, given multiple writes

Includes new or updated dependencies?

NO

Changes expectations for external applications?

NO

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/USE-531

Code review

• Code review best practices are documented here and you are encouraged to have a constructive dialogue with your reviewers about their preferences and expectations.

[ehanson8]

Works as expected and code looks good!