Benchmarks for protein data

[Hrovatin]

Adds convergence benchmarks for deep mutational scanning (DMS) datasets, optimizing measured score over pools of single-amino-acid mutants.

What's included

• New domain protein with one benchmark per dataset (protein_), registered in BENCHMARKS.
• Encoding: ESMpp_small mean-pooled embeddings (raw, no PCA) via a CustomDiscreteParameter.
• Objective: maximize score; left-skewed datasets are negated so all benchmarks maximize right-skewed scores.
• Recommenders: default BayBE vs. a random baseline.
• Scenarios: two equal-budget schedules (batch 30×10 and 60×5, both 300 evals). 5 rounds is more realistics for wet lab (ideally 2-3), but could be harder to interpret due to short trajectory.
• Metrics per iteration: score_CumBest, plus instance- and position-retrieval at top 5/10/20%.

Data

• mutations.tsv (information on mutants), embeddings_*.parquet (ESMpp_small/ProtT5XL/ProstT5), and metadata.json (score skewness).
• Other encodings were added besides the one used for benchmark - this may be useful later if adding ensembling functionality.
• The number of datasets could potentially be narrowed down (e.g., keep only one poorly/well performing dataset). However, we saw in other benchmarks that some datasets are easier/harder for certain regressors (e.g., kelsic was chalenging for BayBE although other regressors performed well). To see how other regressor predictors work on this data see https://github.com/merckgroup/baybe-analysis/blob/main/method_improvements/data/prepare_data_protein/readme.md

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds metadata and release notes to support new protein deep mutational scanning (DMS) convergence benchmarks.

Changes:

• Added per-dataset metadata.json files containing score_skewness for multiple protein DMS datasets.
• Updated CHANGELOG.md to document the addition of the new protein_<dataset> benchmarks and associated metrics.

Reviewed changes

Copilot reviewed 10 out of 59 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
benchmarks/data/protein/lee/metadata.json	Adds dataset skewness metadata used by protein benchmark logic.
benchmarks/data/protein/kelsic/metadata.json	Adds dataset skewness metadata used by protein benchmark logic.
benchmarks/data/protein/jones/metadata.json	Adds dataset skewness metadata used by protein benchmark logic.
benchmarks/data/protein/haddox/metadata.json	Adds dataset skewness metadata used by protein benchmark logic.
benchmarks/data/protein/giacomelli/metadata.json	Adds dataset skewness metadata used by protein benchmark logic.
benchmarks/data/protein/doud/metadata.json	Adds dataset skewness metadata used by protein benchmark logic.
benchmarks/data/protein/cov2_S/metadata.json	Adds dataset skewness metadata used by protein benchmark logic.
benchmarks/data/protein/cas12f/metadata.json	Adds dataset skewness metadata used by protein benchmark logic.
benchmarks/data/protein/brenan/metadata.json	Adds dataset skewness metadata used by protein benchmark logic.
CHANGELOG.md	Documents the newly added protein DMS benchmark suite under Unreleased.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Scienfitz]

The addition of protein benchmarks is most welcome, but this PR attempts to add several hundreds of MB in data, which is unacceptable

blocking review until a solution for data deployment is found

[AdrianSosic]

The addition of protein benchmarks is most welcome, but this PR attempts to add several hundreds of MB in data, which is unacceptable

blocking review until a solution for data deployment is found

I think it may be the time now that we either:

• migrate benchmarks to their very own repo or
• at least move their data to a dvc-managed repo

Thoughts?

[Scienfitz]

• moving large data in a git repo is a bit of an unintended use, so a dvc variant seems more reasonable
• we also have the option to sue our factory for a simple s3 bucket (git secrets for access rights)
• not sure where these data are from, but if they're public and the postprocessing is not super complex, they could also be "generated" on the fly

[AVHopp]

@Scienfitz @AdrianSosic as written in Teams, this Branch/PR is for now only meant as a place where we can have access to and discuss about usage of the protein data. It is not necessarily decided/intended to have all of them as part of the BayBE repo. We should (and will) discuss the points you mentioned above in one of our dev meetings, this is already on the agenda

[Scienfitz]

@AVHopp ok I get the intention, this will clearly make progress in the meeting

But please be aware:

• You've not just created a branch, but opened a PR, which by its definition says "I intend to merge something"
• This PR is a green non-draft PR, which notifies all reviewers and says "please review me now"

I take it from your message none of these are the intention you currently have

Also, this PR seemingly manages to break github itself - at least for me. There are no file differences available - this is not exactly helping facilitate discussion (one of the stated aims)

[Hrovatin]

@Scienfitz please review if you agree with the code so I can adjust if needed. @AVHopp will deal with data later, but the benchmark logic is independent of that.

I moved data to sharepoint for now as it was making issues with viewing the PR https://mdigital.sharepoint.com/:f:/r/sites/BayBE-BayesianBackEnd-ExtendedCoreTeam/Shared%20Documents/data/[pull-885](https://mdigital.sharepoint.com/:f:/r/sites/BayBE-BayesianBackEnd-ExtendedCoreTeam/Shared%20Documents/data/pull-885?d=w2e081b7870844ba6bcfdda42df6833a0&csf=1&web=1&e=gNCOLk)?d=w2e081b7870844ba6bcfdda42df6833a0&csf=1&web=1&e=gNCOLk

[AVHopp]

@Scienfitz Github was misbehaving in general yesterday - this is not related to the PR.

I admit that communication around this work was confusing and misleading, sorry for that. This PR actually also contains new benchmarking logic, hence the reason why it should be reviewed already now (in particular as it now doesn't contain any data anymore).

@Hrovatin: Please DELETE THAT LINK and share this via teams. This is a public repository, we should not have links to our company internal data and platforms here. Also, please ensure that the tests pass, currently not even pre-commit is green. Furthermore, make sure that the descriptions in the code are clear as they might stll refer to specific data sets to specific positions which are not there any more.

[Scienfitz]

Summary of Discussion Outcome

• We can handle the data via s3 and our AWS factory
• The auth can be handled similarly to the AWS runners
• The code needs to be adjusted for using these
• PR will be reviewed once thats included