Improve scenario migration safety and process

[louispt1]

Description

At the moment, it is hard to test scenario migrations 100%, because we do not have a stable dump of all migratable scenarios on beta/pro. That means that edge cases which were not considered during development can cause the migration to fail halfway through, leading to complications like twice-migrated scenarios.

---

Current Behavior

We currently have a class, ScenarioMigration, which filters for keep_compatible scenarios, and provides useful output about how many scenarios where migrated etc. This method also raises an error if no scenarios were actually effected by the given migration.

---

Proposed Improvement

The easiest improvement would be enhanced logging:

• 1. More verbose output in the case of failures - it would be helpful to at least know which scenarios were the last to be migrated, and to know that the order of migration is oldest to newest (older scenarios are more likely to break, and generally less likely to be important).

More complicated, but to give us certainty with our migrations would be to add a 'dry-run' option:

• 2. Basically skip the scenario.save step for each scenario, but add more logging in some sort of handy format (store in a tmp file or log directly to logs).

So to test a migration on beta for example, I would push the migration to beta (without schema changes) and then trigger the dry run on beta with a rake/runner task. If happy with the results, just retrigger the task with dry-run:false and the migration will run for real.

---

Success Criteria

• It's harder to make a mistake when running a migration
• When migrations do go wrong, it's easy to identify why and resolve the issue

[github-actions]

This issue has had no activity for 60 days and will be closed in 7 days. Removing the "Stale" label or posting a comment will prevent it from being closed automatically. You can also add the "Pinned" label to ensure it isn't marked as stale in the future.