Skip to content

Resolve Gregorian weekday/month names with CLDR inheritance - #582

Draft
nightsurgex2 wants to merge 1 commit into
Shopify:mainfrom
nightsurgex2:cldr-resolved-hash-inheritance
Draft

Resolve Gregorian weekday/month names with CLDR inheritance#582
nightsurgex2 wants to merge 1 commit into
Shopify:mainfrom
nightsurgex2:cldr-resolved-hash-inheritance

Conversation

@nightsurgex2

Copy link
Copy Markdown

What are you trying to accomplish?

Fixes #581.

Worldwide::Calendar::Gregorian.month_names and .weekday_names return partial results for 104 locales, including en-CA, en-GB, en-IN, en-AU, en-NZ, en-ZA and every other en-001 descendant:

Worldwide::Calendar::Gregorian.month_names(locale: "en-CA", width: :abbreviated)
#=> ["Sept"]                     # before
#=> ["Jan", ..., "Sept", ...]    # after, all twelve

Worldwide::Calendar::Gregorian.weekday_names(locale: "se-FI")
#=> {mon: "mánnodat", tue: "disdat", thu: "duorastat", sat: "lávvordat"}   # before
#=> all seven, sun/wed/fri inherited from `se`                             # after

CLDR resolves locale data item by item: a locale's effective data is the union of its own data with each of its ancestors', the more specific locale winning per item (UTS #35 §4.1). The YAML in data/cldr/locales/ is stored as deltas to match — en-CA/calendars.yml contains only months.stand_alone.abbreviated: {9: Sept}.

I18n::Backend::Fallbacks#translate returns the first non-nil node in the chain. Correct for leaves (...abbreviated.1 still resolves to "Jan" from en), wrong for a whole node: the lookup stops at en-CA's one-entry hash and the eleven inherited entries are never merged in. month_names then calls .values on it, so the truncation is silent.

Full affected-locale table is in the issue. 108 of 4830 method × width × locale combinations were wrong.

What approach did you choose and why?

Chosen: resolve structural keys with CLDR's inheritance rules at read time. Worldwide::Cldr.resolved_hash walks the fallback chain from the least specific ancestor to the most specific, deep-merging each locale's own contribution, and resolves CLDR <alias> nodes against the requested locale rather than the ancestor they were found in. That is what UTS #35 specifies, it is a handful of lines, and it leaves t — and therefore every leaf lookup in the gem — untouched.

Rejected alternatives:

  • Flatten the inheritance at generation time (rake/cldr/locale_generator.rb), writing fully-resolved calendar data into each locale file. Correct output, but it discards the delta structure the CLDR import produces and multiplies data/cldr/locales/ for 805 locales. It also has to be redone for every component that grows this problem, and diverges from what cldr:data:import emits.
  • Deep-merge inside a custom I18n backend. Fixes every key at once, but changes lookup semantics for the host application's own translations, not just CLDR's, and I18n's first-match behaviour is the right one for leaves.
  • Special-case stand_aloneformat in Gregorian. This was my first read of the bug, and it is wrong: the stand_aloneformat alias already works (it lives in root/calendars.yml and I18n resolves symbol links against fallback_original_locale). Hard-coding it would have masked the real problem while leaving se-FI, ar-MA and the en-001 tree broken.

weekday_names / month_names now also raise MissingCalendarDataError when an entry is still missing after resolution. Previously Worldwide::I18nExceptionHandler#degraded_translation turned an unresolved structural key into a humanized copy of its last segment, so a missing weekday table came back as the string "wide" and month_names then died on String#values some distance from the cause. Degrading a leaf into readable text is reasonable; degrading a hash-valued CLDR node into a plausible-looking string is not.

What should reviewers focus on?

  • Worldwide::Cldr#unresolved_node reads a single locale's own contribution with default: nil, fallback: false, resolve: false. default: nil is what keeps a miss returning nil instead of going through the exception handler, and resolve: false is what returns a CLDR alias as a raw Symbol so it can be re-resolved against the requested locale.
  • Alias resolution recurses. The seen set guards against two aliases pointing at each other; I did not find such a cycle in the current data, it is there so a future CLDR import cannot hang.
  • resolved_hash is public because it is generally useful for structural CLDR keys, but nothing else in the gem needs it today — Cldr::DateFormatPattern only reads leaves. Happy to make it private if you'd rather not add API surface.
  • Both methods now key their result off a canonical list (sun..sat, 1..12) rather than the hash's insertion order.

The impact of these changes

Bug fix, no API change for callers that were already getting complete data. Two behaviour changes worth calling out:

  • 104 locales start returning complete names. Anything snapshotting the truncated output will need updating.
  • A structurally missing key now raises Worldwide::Calendar::Gregorian::MissingCalendarDataError instead of returning a string or raising NoMethodError. With this change root always supplies a complete set, so this should be unreachable with the shipped data.

Testing

bundle exec rake test → 2256 tests, 0 failures. bundle exec rubocop → clean.

New coverage:

  • test/worldwide/calendar/gregorian_test.rb asserts complete, non-empty weekday and month names for all 805 known locales at all three widths, plus explicit cases for the alias path (en, nb, which holds no calendar data of its own and inherits from no) and the partial-override path (en-CA, en-GB, ar-MA, se-FI).
  • test/worldwide/cldr_test.rb covers resolved_hash directly, including that it returns {} for a missing key where t returns the degraded "wide".

103 of the new per-locale assertions fail on main without the lib/ change.

Checklist

  • I have added a CHANGELOG entry for this change (or determined that it isn't needed)

CLDR resolves locale data item by item: a locale's effective data is the
union of its own data with each of its ancestors', the more specific
locale winning per item (UTS Shopify#35 4.1). I18n's fallback backend returns
the first non-nil node it finds in the chain instead, which is right for
leaves and wrong for whole nodes.

en-CA overrides exactly one abbreviated month name (Sept), so
month_names(locale: "en-CA", width: :abbreviated) returned ["Sept"]
rather than all twelve. 104 locales were affected across the two methods
and three widths, including every en-001 descendant (en-GB, en-IN,
en-AU, en-NZ, en-ZA, ...), ar-DZ/ar-MA/ar-TN, hi-Latn, sr-Latn-ME,
sr-Latn-XK and se-FI.

Worldwide::Cldr.resolved_hash merges the whole fallback chain and
resolves CLDR <alias> nodes against the requested locale, matching what
CLDR specifies. Gregorian now builds its names from that and raises
MissingCalendarDataError if an entry is still missing, rather than
letting the I18n exception handler degrade the key into a humanized
copy of its last segment (a missing weekday lookup used to come back as
the string "wide", and month_names then died on String#values).

Regression coverage asserts complete data for all 805 known locales at
all three widths; 103 of those cases fail without this change.

Assisted-By: devx/2f3a0f30-70b8-4af7-ae28-67d5a94715cb
@nightsurgex2

Copy link
Copy Markdown
Author

Companion PR for the byte-identical copy in shopify-i18n: Shopify/shopify-i18n#2300

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gregorian#month_names and #weekday_names return partial data for 104 locales (en-CA, en-GB, en-IN, ...)

1 participant