Skip to content

Use UTF-8 for expected ledger fixtures - #218

Open
ryanduguid wants to merge 1 commit into
beancount:masterfrom
ryanduguid:fix/utf8-expected-fixtures
Open

ryanduguid wants to merge 1 commit into
beancount:masterfrom
ryanduguid:fix/utf8-expected-fixtures

Conversation

@ryanduguid

Copy link
Copy Markdown

Summary

Expected ledger fixtures now use UTF-8 for generation and comparison. With a GBK default, the previous writer produced bytes that UTF-8 readers could not decode, and the reader corrupted valid UTF-8 text before comparing it. Fixes #194.

The change adds encoding="utf-8" to the two existing file opens. Current Beancount requires UTF-8 ledger input; the tests verify these fixtures with Beancount 2.x and 3.x.

Evidence

  • Both new tests fail on the original source. They use real files and emulate a GBK default only for this module's opens. They cover UTF-8 output and comparison, exclusive creation, forced overwrite and a Beancount parser roundtrip.
  • All 164 tests pass on Linux with Python 3.8.20/Beancount 2.3.6, 3.11.16/3.0.0, 3.11.16/3.2.3 and 3.14.7/3.2.3. Both new tests pass on Windows with Python 3.14.7/Beancount 3.2.3.
  • Full CI Ruff, source and wheel builds, fresh wheel installation and all nine CI example commands pass. The installed production module matches the tested source hash.
  • Three Aikido findings concern existing file accesses and an assertion. Removing the two new encoding arguments makes the complete production AST identical to upstream.

Merge danger

Reversible. Affects expected fixture generation and comparison. Existing non-UTF-8 expected files containing locale-encoded text must be converted or regenerated as UTF-8; comparison may otherwise raise UnicodeDecodeError.

Unverified

An actual Windows GBK system locale, Python 3.7, remaining CI runtime/dependency combinations and upstream CI. The full Windows suite retains the same 16 failed nodes before and after, with 143 passing tests becoming 145 and three skips.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Problem about encoding handling in testing module on Windows with Chinese locale

1 participant