Replace newline token with automatic semicolons

[aspeddro]

Summary

• replace the external _newline token with _automatic_semicolon
• define statement boundaries through an internal _semicolon rule
• treat ordinary newlines as whitespace and insert semicolons only when the current statement can end
• preserve multiline ReScript constructs across comments, including pipes, member access, variants, ternaries, recursive and bindings, else clauses, and module with constraints
• regenerate the parser artifacts and add focused semicolon corpus coverage

Why

The previous scanner emitted a general-purpose newline token and the grammar used that token directly as a statement delimiter. This made newline significance part of grammar rules and required special newline handling in declarations, variants, switch expressions, and extension payloads.

Automatic semicolon handling keeps newlines as ordinary whitespace. The scanner now emits a hidden semicolon only at EOF, after a line terminator when the following token cannot continue the current construct, and before a closing brace when appropriate.

ReScript's external nested block comments require carrying line-terminator state across comment tokens. A hidden continuation extra clears that state when the token after a comment continues the current expression or declaration.

The closing-brace behavior is also adapted for ReScript record patterns. A same-line } does not force insertion because doing so would commit ({id}) => ... to the competing block-expression parse.

Validation

• npx tree-sitter test: 154 corpus tests and all highlighting assertions pass
• ./node_modules/.bin/tree-sitter parse test_wild/**/*.res --quiet: all 2,092 ReScript fixtures parse without errors
• EOF insertion is exercised by a real fixture without a trailing newline
• git diff --check passes

[aspeddro]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 4746b7980a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".