Skip to content

Commit 7e813b7

Browse files
Fix bugs found by the pre-release sweep, update docs (#46)
## Summary Fixes the bugs found by a pre-release sweep of `main` (real PostgreSQL 16 / MySQL 8.3 / SQLite apps on Ruby 3.1–3.4), and brings the docs up to date. **Fixes** - **pg `:batchexec` / `:batchone` / `:batchmany`** now encode parameters like every other command (`Driver#encode_param`), so `bytea`, `json` and arrays work. Before: `string contains null byte` / type errors. - **pg transactions:** a block that rescues a failed statement and returns normally used to "commit" (the server answered `ROLLBACK`, silently). It now raises `PG::InFailedSqlTransaction`. - **mysql2 `:copyfrom`:** binary-safe (`CHARACTER SET binary`), all-or-nothing (transaction, or savepoint inside one), and `LOAD DATA LOCAL` warnings (skipped duplicate keys, coerced invalid values) raise `Mysql2::Error`. - **sqlite3 `:copyfrom`** runs in a savepoint: a failing row no longer leaves earlier batches committed. - **sqlite3 transactions:** threads / fibers sharing one `SQLite3::Database` take turns (one reentrant lock per database, in the new `connection.rb`) instead of breaking each other's savepoints. Unknown `mode:` values raise `ArgumentError` instead of producing invalid SQL. - **Types:** SQLite `JSON` / `JSONB` / `UUID` columns are `String` (were `Float`); a `db_type` override now types the elements of arrays of that type (`uuid[]` → `Array[String]`) instead of replacing the array type. - **RuboCop:** schema names that break a naming cop (`"UserId"`, two-letter or non-ASCII parameters) get a `rubocop:disable` / `enable` pair around only the class or method using them. - **WASM:** an empty `syntax_suggest/core_ext.rb` stub, so plugin errors are no longer prefixed by "`syntax_suggest' was not loaded". - **Options:** unknown keys in an override are rejected (with a "did you mean" hint); clearer `rubyVersion` range errors. **Behaviour changes worth a release note:** pg code that swallowed a failed statement inside `transaction` now raises; MySQL loads that skipped duplicates now fail; users running Steep on generated SQLite code need `library "monitor"`. **Docs:** README reorganised (quick start, one options table, transactions / types / names / limitations sections) without dropping content; Design / Architecture / Testing / Releasing, CLAUDE.md and the PR template updated to the current code (new commands and hooks, stale limitations removed, labels matching the repo). **Tests:** unit tests for each generator change; new E2E tests for batch encoding, the rescued pg statement, MySQL binary / duplicate / invalid / nested `:copyfrom`, SQLite copy atomicity, thread-shared transactions and invalid modes. `examples/`, golden files, request fixtures and Steep expectations regenerated. Locally (Ruby 3.3): `rake test` 85 runs, RuboCop and Steep clean, `sqlc diff` clean with the rebuilt WASM, E2E SQLite 35 / PostgreSQL 49 / MySQL 41 runs green. ## Checklist - [x] `examples/` is regenerated (`rake sqlc_generate_process`) and committed - [x] Tests added or updated (`rake test`, and `rake end2end_*` when the generated code changes) - [x] The generated code stays RuboCop-clean (`rake rubocop`) and Steep-clean (`rake steep`) - [x] Labelled (`bug`, `enhancement`, `documentation`, or `breaking`) so the release notes are grouped correctly --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Doron Eli Rachman <doron050@gmail.com>
1 parent 6102eb9 commit 7e813b7

87 files changed

Lines changed: 1645 additions & 554 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/pull_request_template.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,4 +5,4 @@
55
- [ ] `examples/` is regenerated (`rake sqlc_generate_process`) and committed
66
- [ ] Tests added or updated (`rake test`, and `rake end2end_*` when the generated code changes)
77
- [ ] The generated code stays RuboCop-clean (`rake rubocop`) and Steep-clean (`rake steep`)
8-
- [ ] Labelled (`feature`, `bug`, `breaking`, `documentation`) so the release notes are grouped correctly
8+
- [ ] Labelled (`bug`, `enhancement`, `documentation`, or `breaking`) so the release notes are grouped correctly

‎CLAUDE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ UPDATE_GOLDEN=1 bundle exec ruby -Itest test/extended_golden_test.rb # rewrite
3636
relative to the config file.
3737

3838
## Git, CI and releases
39-
* Develop on a feature branch, small commits; PRs target `main`. Label PRs (`feature`, `bug`, `breaking`, `documentation`).
39+
* Develop on a feature branch, small commits; PRs target `main`. Label PRs (`bug`, `enhancement`, `documentation`; `breaking` for incompatible changes, see `.github/release.yml`).
4040
* CI: `build.yml` (WASM, lint, `sqlc diff`, Steep, option matrix) and `tests.yml` (unit, Ruby 3.1/3.2/3.4, E2E on real
4141
PostgreSQL/MySQL/SQLite). Both must be green.
4242
* **Never create tags or releases by hand and never add release markers to commit messages.** Releases are started by a

‎README.md‎

Lines changed: 157 additions & 178 deletions
Large diffs are not rendered by default.

‎Steepfile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@ target :gen do
22
check "examples"
33
signature "examples"
44

5-
# stdlib signatures for the types the generated code refers to (BigDecimal, Date, Time, IPAddr, Tempfile; Set is core since RBS 4)
6-
library "bigdecimal", "date", "time", "ipaddr", "tempfile"
5+
# stdlib signatures for the types the generated code refers to (BigDecimal, Date, Time, IPAddr, Tempfile, Monitor; Set is core since RBS 4)
6+
library "bigdecimal", "date", "time", "ipaddr", "tempfile", "monitor"
77
end

‎docs/Architecture.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,14 +21,14 @@ stdout ◄─ Proto (encode) ◄─ GenerateResponse (query_sql.rb, query_sql.rb
2121
| `exe/plugin.rb` | entry point. Errors go to stderr with exit code 1 (sqlc shows them); `SQLC_GEN_RUBY_DEBUG=1` adds a backtrace |
2222
| `proto.rb`, `plugin_pb.rb` | a ~150 line protobuf wire codec and the subset of sqlc's `codegen.proto` we read. Field numbers must match upstream. `google-protobuf` is a C extension and cannot run inside ruby.wasm |
2323
| `options.rb`, `ruby_version.rb` | parse + validate the `options:` JSON (`driver`, `rubyVersion`, `overrides`, `rename`, ...; unknown keys are errors) |
24-
| `generator.rb` | the orchestrator: one file per query file (`query_sql.rb`), plus `models.rb`, `connection.rb`, `Gemfile`; builds the module / class / `Data` skeleton |
25-
| `method_gen.rb` | the code of one query: signature (keyword arguments), SQL constant, command (`:one`, `:many`, `:exec`, `:execrows`, `:execlastid`, `:copyfrom`), `sqlc.slice`, bang finder, and the RBS signature |
24+
| `generator.rb` | the orchestrator: one file per query file (`query_sql.rb`), plus `models.rb`, `connection.rb` (per-connection state and helpers the query classes share), `Gemfile`; builds the module / class / `Data` skeleton |
25+
| `method_gen.rb` | the code of one query: signature (keyword arguments), SQL constant, command (`:one`, `:many`, `:exec`, `:execrows`, `:execlastid`, `:execresult`, `:batchexec` / `:batchone` / `:batchmany`, `:copyfrom`), `sqlc.slice`, bang finder, and the RBS signature |
2626
| `row_layout.rb` | turns result columns into row attributes; `sqlc.embed` collapses a table's columns into one model attribute |
27-
| `type_resolver.rb` | RBS type of a column / parameter: driver type table, arrays, nullability (`T?`), `overrides:` |
27+
| `type_resolver.rb` | RBS type of a column / parameter: driver type table, arrays, nullability (`T?`), `overrides:` (a `db_type` override types array elements too) |
2828
| `catalog.rb` | the schema's tables and enums, and their Ruby names |
2929
| `drivers/base.rb` | the driver interface and what all drivers share: `instrument` logging, `NotFound`, transaction RBS |
3030
| `drivers/{pg,mysql2,sqlite3}.rb` | the per-library parts: type mapping, connecting / pooling, preparing and executing statements, transactions, `:copyfrom` |
31-
| `code.rb` | text helpers: indentation, `snake_case`, `camelize`, Ruby-keyword-safe parameter names |
31+
| `code.rb` | text helpers: indentation, `snake_case`, `camelize`, Ruby-keyword-safe parameter names, `rubocop:disable` pairs for schema names that break a naming cop |
3232

3333
**Rule of thumb:** `MethodGen` / `Generator` decide *what* is generated and are the same for every database; a driver only
3434
supplies *snippets* (how to prepare a statement, how to read the affected rows, ...). If you find a `case driver` in
@@ -67,6 +67,10 @@ Design decisions worth knowing (the *why* is in the comments of `drivers/base.rb
6767
* **Pool injection.** `QuerySql.new(pool)` accepts anything that responds to `#with`, so an app can share its pool
6868
and tests can pass a fake. `connect(...)` is only a convenience that builds a `connection_pool`.
6969
* **Statements are closed.** SQLite/MySQL statements are closed in `ensure` (an open SQLite statement blocks `COMMIT`).
70+
* **Parameters are encoded by the driver.** `Driver#encode_param` (e.g. `bytea`, `json` and arrays on `pg`) is used by
71+
every command, batches included; add a type's encoding there, not in `MethodGen`.
72+
* **`:copyfrom` is all or nothing** on every driver (a savepoint / transaction around the load), and MySQL's
73+
`LOAD DATA` warnings (skipped duplicates, coerced values) are raised as errors.
7074
* **Nothing from the plugin leaks into the app.** The generated code only requires the driver gem and `connection_pool`.
7175

7276
## Adding things

‎docs/Design.md‎

Lines changed: 14 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -4,30 +4,22 @@ sqlc plugins receive a protobuf `GenerateRequest` on stdin and answer with a `Ge
44
process or as a WASI module. This plugin is pure Ruby and is distributed as a WASI module (`plugin.wasm`).
55

66
## Source layout
7-
8-
| File | Role |
9-
|---|---|
10-
| `exe/plugin.rb` | entry point (stdin → `Generator` → stdout) |
11-
| `lib/sqlc_gen_ruby/proto.rb` | ~150 line protobuf wire codec. `google-protobuf` is a C extension and cannot be loaded inside ruby.wasm, so the plugin brings its own |
12-
| `lib/sqlc_gen_ruby/plugin_pb.rb` | the subset of sqlc's `codegen.proto` the generator needs (field numbers must match upstream) |
13-
| `lib/sqlc_gen_ruby/options.rb`, `ruby_version.rb` | the `options:` JSON, `rubyVersion` parsing / validation (3.1 – 3.4; `Struct` below 3.2, `Data` from 3.2) |
14-
| `lib/sqlc_gen_ruby/generator.rb` | module / class / dataclass (`Data.define`, `Struct.new`) layout, `.rb` / `.rbs` / `Gemfile` files, `models.rb` (tables + enums) |
15-
| `lib/sqlc_gen_ruby/method_gen.rb` | one generated method per query: `:one`, `:many`, `:exec`, `:execrows`, `:execlastid`, `:copyfrom`, `sqlc.slice` |
16-
| `lib/sqlc_gen_ruby/row_layout.rb` | how result columns become `<Query>Row` attributes (`sqlc.embed` expands a table into one model attribute) |
17-
| `lib/sqlc_gen_ruby/type_resolver.rb`, `catalog.rb` | RBS type of a column (DB type mapping + array + nullability + `overrides:`), tables / enums of the catalog |
18-
| `lib/sqlc_gen_ruby/drivers/*.rb` | `Base` + `Pg`, `Mysql2`, `Sqlite3`: column type mapping, statement preparation / execution, pooling, row access |
19-
| `lib/sqlc_gen_ruby/code.rb` | indent / snake_case helpers |
20-
21-
Driver differences are data, not copy/paste: `Base` has `pooling_enabled?` (ConnectionPool vs a single injected
22-
connection), `row_data_type` (`row['col']` vs `row[0]`), `supports_exec_last_id?` (mysql2 only) and a column mapping
23-
table (exact or prefix based matching).
7+
What each file in `lib/sqlc_gen_ruby/` does is in [Architecture.md](Architecture.md). Two constraints shape it:
8+
* `google-protobuf` is a C extension and cannot be loaded inside ruby.wasm, so the plugin brings its own ~150 line
9+
protobuf codec (`proto.rb`) and the subset of sqlc's `codegen.proto` it reads (`plugin_pb.rb`, field numbers must
10+
match upstream).
11+
* Driver differences are data, not copy/paste: `Drivers::Base` defines hooks (how to prepare and run a statement, how
12+
rows are read - `row_data_type` `:hash` for `row['col']` vs `:array` for `row[0]` -, how `:execlastid` /
13+
`:execresult` / `:copyfrom` / transactions work) and a column mapping table; `Pg`, `Mysql2` and `Sqlite3` fill them in.
2414

2515
## The WASM build (`scripts/wasm/build.sh`)
2616

2717
1. downloads the *prebuilt* `ruby-3.3-wasm32-unknown-wasip1-full` release from ruby/ruby.wasm (this avoids compiling
2818
CRuby with a wasi-sdk);
2919
2. strips the ~18 MB of DWARF debug info / name section from the interpreter (`strip_wasm.rb`);
30-
3. stages a slim stdlib (the plugin only uses core + `json`) + `lib/` + `exe/`;
20+
3. stages a slim stdlib (the plugin only uses core + `json`) + `lib/` + `exe/`, plus an empty
21+
`syntax_suggest/core_ext.rb` (the interpreter loads it to format errors and would otherwise print a warning before
22+
the plugin's error message);
3123
4. embeds that filesystem into the interpreter with `rbwasm pack` (wasi-vfs), because sqlc gives plugins no filesystem.
3224

3325
**The argv trick:** sqlc runs the module as `plugin.wasm /plugin.CodegenService/Generate`, so the interpreter treats that
@@ -52,7 +44,7 @@ transactions, nullability, `overrides`) and Ruby conventions (bang finders, logg
5244
now idiomatic Ruby rather than a copy of the C# one.
5345

5446
## Known limitations
55-
* The SQLite `NUMERIC` affinity can return Integers where `Float` is declared.
56-
* `:copyfrom` for MySQL writes binary (`BLOB`) values through a text CSV; only text-safe values round-trip.
57-
* The generated query classes are not thread-safe for SQLite (a single connection is injected).
58-
* `3.3.x` (a patch version) is rejected as `rubyVersion`; use `3.3`.
47+
See [Limitations in the README](../README.md#limitations). Design choices that look like limitations:
48+
* Overrides only change the RBS types; values are never converted, so the generated code stays a thin layer over the driver.
49+
* SQLite queries do not overlap: a `SQLite3::Database` is one connection, so the generated code takes one lock per
50+
database (held for a whole transaction) to keep threads and fibers from mixing their transactions.

‎docs/Releasing.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,8 @@ code and the options**, because that is what users depend on:
3939
| minor | new options, new query features, new drivers; output that only adds API (e.g. a new method) |
4040
| major | existing generated method signatures, constants or options change incompatibly |
4141

42-
Label pull requests (`breaking`, `feature`, `bug`, `documentation`; `skip-changelog` to hide one) so the generated notes
42+
Label pull requests `bug`, `enhancement` or `documentation` (`breaking` for incompatible changes, `skip-changelog` to
43+
hide one; create those two labels the first time they are needed) so the generated notes
4344
are grouped sensibly, and pick the bump from the labels of what is merged since the last release.
4445

4546
## Using a release

‎docs/Testing.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,13 @@ They `require` the generated code from `examples/` and run it against a real dat
2828
`rbs_types.rb` additionally checks that returned values match the generated `.rbs` types.
2929
* SQLite needs nothing: `rake end2end_sqlite3`.
3030
* PostgreSQL / MySQL: provide a server, load `examples/authors/<db>/schema.sql`, and set `DB_NAME DB_USER DB_PASS
31-
POSTGRES_HOST MYSQL_HOST` (optionally `*_PORT`). MySQL needs `local_infile=1` for `:copyfrom`.
31+
POSTGRES_HOST MYSQL_HOST` (optionally `*_PORT`). MySQL needs `local_infile=1` for `:copyfrom`, and the MySQL tests
32+
connect without a password (`DB_PASS` is only used for PostgreSQL), so use a user without one.
3233
Or run everything in Docker: `rake run_end2end_tests` (docker-compose).
33-
* They cover every query command, transactions (nested, isolation), pool injection, logging, bang finders, `copyfrom`
34-
with awkward values, Fiber-scheduler concurrency (`async` gem) and statement-leak checks.
34+
* They cover every query command (batches with encoded `bytea` / `json` values too), transactions (nested,
35+
isolation, `break` / `throw`, a rescued failed statement on PostgreSQL, SQLite threads sharing a database), pool
36+
injection, logging, bang finders, `:copyfrom` with awkward and binary values and its all-or-nothing behaviour,
37+
Fiber-scheduler concurrency (`async` gem) and statement-leak checks.
3538

3639
## CI workflows (`.github/workflows/`)
3740
| Workflow | When | What |

‎examples/authors/mysql/query.sql‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,5 +110,11 @@ SELECT id, `order`, note FROM regressions WHERE id IN (?, ?, ?) ORDER BY id;
110110
-- name: CopyRegressions :copyfrom
111111
INSERT INTO regressions (`order`, note) VALUES (?, ?);
112112

113+
-- name: CopyRegressionRows :copyfrom
114+
INSERT INTO regressions (id, `order`, note, payload) VALUES (?, ?, ?, ?);
115+
116+
-- name: ListRegressionPayloads :many
117+
SELECT id, payload FROM regressions ORDER BY id;
118+
113119
-- name: TruncateRegressions :exec
114120
TRUNCATE TABLE regressions;

‎examples/authors/mysql/schema.sql‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,5 +67,6 @@ CREATE TABLE extended_bios (
6767
CREATE TABLE regressions (
6868
id bigint NOT NULL AUTO_INCREMENT PRIMARY KEY,
6969
`order` varchar(255) NOT NULL,
70-
note text
70+
note text,
71+
payload varbinary(16)
7172
);

0 commit comments

Comments
 (0)