Skip to content

feat(storage): add model metadata tables#6799

Open
tanishqgandhi1908 wants to merge 1 commit into
apache:mainfrom
tanishqgandhi1908:feat/model-metadata-tables
Open

feat(storage): add model metadata tables#6799
tanishqgandhi1908 wants to merge 1 commit into
apache:mainfrom
tanishqgandhi1908:feat/model-metadata-tables

Conversation

@tanishqgandhi1908

@tanishqgandhi1908 tanishqgandhi1908 commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Adds the database tables that track ML models as a first-class resource in Texera. The model* tables mirror the existing dataset* tables — their own primary key (mid), their own LakeFS repo namespace — and
add model-specific attributes (framework, format):

  • model — the model asset (owner, name, visibility, description, cover image, framework, format)
  • model_user_access — per-user privilege (reuses privilege_enum)
  • model_version — versions of a model (mvid, version_hash)
  • model_upload_session / model_upload_session_part — multipart upload bookkeeping
  • model_user_likes / model_view_count — like/view counters

The change is pure schema, with no application code and no behavior change:

  • Fresh installs: tables added to sql/texera_ddl.sql (with matching DROP TABLE entries and FK-safe ordering).
  • Existing deployments: a new idempotent migration sql/updates/30.sql (all CREATE TABLE IF NOT EXISTS, no data backfill), registered as changeSet id="30" in sql/changelog.xml.
  • The JOOQ data-access layer is git-ignored and regenerated from the live DB on build, so no generated code is committed.

Migration ordering note: this uses id=30 to sit after #6495's migration 29 (PR #6502, currently in review). These model tables do not depend on #6495, so this PR can be reviewed independently.

Any related issues, documentation, discussions?

Closes #6496.

How was this PR tested?

Validated against a throwaway groonga/pgroonga:4.0.1-debian-15 Postgres:

  • sql/texera_ddl.sql runs end-to-end with no errors; all 7 model* tables are created with the expected primary keys, foreign keys (ON DELETE CASCADE), unique constraints, and CHECK constraints.
  • sql/updates/30.sql applies cleanly on a DB that already has the dataset tables but no model tables, and is idempotent

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

@github-actions github-actions Bot added feature ddl-change Changes to the TexeraDB DDL labels Jul 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @kunwp1
    You can notify them by mentioning @kunwp1 in a comment.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 2 worse · ⚪ 13 noise (<±5%) · 0 without baseline

Compared against main a9f384e benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
bs=10 sw=10 sl=64 419 0.256 23,401/31,031/31,031 us ⚪ within ±5% / 🔴 +92.5%
🔴 bs=100 sw=10 sl=64 927 0.566 105,383/155,687/155,687 us 🔴 +16.5% / 🔴 +44.9%
bs=1000 sw=10 sl=64 1,127 0.688 885,944/918,819/918,819 us ⚪ within ±5% / 🟢 -13.0%
Baseline details

Latest main a9f384e from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 419 tuples/sec 439 tuples/sec 760.2 tuples/sec -4.6% -44.9%
bs=10 sw=10 sl=64 MB/s 0.256 MB/s 0.268 MB/s 0.464 MB/s -4.5% -44.8%
bs=10 sw=10 sl=64 p50 23,401 us 22,596 us 12,781 us +3.6% +83.1%
bs=10 sw=10 sl=64 p95 31,031 us 31,851 us 16,118 us -2.6% +92.5%
bs=10 sw=10 sl=64 p99 31,031 us 31,851 us 19,074 us -2.6% +62.7%
bs=100 sw=10 sl=64 throughput 927 tuples/sec 960 tuples/sec 988.68 tuples/sec -3.4% -6.2%
bs=100 sw=10 sl=64 MB/s 0.566 MB/s 0.586 MB/s 0.603 MB/s -3.4% -6.2%
bs=100 sw=10 sl=64 p50 105,383 us 100,768 us 101,448 us +4.6% +3.9%
bs=100 sw=10 sl=64 p95 155,687 us 133,606 us 107,410 us +16.5% +44.9%
bs=100 sw=10 sl=64 p99 155,687 us 133,606 us 120,258 us +16.5% +29.5%
bs=1000 sw=10 sl=64 throughput 1,127 tuples/sec 1,122 tuples/sec 1,023 tuples/sec +0.4% +10.2%
bs=1000 sw=10 sl=64 MB/s 0.688 MB/s 0.685 MB/s 0.624 MB/s +0.4% +10.2%
bs=1000 sw=10 sl=64 p50 885,944 us 888,810 us 983,888 us -0.3% -10.0%
bs=1000 sw=10 sl=64 p95 918,819 us 930,691 us 1,030,100 us -1.3% -10.8%
bs=1000 sw=10 sl=64 p99 918,819 us 930,691 us 1,056,513 us -1.3% -13.0%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,476.80,200,128000,419,0.256,23401.03,31030.57,31030.57
1,100,10,64,20,2157.09,2000,1280000,927,0.566,105383.05,155687.17,155687.17
2,1000,10,64,20,17739.14,20000,12800000,1127,0.688,885944.17,918818.90,918818.90

@tanishqgandhi1908

Copy link
Copy Markdown
Contributor Author

/request-review @aicam

@github-actions
github-actions Bot requested a review from aicam July 22, 2026 19:46
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.20%. Comparing base (a9f384e) to head (de806ba).
⚠️ Report is 7 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #6799      +/-   ##
============================================
- Coverage     75.20%   75.20%   -0.01%     
+ Complexity     3447     3446       -1     
============================================
  Files          1160     1160              
  Lines         45807    45807              
  Branches       5071     5071              
============================================
- Hits          34451    34450       -1     
  Misses         9745     9745              
- Partials       1611     1612       +1     
Flag Coverage Δ *Carryforward flag
access-control-service 70.00% <ø> (ø)
agent-service 76.76% <ø> (ø) Carriedforward from a9f384e
amber 67.17% <ø> (-0.01%) ⬇️
computing-unit-managing-service 20.49% <ø> (ø)
config-service 66.66% <ø> (ø)
file-service 67.21% <ø> (ø)
frontend 79.42% <ø> (ø) Carriedforward from a9f384e
notebook-migration-service 78.94% <ø> (ø)
pyamber 92.15% <ø> (ø) Carriedforward from a9f384e
workflow-compiling-service 55.14% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ddl-change Changes to the TexeraDB DDL feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add model metadata tables

2 participants