Add JSON metadata export feature by jazairi · Pull Request #1528 · MITLibraries/thing

jazairi · 2026-08-04T22:30:57Z

Why these changes are being introduced:

The metadata team has requested a JSON metadata
export, as it would be easier for them to parse
than the MARC file.

Relevant ticket(s):

ETD-695

How this addresses that need:

This adds JsonExporter and JsonBatch classes, which serialize thesis metadata and writes the metadata
to a JSON file. This workflow is invoked in the
MarcExportJob, and the output is included
alongside the MARC file in the report email.

It also creates two rake tasks: one to export
the JSON metadata of a single thesis, and one
to export all theses in a given term. This is
primarily for testing purposes.

Side effects of this change:

The export job and batch emails are not clearly
named for the time being. This feels like an
acceptable short-term risk. We plan to retire the
MARC export altogether, at which point we can
rename those files accordingly.

Developer

All new ENV is documented in README
All new ENV has been added to Heroku Pipeline, Staging and Prod
ANDI or Wave has been run in accordance to
our guide and
all issues introduced by these changes have been resolved or opened as new
issues (link to those issues in the Pull Request details above)
Stakeholder approval has been confirmed (or is not needed)

Code Reviewer

The commit message is clear and follows our guidelines
(not just this pull request message)
There are appropriate tests covering any new functionality
The documentation has been updated or is unnecessary
The changes have been verified
New dependencies are appropriate or there were no changes

Requires database migrations?

NO

Includes new or updated dependencies?

NO

Why these changes are being introduced: The metadata team has requested a JSON metadata export, as it would be easier for them to parse than the MARC file. Relevant ticket(s): - [ETD-695](https://mitlibraries.atlassian.net/browse/ETD-695) How this addresses that need: This adds JsonExporter and JsonBatch classes, which serialize thesis metadata and writes the metadata to a JSON file. This workflow is invoked in the MarcExportJob, and the output is included alongside the MARC file in the report email. It also creates two rake tasks: one to export the JSON metadata of a single thesis, and one to export all theses in a given term. This is primarily for testing purposes. Side effects of this change: The export job and batch emails are not clearly named for the time being. This feels like an acceptable short-term risk. We plan to retire the MARC export altogether, at which point we can rename those files accordingly.

coveralls · 2026-08-04T22:34:19Z

coverage: 98.286% (+0.03%) from 98.252% — etd-695 into main

Copilot

Pull request overview

Adds a JSON metadata export alongside the existing MARC batch export workflow, so downstream metadata consumers can parse thesis metadata without MARC processing.

Changes:

Introduces JsonExporter and JsonBatch to serialize thesis metadata and write a JSON export file.
Updates MarcExportJob and BatchMailer to generate and email both MARC (zip) and JSON attachments.
Adds rake tasks and tests covering JSON export generation and email attachments.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`app/models/json_exporter.rb`	Defines per-thesis JSON-serializable metadata structure.
`app/models/json_batch.rb`	Builds a batch JSON file wrapping exported theses.
`app/jobs/marc_export_job.rb`	Generates JSON alongside MARC and attaches both to the batch email.
`app/mailers/batch_mailer.rb`	Extends batch email to include a JSON attachment and updates subject.
`app/views/batch_mailer/marc_batch_email.html.erb`	Updates email body to mention JSON alongside MARC.
`lib/tasks/metadata.rake`	Adds rake tasks for exporting JSON for a single thesis or a term batch.
`test/models/json_exporter_test.rb`	Adds unit tests for `JsonExporter#to_hash`.
`test/models/json_batch_test.rb`	Adds tests for JSON batch file structure/validity.
`test/mailers/batch_mailer_test.rb`	Updates mailer tests to expect both MARC+JSON attachments and new subject.
`test/jobs/marc_export_job_test.rb`	Adds coverage for both attachments and JSON validity in the export job email.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

jazairi · 2026-08-04T23:08:13Z

I'm working with Ben on the content of the JSON file. I think this is ready for code review, but more changes may be coming depending on stakeholder feedback.

JPrevost · 2026-08-05T17:45:19Z

@@ -0,0 +1,25 @@
+class JsonBatch


I think JsonBatch and JsonExporter could use better intention-based names. Yes, the output is JSON, but that feels like it doesn't tell us why these models exist. Maybe CatalogBatch and CatalogExporter would be clearer intention?

This change would trickle into most of the other changes being introduced here to make the naming consistency.

JPrevost · 2026-08-05T17:48:44Z

+
+  def write_json_file(json_file)
+    theses_data = @theses.map do |thesis|
+      JsonExporter.new(thesis).to_hash


This line supports my suggestion for a different name for JsonExporter... I was really confused why we'd convert a JsonExporter to a hash until I realized it isn't a JsonExporter, it's a CatalogExporter

JPrevost · 2026-08-05T17:51:42Z

+    json_output = { theses: theses_data }
+
+    json_file.write(JSON.pretty_generate(json_output))
+    json_file.rewind


Interesting. This keeps returns the file but in a ready to use state rather than closing and reopening?

Sort of, yeah. It resets to the beginning of the stream, so calling read actually sees the data. (Otherwise, it would parse it as an empty string.) The alternative is to close the file, which would also reset the pointer to the beginning.

JPrevost · 2026-08-05T17:53:17Z

+  end
+
+  def title
+    @thesis.title&.squish


I'm curious why we have a save navigator here and not on other fields? There is likely no harm here, but don't we check for titles before publication so we technically shouldn't need safe navigation? I'm not asking for a change, just making sure you considered whether others might also need similar treatment I guess?

This came from the MARC exporter. It was undocumented, but my best guess is that we call squish because of weirdly formatted titles.

As for the safe navigation, I don't think it's actually required for this field, as it's technically impossible for a thesis to be published without a title.

I believe the same is true of all the other fields we export, although they're required in different ways. For example, title, advisors, and degrees are in the required_fields? method, whereas departments and graduation_year have conventional Rails validations. Iirc, this is because they are added at different points in the transfer/submission/processing workflows.

JPrevost · 2026-08-05T17:58:51Z

@@ -0,0 +1,47 @@
+namespace :metadata do
+  desc 'Generate a JSON export of a single published thesis for testing'


Would you mind rewording testing to debugging or similar?

JPrevost · 2026-08-05T17:59:51Z

+    end
+  end
+
+  desc 'Generate a JSON export batch for a specific term (e.g., "2024-June") and save to temp file'


Maybe we should note this isn't intended for use in production? I'm worried we may be confused later when we save a temp file in a temp dyno and can't figure out where it went or something? Not sure if this is necessary so just think about it and make a note or not!

Good call. I might be more inclined to call close! or unlink instead, as a safety precaution. Even though we haven't needed to manually export metadata in production, I imagine we might at some point.

- Rename and document new batch/export classes - Remove unnecessary safe navigation - Call `close!` on tempfiles to ensure immediate cleanup

jazairi · 2026-08-06T00:00:44Z

@JPrevost For some reason, the git mv for CatalogBatch got lost in the rebase. Sorry if that makes this re-review more difficult!

mitlib temporarily deployed to thesis-submit-pr-1528 August 4, 2026 22:31 Inactive

jazairi requested a review from Copilot August 4, 2026 22:37

Copilot started reviewing on behalf of jazairi August 4, 2026 22:38 View session

Copilot AI reviewed Aug 4, 2026

View reviewed changes

Comment thread app/models/json_batch.rb Outdated

Comment thread lib/tasks/metadata.rake Outdated

Comment thread app/views/batch_mailer/marc_batch_email.html.erb Outdated

Apply Copilot code review suggestions

ff7c8c3

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

jazairi requested review from JPrevost and matt-bernhardt August 4, 2026 23:06

jazairi temporarily deployed to thesis-submit-pr-1528 August 4, 2026 23:07 Inactive

JPrevost self-assigned this Aug 5, 2026

JPrevost requested changes Aug 5, 2026

View reviewed changes

Address code review feedback

0fc146c

- Rename and document new batch/export classes - Remove unnecessary safe navigation - Call `close!` on tempfiles to ensure immediate cleanup

jazairi deployed to thesis-submit-pr-1528 August 5, 2026 23:59 View deployment

jazairi requested a review from JPrevost August 5, 2026 23:59

		@@ -0,0 +1,47 @@
		namespace :metadata do
		desc 'Generate a JSON export of a single published thesis for testing'

Uh oh!

Conversation

jazairi commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why these changes are being introduced:

Relevant ticket(s):

How this addresses that need:

Side effects of this change:

Developer

Code Reviewer

Requires database migrations?

Includes new or updated dependencies?

Uh oh!

coveralls commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jazairi commented Aug 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jazairi Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jazairi commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jazairi commented Aug 4, 2026 •

edited

Loading

coveralls commented Aug 4, 2026 •

edited

Loading

jazairi Aug 5, 2026 •

edited

Loading