Skip to content

Zipper via Reranker. Load more. Result numbers. - #448

Open
JPrevost wants to merge 5 commits into
mainfrom
reranker
Open

Zipper via Reranker. Load more. Result numbers.#448
JPrevost wants to merge 5 commits into
mainfrom
reranker

Conversation

@JPrevost

@JPrevost JPrevost commented Aug 3, 2026

Copy link
Copy Markdown
Member

This pull request introduces our Reranker gem, but keeps the Zipper merge default. This sets us up for future algorithms, but allows us to separate this structural change from algorithm selection.

It also reworks pagination to be load more that relies heavily on a new redis cache approach. This is quite a bit simpler than our initial approach because we are dropping support for jumping to arbitrary pages.

It also introduces a first pass at reintroducing result numbers. A separate ticket has been created and assigned to Dave to adjust as needed and ensure it works well in mobile sized viewports.

It tried to break this into separate PRs, but failed.

Developer

Accessibility
  • ANDI or WAVE has been run in accordance to our guide.
  • This PR contains no changes to the view layer.
  • New issues flagged by ANDI or WAVE have been resolved.
  • New issues flagged by ANDI or WAVE have been ticketed (link in the Pull Request details above).
  • No new accessibility issues have been flagged.
New ENV
  • All new ENV is documented in README.
  • All new ENV has been added to Heroku Pipeline, Staging and Prod.
  • ENV has not changed.
Approval beyond code review
  • UXWS/stakeholder approval has been confirmed.
  • UXWS/stakeholder review will be completed retroactively.
  • UXWS/stakeholder review is not needed.
Additional context needed to review

E.g., if the PR includes updated dependencies and/or data
migration, or how to confirm the feature is working.

Code Reviewer

Code
  • I have confirmed that the code works as intended.
  • Any CodeClimate issues have been fixed or confirmed as
    added technical debt.
Documentation
  • The commit message is clear and follows our guidelines
    (not just this pull request message).
  • The documentation has been updated or is unnecessary.
  • New dependencies are appropriate or there were no changes.
Testing
  • There are appropriate tests covering any new functionality.
  • No additional test coverage is required.

@coveralls

coveralls commented Aug 3, 2026

Copy link
Copy Markdown

Coverage Report for CI Build 30911640308

Coverage decreased (-0.06%) to 98.265%

Details

  • Coverage decreased (-0.06%) from the base build.
  • Patch coverage: 1 uncovered change across 1 file (126 of 127 lines covered, 99.21%).
  • No coverage regressions found.

Uncovered Changes

File Changed Covered %
app/models/merged_search_service.rb 101 100 99.01%
Total (2 files) 127 126 99.21%

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 1556
Covered Lines: 1529
Line Coverage: 98.26%
Coverage Strength: 81.91 hits per line

💛 - Coveralls

@qltysh

qltysh Bot commented Aug 3, 2026

Copy link
Copy Markdown

❌ 26 blocking issues (33 total)

Tool Category Rule Count
rubocop Lint Unused block argument - query. If it's necessary, use \_ or \_query as an argument name to indicate that it won't be used. 9
rubocop Lint Method has too many lines. [25/10] 6
rubocop Lint Assignment Branch Condition size for results is too high. [<6, 33, 8> 34.48/17] 5
rubocop Lint Class has too many lines. [340/100] 2
rubocop Lint Cyclomatic complexity for results is too high. [8/7] 1
rubocop Lint Avoid parameter lists longer than 5 parameters. [6/5] 1
rubocop Lint Duplicate branch body detected. 1
rubocop Lint Rename has\_more? to more?. 1
qlty Structure Function with many parameters (count = 4): ensure_ordered_results 4
qlty Structure Function with high complexity (count = 6): ensure_ordered_results 2
qlty Structure Function with many returns (count = 4): exhaustion_reason 1

format.json { render json: { results: @results, pagination: @pagination, errors: @errors } }
format.turbo_stream { render :results }
format.html { render :results }
end

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 3 issues:

1. Assignment Branch Condition size for results is too high. [<6, 33, 8> 34.48/17] [rubocop:Metrics/AbcSize]


2. Cyclomatic complexity for results is too high. [8/7] [rubocop:Metrics/CyclomaticComplexity]


3. Method has too many lines. [25/10] [rubocop:Metrics/MethodLength]

@results = NormalizeTimdexResults.new(raw_results, @enhanced_query[:q]).normalize
@filters = extract_filters(response)
@append_results = @results
@load_more = load_more_from_pagination(@pagination)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 2 issues:

1. Assignment Branch Condition size for load_geodata_results is too high. [<10, 14, 2> 17.32/17] [rubocop:Metrics/AbcSize]


2. Method has too many lines. [11/10] [rubocop:Metrics/MethodLength]

@errors = data[:errors]
@pagination = data[:pagination]
@show_primo_continuation = data[:show_primo_continuation]
@load_more = data[:load_more]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 2 issues:

1. Assignment Branch Condition size for load_all_results is too high. [<14, 14, 0> 19.8/17] [rubocop:Metrics/AbcSize]


2. Method has too many lines. [13/10] [rubocop:Metrics/MethodLength]

def combine_errors(*error_arrays)
all_errors = error_arrays.compact.flatten
all_errors.any? ? all_errors : nil
end

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Class has too many lines. [172/100] [rubocop:Metrics/ClassLength]

has_more: has_more?(state, display_count),
total_hits: total_hits
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 2 issues:

1. Assignment Branch Condition size for fetch is too high. [<9, 28, 4> 29.68/17] [rubocop:Metrics/AbcSize]


2. Method has too many lines. [22/10] [rubocop:Metrics/MethodLength]

].slice(offset, per_page) || []
{ results: results, hits: 4, errors: nil, show_continuation: false }
end
timdex_fetcher = lambda do |offset:, per_page:, query: nil|

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused block argument - query. If it's necessary, use _ or _query as an argument name to indicate that it won't be used. [rubocop:Lint/UnusedBlockArgument]

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making this change fails the tests. It is incorrect.

assert_equal 2, timdex_calls.length
test 'fetch stops when duplicate source chunks do not grow the ordered result set' do
calls = []
fetcher = lambda do |offset:, per_page:, query: nil|

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 2 issues:

1. Unused block argument - per_page. If it's necessary, use _ or _per_page as an argument name to indicate that it won't be used. [rubocop:Lint/UnusedBlockArgument]


2. Unused block argument - query. If it's necessary, use _ or _query as an argument name to indicate that it won't be used. [rubocop:Lint/UnusedBlockArgument]

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making this change fails the tests. It is incorrect.

called << [:timdex, offset, per_page]
test 'fetch defaults to 50 results per source when env var is not set' do
per_page_seen = []
fetcher = lambda do |offset:, per_page:, query: nil|

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 2 issues:

1. Unused block argument - offset. If it's necessary, use _ or _offset as an argument name to indicate that it won't be used. [rubocop:Lint/UnusedBlockArgument]


2. Unused block argument - query. If it's necessary, use _ or _query as an argument name to indicate that it won't be used. [rubocop:Lint/UnusedBlockArgument]

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making this change fails the tests. It is incorrect.

primo_all = (1..3).map { |i| { title: "P#{i}", score: 1.0 - (i * 0.01), api: 'primo', identifier: "p#{i}" } }
timdex_all = (1..5).map { |i| { title: "T#{i}", score: 1.0 - (i * 0.01), api: 'timdex', identifier: "t#{i}" } }

primo_fetcher = lambda do |offset:, per_page:, query: nil|

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused block argument - query. If it's necessary, use _ or _query as an argument name to indicate that it won't be used. [rubocop:Lint/UnusedBlockArgument]

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making this change fails the tests. It is incorrect.

{ results: primo_all.slice(offset, per_page) || [], hits: primo_all.length, errors: nil,
show_continuation: false }
end
timdex_fetcher = lambda do |offset:, per_page:, query: nil|

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused block argument - query. If it's necessary, use _ or _query as an argument name to indicate that it won't be used. [rubocop:Lint/UnusedBlockArgument]

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making this change fails the tests. It is incorrect.

@mitlib
mitlib temporarily deployed to timdex-ui-pi-reranker-msjxghak August 3, 2026 21:19 Inactive

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR replaces merged “all tab” pagination with a reranked, cached candidate pool (via the reranker gem) and updates the UI to use a Turbo Stream “Load more results” interaction across results pages.

Changes:

  • Introduces MergedSearchService state caching + reranking flow for the “all” tab, with new ENV controls (ALL_TAB_RESULTS_PER_SOURCE, ALL_TAB_SCORER, LOAD_MORE_MAX_RESULTS).
  • Switches results rendering from pagination to a shared “load more” control and Turbo Stream append/replace responses.
  • Removes MergedSearchPaginator and updates/expands tests to reflect the new behavior.

Reviewed changes

Copilot reviewed 19 out of 20 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
app/models/merged_search_service.rb Reworks all-tab orchestration to cache candidates, rerank with reranker, and serve stable “load more” slices.
app/controllers/search_controller.rb Adds Turbo Stream responses and load-more parameter handling; adapts all-tab to display-count based fetching.
app/views/search/results.html.erb Renders results via shared item partial + load-more control instead of pagination.
app/views/search/results.turbo_stream.erb Turbo Stream append/replace template for load-more updates.
app/views/search/results_geo.html.erb Updates geodata results to use load-more UI and results-list target.
app/views/search/results_geo.turbo_stream.erb Turbo Stream append/replace template for geodata load-more updates.
app/views/search/_load_more.html.erb New shared “Load more results” control for all tabs.
app/views/search/_result_item.html.erb New shared wrapper partial to render either Primo or TIMDEX result markup.
app/javascript/loading_spinner.js Extends spinner behavior to cover Turbo Stream load-more interactions.
app/assets/stylesheets/partials/_results.scss Adjusts list styling to support numbered result lists.
app/assets/stylesheets/partials/_pagination.scss Adds styling for the new .load-more control.
Gemfile Adds reranker gem from GitHub tag.
Gemfile.lock Locks reranker and enumerable-stats.
README.md Documents new ENV variables for all-tab reranking and load-more behavior.
AGENTS.md Updates internal agent guidance to reflect reranking + load-more approach.
.env.test Adds test ENV defaults for reranking and load-more.
test/models/merged_search_service_test.rb Replaces paginator-oriented tests with reranking/state-cache/load-more tests.
test/models/merged_search_paginator_test.rb Removes tests for deleted paginator.
test/controllers/search_controller_test.rb Updates assertions for load-more UI and controller delegation changes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread app/models/merged_search_service.rb
Comment thread README.md Outdated
Comment thread app/javascript/loading_spinner.js
@JPrevost JPrevost changed the title Reranker Zipper via Reranker. Load more. Result numbers. Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants