Skip to content

Consolidate list page endpoints + better QA sorting #2417

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
Feb 21, 2025

Conversation

ikreymer
Copy link
Member

@ikreymer ikreymer commented Feb 20, 2025

  • consolidate list_pages() and list_replay_query_pages() into list_pages()
  • to keep backwards compatibility, add /pagesSearch that does not include page totals, keep /pages with page total (slower)
  • qa frontend: add default 'Crawl Order' sort order, to better show pages in QA view
  • bgjob: account for parallelism in bgjobs, add logging if succeeded mismatches parallelism

@ikreymer ikreymer requested a review from tw4l February 20, 2025 21:55
@ikreymer ikreymer changed the title consolidate list_pages() and list_replay_query_pages() Consolidate list page endpoints + better QA sorting Feb 20, 2025
Base automatically changed from issue-2406-crawl-migration-rework to main February 20, 2025 23:26
@ikreymer ikreymer force-pushed the consolidate-list-pages branch from ffb8301 to 39cb347 Compare February 20, 2025 23:31
@ikreymer
Copy link
Member Author

All tests now pass - also updated pageUrlCounts to also use list_pages() to avoid duplication.

@ikreymer ikreymer force-pushed the consolidate-list-pages branch from 5a39135 to f2661a6 Compare February 21, 2025 21:10
Copy link

socket-security bot commented Feb 21, 2025

👍 Dependency issues cleared. Learn more about Socket for GitHub ↗︎

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

View full report↗︎

@ikreymer ikreymer force-pushed the consolidate-list-pages branch from f2661a6 to 8d65f7e Compare February 21, 2025 21:19
Copy link

New, updated, and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package New capabilities Transitives Size Publisher
npm/@polka/url@1.0.0-next.24 None 0 4.38 kB lukeed
npm/@types/bonjour@3.5.13 None 0 5.57 kB types
npm/@types/connect-history-api-fallback@1.5.4 None 0 4.86 kB types
npm/@types/send@0.17.4 None 0 9.85 kB types
npm/@types/serve-index@1.9.4 None 0 4.71 kB types
npm/@types/sockjs@0.3.36 None 0 5.35 kB types
npm/acorn-walk@8.3.1 None 0 52.3 kB marijn
npm/ansi-html-community@0.0.8 None 0 20.1 kB mahdyar
npm/bonjour-service@1.3.0 None 0 66.6 kB mdidon
npm/bytes@3.0.0 None 0 10.8 kB dougwilson
npm/compressible@2.0.18 None 0 7.36 kB dougwilson
npm/compression@1.7.4 None 0 23.3 kB dougwilson
npm/connect-history-api-fallback@2.0.0 None 0 10.2 kB bripkens
npm/debounce@1.2.1 None 0 12 kB stephenmathieson
npm/duplexer@0.1.2 None 0 5.47 kB raynos
npm/express@4.21.2 environment, filesystem, network 0 221 kB jonchurch
npm/gzip-size@6.0.0 filesystem 0 7.93 kB sindresorhus
npm/html-escaper@2.0.2 None 0 13.1 kB webreflection
npm/http-proxy-middleware@2.0.7 network 0 64.3 kB chimurai
npm/ipaddr.js@2.2.0 🔁 npm/ipaddr.js@1.9.1 None 0 62.3 kB whitequark
npm/launch-editor@2.9.1 environment, filesystem, shell 0 17.1 kB soda
npm/mrmime@1.0.1 None 0 31 kB lukeed
npm/multicast-dns@7.2.5 network 0 22 kB mafintosh
npm/on-headers@1.0.2 None 0 7.54 kB dougwilson
npm/opener@1.5.2 shell 0 6.21 kB domenic
npm/p-retry@6.2.1 None 0 13.2 kB sindresorhus
npm/regenerator-runtime@0.14.0 None 0 27.8 kB benjamn
npm/selfsigned@2.4.1 None 0 25.4 kB jfromaniello
npm/serve-index@1.9.1 filesystem, network 0 93.4 kB dougwilson
npm/sirv@2.0.3 filesystem 0 20.7 kB lukeed
npm/sockjs@0.3.24 filesystem, network 0 82.8 kB brycekahle
npm/spdy@4.0.2 network 0 57.7 kB indutny
npm/totalist@3.0.1 filesystem 0 7.46 kB lukeed
npm/vscode-nls@4.1.2 environment, filesystem 0 26.1 kB dbaeumer
npm/vscode-uri@2.1.2 None 0 65.6 kB jrieken
npm/webpack-bundle-analyzer@4.10.1 environment, filesystem, network 0 1.23 MB valscion
npm/webpack-dev-middleware@7.4.2 filesystem +1 144 kB evilebottnawi
npm/webpack-dev-server@5.2.0 environment, eval, network 0 541 kB evilebottnawi

View full report↗︎

@ikreymer ikreymer merged commit 8a507f0 into main Feb 21, 2025
27 checks passed
@ikreymer ikreymer deleted the consolidate-list-pages branch February 21, 2025 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants