Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add initial pages + pagesQuery endpoint to /replay.json APIs #2380

Merged
merged 41 commits into from
Feb 14, 2025
Merged

Add initial pages + pagesQuery endpoint to /replay.json APIs #2380

merged 41 commits into from
Feb 14, 2025
+425 −184

Conversation

ikreymer
Copy link
Member

@ikreymer ikreymer commented Feb 11, 2025 •
edited by tw4l
Loading

Fixes #2360

  • Adds seedPages to /replay.json response for collections, returning upto 25 seed pages.
  • Adds pagesQueryUrl to /replay.json
  • Adds a public pages search endpoint to support public collections.
  • Adds preloadResources, including list of WACZ files that should always be loaded, to /replay.json

Draft pending work in wabac.js to ensure this is complete.

@ikreymer ikreymer requested a review from tw4l February 11, 2025 00:48
tw4l
tw4l reviewed Feb 11, 2025
View reviewed changes
tw4l
tw4l reviewed Feb 11, 2025
View reviewed changes
@tw4l
Copy link
Member

tw4l commented Feb 11, 2025

Do we also want to return seeds and the additional pages query url in the replay.json for crawls?

tw4l
tw4l reviewed Feb 11, 2025
View reviewed changes
ikreymer and others added 15 commits February 11, 2025 23:18
@ikreymer
add pages to coll replay.json endpoint
e5ee366
@ikreymer
create CollOut after
cb1715c
@ikreymer
add pagesQuery to replay.json

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
cbeb6e7
@ikreymer
update model
291dd1c
@ikreymer
add /public/pages endpoint for public collections
988cc18 
tests: add test 'pagesQuery' for both private and public collections
@ikreymer
work on tests
b7243f4
@ikreymer
test work
d878b53
@tw4l @ikreymer
Add initial URL + title search to collection pages endpoint
ade353d 
Needs some testing and possible refinement
@tw4l @ikreymer
Add alwaysLoad to collection replay.json
9c0c3b2 
Includes information for all WACZ files in collection that contain
seed pages or have no associated pages in the database.
@tw4l @ikreymer
Add pylint comment
964ef09
@ikreymer
crawlfileout: rename to itemId, use basename for 'name'
6552034
@ikreymer
switch to crawlId instead
477bb84
@ikreymer
handle CORS for public pages endpoint
ea776cd
@ikreymer
fix typo
b8a8c18
@ikreymer
refactor / cleanup from code review
Loading
Loading status checks…
1e1cf12 
- rename pages -> seedPages
- rename alwaysLoad -> preloadResources
- rename pagesQuery -> pagesQueryUrl
- optimize loading preloadResources as part of resource lookup
- remove seed page wacz files from preloadResources, can be computed from seedPages list
- tests: add additional tests for preloadResources, seedPages
@ikreymer
fix test?
Loading
Loading status checks…
4a6d655
@ikreymer
ensure public pages access to private collecton is 404
Loading
Loading status checks…
cb448a1
@ikreymer
add seedPages and pagesQueryUrl to crawl /replay.json for consistency
Loading
Loading status checks…
7ab0c51 
update tests
@ikreymer
fix typo
Loading
Loading status checks…
0e43c95
@ikreymer
Copy link
Member Author

Do we also want to return seeds and the additional pages query url in the replay.json for crawls?

Added pagesQueryUrl and seedPages for crawls.

ikreymer and others added 18 commits February 12, 2025 12:23
@ikreymer @tw4l
add pagesQuery to replay.json
b072934
@ikreymer @tw4l
update model
10e2721
@ikreymer @tw4l
add /public/pages endpoint for public collections
84bacbd 
tests: add test 'pagesQuery' for both private and public collections
@ikreymer @tw4l
work on tests
f1fdc5f
@ikreymer @tw4l
test work
05b714c
@tw4l
Add initial URL + title search to collection pages endpoint
79522b8 
Needs some testing and possible refinement
@tw4l
Add alwaysLoad to collection replay.json
a7b11df 
Includes information for all WACZ files in collection that contain
seed pages or have no associated pages in the database.
@tw4l
Add pylint comment
a80c52f
@ikreymer @tw4l
crawlfileout: rename to itemId, use basename for 'name'
d6800f7
@ikreymer @tw4l
switch to crawlId instead
fcdbc40
@ikreymer @tw4l
handle CORS for public pages endpoint
b965004
@ikreymer @tw4l
fix typo
2690b2a
@ikreymer @tw4l
refactor / cleanup from code review
09d1895 
- rename pages -> seedPages
- rename alwaysLoad -> preloadResources
- rename pagesQuery -> pagesQueryUrl
- optimize loading preloadResources as part of resource lookup
- remove seed page wacz files from preloadResources, can be computed from seedPages list
- tests: add additional tests for preloadResources, seedPages
@ikreymer @tw4l
fix test?
529f260
@ikreymer @tw4l
ensure public pages access to private collecton is 404
e394646
@ikreymer @tw4l
add seedPages and pagesQueryUrl to crawl /replay.json for consistency
2f27531 
update tests
@ikreymer @tw4l
fix typo
e9d6c9e
@tw4l
Add tests for collection pages search filter
Loading
Loading status checks…
dd664cb
@tw4l tw4l force-pushed the replay-json-pages branch from 61d0a15 to dd664cb Compare February 12, 2025 17:23
@ikreymer ikreymer added this to the Public Collections milestone Feb 13, 2025
@ikreymer ikreymer marked this pull request as ready for review February 13, 2025 04:21
@ikreymer
Copy link
Member Author

One more small change, what if instead of seedPages we went with initialPages and it included the first 25 pages?
The default sort order can be {"$sort": {"isSeed": -1, "ts": 1}}, so we still list all seeds, then all non-seeds, sorted by time. This would allow for infinite scroll to load additional pages w/o a query.

@ikreymer
seedPages -> initialPages
bfb7076 
add totalPages
collection pages: default sort order seeds first, then by timestamp
@ikreymer
Merge branch 'replay-json-pages' of github.com:webrecorder/browsertri…
Loading
Loading status checks…
a6d848e 
…x-cloud into replay-json-pages
@ikreymer ikreymer merged commit 7b2932c into main Feb 14, 2025
23 checks passed
@ikreymer ikreymer deleted the replay-json-pages branch February 14, 2025 00:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tw4l tw4l

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Public Collections Release (1.14)
Development

Successfully merging this pull request may close these issues.

[Feature]: Add initial list of pages + search endpoint to /replay.json APIs
2 participants
@ikreymer @tw4l