Repository navigation
store: paginate catalogue sync, enable progressive UI population, and add 1-hour cache - #1022
Merged
JingMatrix merged 9 commits intoNov 15, 2026
Merged
Conversation
mubashardev
force-pushed
the
feat/store-paginated-sync
branch
from
October 20, 2026 09:37
d344aee to
647a8f3
Compare
JingMatrix
force-pushed
the
feat/store-paginated-sync
branch
from
November 10, 2026 13:30
647a8f3 to
18fd114
Compare
The premise this branch started from holds only in part. The site does answer modules.json with a 403, and module-list/<n>.json is current to the minute. But the mirror is not an archive: it is regenerated daily and runs about three days behind, which is all that keeps a new release out of the Store, and it is the only source whose entries carry a release with its APK. A page entry names latestRelease and nothing else, so a catalogue read from the pages alone left releasesOn() empty for every module: the Modules screen's update sheet could tick nothing, and a module's own sheet said "This release has no APK to install" of a release that has one. So the mirror stays the base and the pages correct it. They are sorted newest release first and the mirror is only ever behind, so what it lacks sits at the front: StoreApi.catalog() reads until a page agrees with the mirror entirely, five pages today out of 45. A corrected entry drops the mirror's releases instead of keeping an APK that is no longer the version it names, and RepoRepository fetches the detail record for the installed ones, which is what the two sheets read. A page corrects an entry only where what it states is no older than what the mirror states. That the mirror is the one behind is what was measured, not what is promised: a site that stopped being rebuilt would otherwise take back every version the mirror had learnt since. Pull-to-refresh still reads to the last page, and so does a start with the mirror out of reach. Those read every page after the first at once: the site spends a third of a second on each, which in a row was fourteen seconds under the spinner and together is about one. The count of pages is the site's to state, so it is bounded before it decides how many requests go out. refresh() replaced the catalogue with page one before reading the rest, so every pull collapsed the list to thirty rows and took the update badges of the Modules screen with it. Only the mirror's list is now published early, and only onto an empty screen. One page failing left a short catalogue marked as loaded, indistinguishable from a whole one and kept until the next pull; a read that breaks off without the mirror behind it is now a failure, and falls to the cache. Publishing the mirror's list first has a cost the screen has to answer. What the site then adds is the newest releases, which sort above everything showing, and a LazyColumn keeps its first visible row where it is when rows arrive above it. A reader who opened the Store while it loaded was left looking at the mirror's newest, with today's releases out of sight above. RepoScreen now asks for the top again whenever the first row changes under a list that has not been scrolled. That cache could not serve the pages. The site marks them must- revalidate, under which OkHttp refuses a FORCE_CACHE read however stale the caller allows, so the replay an offline start depends on came back empty. OkHttp also stores a 404 under the same key as the 200 before it, so a retired host would overwrite the last good catalogue with its error page. A network interceptor on the Store's client relaxes the one and keeps the other out. With the mirror's snapshot still fresh on disk and the site unreachable, the pages saved beside it are used, provided they were read no earlier than it was. The one-hour gate was a timestamp beside the catalogue; it is now read off loadedAtMillis, at the mirror's own ten minutes. It is still needed, because the pages never answer 304 and an unforced refresh runs on every launch. A beta published with no stable release beside it changes no latestRelease, so correct() compares latestBetaReleaseTime as well, which is all a page says of a beta. It cannot name the new one, so the entry is left naming none, and the detail record names it. releasesOn() returned a detail record's releases in the order they came on the prerelease channel. A detail record has no betaReleases: its betas sit in releases, flagged, and the whole list is by date, while latestOn() names the highest code. For io.github.wumisc.antiburn, stable code 10 published after beta code 45, the offer named one release and the first element was another. That channel is now ordered by code whichever shape the payload has. Also here: the progress display is gone. With the pages read together there is nothing left to step through, its label was English text in the composable, which the translation gate does not see, and the filter button it disabled had no reason to be. A detail record is accepted only when it names the package it was asked for. One gap: such a beta is listed by its stable release's date, so an unforced refresh, which stops at the first page that agrees, sees it only if it falls in the pages read. A pull finds it. Reported in discussion JingMatrix#1082.
LSPatch hosts the same Store screen and reads the same two hosts, and did so through a copy of RepoRepository made by hand before the site's pages were read at all. So it still showed the mirror alone, three days behind, asked the mirror first for a module's detail, and never replayed the cache its client has. Every fix to this code had to be made twice, and had been made once. StoreApi moves to manager-ui, which LSPatch already depends on for the screen itself, and the rest of RepoRepository moves with it as StoreRepository: the refresh lock, the ten-minute gate, the replay against the cache and the detail records for installed modules. None of that is about the app. What is, arrives as two arguments. How the device is asked what is installed is a function, which for Vector is the daemon's getInstalledPackagesFromAllUsers and is all that RepoRepository now holds. Where a log line goes is a StoreLog, because Vector's lines have to pass through logI and logW to reach the Verbose tab, and manager-ui cannot see those. Nothing changes in what Vector does.
JingMatrix
force-pushed
the
feat/store-paginated-sync
branch
from
November 10, 2026 16:46
18fd114 to
ce2cb33
Compare
correct() decides per channel whether the page or the base states the later release, then took both channels' fields from the page whichever answer it got. A page ahead on the beta and behind on the stable — a beta published since the rebuild, a stable release the page has yet to name — would carry its older latestRelease into the entry, which is the regression the >= guard on latestReleaseTime is there to refuse, and would drop the releases that install the newer one. A page stating an older beta time would take that too, leaving the entry naming a beta whose time no longer matches it, a pair complete() refuses to lay a record over, so an installed module stays without an APK until something else changes its name. Each channel now keeps what it holds unless this page states something newer on it. Both release lists still go together, since a detail record carries both. Also here: the Store's comments lose what cannot be checked and what dates. The site does not mark its index pages must-revalidate, and nothing either host serves does, so the interceptor is described by the response it answers rather than by a host that sends one. Entry counts, null-field tallies, asset censuses, page timings and the modules named for an odd release history are gone with them; the payload shapes they illustrated are what the code reads, and those are stated on their own. StoreDataSource no longer says it is where the two apps' network code differs, which StoreRepository ended. One gap: nothing the two hosts serve today reaches either branch of the first paragraph, which is why no report names it.
LxyaoX
added a commit
to LxyaoX/Aevix
that referenced
this pull request
Nov 15, 2026
Ports JingMatrix/Vector#1022 (`6f6b976e`), which rewrote where the Store's catalogue comes from. The two hosts answer different questions and neither is enough alone: * the mirror, `backup.modules.lsposed.org`, serves `modules.json` in one cacheable response, each entry carrying its newest release with the APK — but it is a periodic rebuild, so a release published since the last one is missing; * the site, `modules.lsposed.org`, answers that path with a 403 and serves the index only as `module-list/<n>.json`, a page at a time, newest first and current — but those entries name `latestRelease` and carry no release, scope or collaborators, and every page answers a conditional request with the whole page. So the mirror is the base and the site corrects it, and the walk stops at the first page that agrees with the mirror entirely. An entry the site knows better loses its now-outdated releases rather than keep an APK that is no longer the one its version names; `StoreRepository` fetches the detail record for the installed ones so the Store never offers an update it cannot install. Structure: the network side moves out of `manager` and into `manager-ui`, behind `StoreRepository`, with `StoreLog` and a suspend function for the installed map as the only things a host supplies. Aevix's `RepoRepository` is now 68 lines: the daemon as the source of installed versions, and the manager's log routing. Two behaviour changes follow the payload rather than the old shape: * `releasesOn` no longer returns early when `betaReleases` is empty. A detail record keeps its prereleases in `releases` with `isPrerelease` set, so an empty beta array says nothing. * the list keeps its first visible row at the top while newer rows arrive above it, which is what the site's corrections look like. Signed-off-by: LxyaoX <275498321+LxyaoX@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background and Endpoint Status
The current store implementation queries the monolithic catalogue at:
https://modules.lsposed.org/modules.jsonThis endpoint now returns HTTP 403 Forbidden. As a result, the client permanently falls back to:
https://backup.modules.lsposed.org/modules.jsonHowever, this backup host serves an archived, static snapshot of the catalogue. It does not reflect new module submissions or new module versions.
The live LSPosed repository infrastructure exposes two active endpoints:
GET https://modules.lsposed.org/module-list/{page}.jsonServes the complete module catalogue in chunks of ~30 modules per page (currently 40 pages, 1,180+ active modules), including fields for
page,pageCount, andtotal.GET https://modules.lsposed.org/module/{packageName}.jsonServes full release notes, README HTML, and historical releases for an individual module.
Technical Changes
1. Endpoint Architecture and Mirror Routing (
RepoRepository.kt)https://modules.lsposed.org/module-list/{page}.json.DETAIL_MIRRORSsohttps://modules.lsposed.org/takes precedence overhttps://backup.modules.lsposed.org/, ensuring release changelogs and download assets are served from the primary host.https://backup.modules.lsposed.org/modules.jsonas an offline/legacy fallback if page 1 network requests fail.2. Sequential Background Sync Pipeline
pageCountin the background. Sequenced requests prevent mirror rate-limiting and ensure deterministic catalog accumulation.JsonParser.parseReaderinto structuredOnlineModuleentries, passing throughusable()to filter hidden entries and deduplicate across pages.3. Progressive UI Reactivity (
RepoViewModel.kt&RepoScreen.kt)_catalog.valueis emitted with updatedmodules, currentpage, andsyncProgress.RepoViewModelreacts todataSource.catalogvia Kotlin Coroutinescombine, mapping toallEntriesand applying search query filtering and sorting off the main thread (Dispatchers.Default). Newly ingested modules appear live in the list as each page arrives.4. UI Indicators and Interaction Locking
LinearProgressIndicator) and status label above the bottom navigation bar showing sync progress percentage and page count whilecatalog.isSyncingis active.enabled = !isSyncing) to prevent users from sorting partial datasets before all pages arrive.5. 1-Hour In-Memory Cache and Tab Navigation
lastFullSyncTimeMillisupon completion of full catalog sync.RepoRepository.refresh(), ifforce == falseand the existing catalogue is fully synced withinCATALOG_MAX_AGE_MINUTES(60 minutes), the network sync is skipped.force == true) bypasses the cache, enforcesCacheControl.FORCE_NETWORK, and resets the 1-hour window.