Skip to content

store: paginate catalogue sync, enable progressive UI population, and add 1-hour cache - #1022

Merged
JingMatrix merged 9 commits into
JingMatrix:masterfrom
mubashardev:feat/store-paginated-sync
Nov 15, 2026
Merged

JingMatrix merged 9 commits into
JingMatrix:masterfrom
mubashardev:feat/store-paginated-sync

Conversation

@mubashardev

@mubashardev mubashardev commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Background and Endpoint Status

The current store implementation queries the monolithic catalogue at:
https://modules.lsposed.org/modules.json

This endpoint now returns HTTP 403 Forbidden. As a result, the client permanently falls back to:
https://backup.modules.lsposed.org/modules.json

However, this backup host serves an archived, static snapshot of the catalogue. It does not reflect new module submissions or new module versions.

The live LSPosed repository infrastructure exposes two active endpoints:

  1. Paginated Catalogue Index:
    GET https://modules.lsposed.org/module-list/{page}.json
    Serves the complete module catalogue in chunks of ~30 modules per page (currently 40 pages, 1,180+ active modules), including fields for page, pageCount, and total.
  2. Module Detail and Releases:
    GET https://modules.lsposed.org/module/{packageName}.json
    Serves full release notes, README HTML, and historical releases for an individual module.

Technical Changes

1. Endpoint Architecture and Mirror Routing (RepoRepository.kt)

  • Primary Catalogue Endpoint: Replaced the blocked monolithic URL with sequential calls to https://modules.lsposed.org/module-list/{page}.json.
  • Detail Mirror Precedence: Updated DETAIL_MIRRORS so https://modules.lsposed.org/ takes precedence over https://backup.modules.lsposed.org/, ensuring release changelogs and download assets are served from the primary host.
  • Fallbacks: Retained https://backup.modules.lsposed.org/modules.json as an offline/legacy fallback if page 1 network requests fail.

2. Sequential Background Sync Pipeline

  • Initial Page Fast Path: Fetches page 1 synchronously on the IO dispatcher so the first ~30 modules render immediately when opening the Store tab without waiting for the full repository.
  • Non-blocking Sequential Ingestion: Dispatches a sequential loop for pages 2 through pageCount in the background. Sequenced requests prevent mirror rate-limiting and ensure deterministic catalog accumulation.
  • Stream Parsing and Memory Efficiency: Parses the paginated JSON body through JsonParser.parseReader into structured OnlineModule entries, passing through usable() to filter hidden entries and deduplicate across pages.

3. Progressive UI Reactivity (RepoViewModel.kt & RepoScreen.kt)

  • As each page completes, _catalog.value is emitted with updated modules, current page, and syncProgress.
  • RepoViewModel reacts to dataSource.catalog via Kotlin Coroutines combine, mapping to allEntries and applying search query filtering and sorting off the main thread (Dispatchers.Default). Newly ingested modules appear live in the list as each page arrives.
  • Client-side Filtering & Sorting: With all active modules populated directly into the repository catalogue, search filtering and sorting (recently updated, name, and star count) evaluate cleanly against the comprehensive in-memory model without requiring separate search index endpoints.

4. UI Indicators and Interaction Locking

  • Added a sync indicator (LinearProgressIndicator) and status label above the bottom navigation bar showing sync progress percentage and page count while catalog.isSyncing is active.
  • Filter and sort selection is disabled while sync is in progress (enabled = !isSyncing) to prevent users from sorting partial datasets before all pages arrive.

5. 1-Hour In-Memory Cache and Tab Navigation

  • Tracked lastFullSyncTimeMillis upon completion of full catalog sync.
  • In RepoRepository.refresh(), if force == false and the existing catalogue is fully synced within CATALOG_MAX_AGE_MINUTES (60 minutes), the network sync is skipped.
  • This eliminates unnecessary automatic resyncs and progress bar flashes when switching between bottom navigation tabs.
  • Manual pull-to-refresh (force == true) bypasses the cache, enforces CacheControl.FORCE_NETWORK, and resets the 1-hour window.

@mubashardev
mubashardev force-pushed the feat/store-paginated-sync branch from d344aee to 647a8f3 Compare October 20, 2026 09:37
@mubashardev mubashardev changed the title store: paginate catalogue sync and update mirror endpoints store: paginate catalogue sync, enable progressive UI population, and add 1-hour cache Oct 20, 2026
@JingMatrix
JingMatrix force-pushed the feat/store-paginated-sync branch from 647a8f3 to 18fd114 Compare November 10, 2026 13:30
The premise this branch started from holds only in part. The site does
answer modules.json with a 403, and module-list/<n>.json is current to
the minute. But the mirror is not an archive: it is regenerated daily
and runs about three days behind, which is all that keeps a new release
out of the Store, and it is the only source whose entries carry a
release with its APK. A page entry names latestRelease and nothing else,
so a catalogue read from the pages alone left releasesOn() empty for
every module: the Modules screen's update sheet could tick nothing, and
a module's own sheet said "This release has no APK to install" of a
release that has one.

So the mirror stays the base and the pages correct it. They are sorted
newest release first and the mirror is only ever behind, so what it
lacks sits at the front: StoreApi.catalog() reads until a page agrees
with the mirror entirely, five pages today out of 45. A corrected entry
drops the mirror's releases instead of keeping an APK that is no longer
the version it names, and RepoRepository fetches the detail record for
the installed ones, which is what the two sheets read.

A page corrects an entry only where what it states is no older than what
the mirror states. That the mirror is the one behind is what was
measured, not what is promised: a site that stopped being rebuilt would
otherwise take back every version the mirror had learnt since.

Pull-to-refresh still reads to the last page, and so does a start with
the mirror out of reach. Those read every page after the first at once:
the site spends a third of a second on each, which in a row was fourteen
seconds under the spinner and together is about one. The count of pages
is the site's to state, so it is bounded before it decides how many
requests go out.

refresh() replaced the catalogue with page one before reading the rest,
so every pull collapsed the list to thirty rows and took the update
badges of the Modules screen with it. Only the mirror's list is now
published early, and only onto an empty screen. One page failing left a
short catalogue marked as loaded, indistinguishable from a whole one and
kept until the next pull; a read that breaks off without the mirror
behind it is now a failure, and falls to the cache.

Publishing the mirror's list first has a cost the screen has to answer.
What the site then adds is the newest releases, which sort above
everything showing, and a LazyColumn keeps its first visible row where
it is when rows arrive above it. A reader who opened the Store while it
loaded was left looking at the mirror's newest, with today's releases
out of sight above. RepoScreen now asks for the top again whenever the
first row changes under a list that has not been scrolled.

That cache could not serve the pages. The site marks them must-
revalidate, under which OkHttp refuses a FORCE_CACHE read however stale
the caller allows, so the replay an offline start depends on came back
empty. OkHttp also stores a 404 under the same key as the 200 before it,
so a retired host would overwrite the last good catalogue with its error
page. A network interceptor on the Store's client relaxes the one and
keeps the other out. With the mirror's snapshot still fresh on disk and
the site unreachable, the pages saved beside it are used, provided they
were read no earlier than it was.

The one-hour gate was a timestamp beside the catalogue; it is now read
off loadedAtMillis, at the mirror's own ten minutes. It is still needed,
because the pages never answer 304 and an unforced refresh runs on every
launch.

A beta published with no stable release beside it changes no
latestRelease, so correct() compares latestBetaReleaseTime as well,
which is all a page says of a beta. It cannot name the new one, so the
entry is left naming none, and the detail record names it.

releasesOn() returned a detail record's releases in the order they came
on the prerelease channel. A detail record has no betaReleases: its
betas sit in releases, flagged, and the whole list is by date, while
latestOn() names the highest code. For io.github.wumisc.antiburn, stable
code 10 published after beta code 45, the offer named one release and
the first element was another. That channel is now ordered by code
whichever shape the payload has.

Also here: the progress display is gone. With the pages read together
there is nothing left to step through, its label was English text in the
composable, which the translation gate does not see, and the filter
button it disabled had no reason to be. A detail record is accepted only
when it names the package it was asked for.

One gap: such a beta is listed by its stable release's date, so an
unforced refresh, which stops at the first page that agrees, sees it
only if it falls in the pages read. A pull finds it.

Reported in discussion JingMatrix#1082.
LSPatch hosts the same Store screen and reads the same two hosts, and
did so through a copy of RepoRepository made by hand before the site's
pages were read at all. So it still showed the mirror alone, three days
behind, asked the mirror first for a module's detail, and never
replayed the cache its client has. Every fix to this code had to be
made twice, and had been made once.

StoreApi moves to manager-ui, which LSPatch already depends on for the
screen itself, and the rest of RepoRepository moves with it as
StoreRepository: the refresh lock, the ten-minute gate, the replay
against the cache and the detail records for installed modules. None of
that is about the app. What is, arrives as two arguments. How the
device is asked what is installed is a function, which for Vector is
the daemon's getInstalledPackagesFromAllUsers and is all that
RepoRepository now holds. Where a log line goes is a StoreLog, because
Vector's lines have to pass through logI and logW to reach the Verbose
tab, and manager-ui cannot see those.

Nothing changes in what Vector does.
@JingMatrix
JingMatrix force-pushed the feat/store-paginated-sync branch from 18fd114 to ce2cb33 Compare November 10, 2026 16:46
correct() decides per channel whether the page or the base states the
later release, then took both channels' fields from the page whichever
answer it got. A page ahead on the beta and behind on the stable — a
beta published since the rebuild, a stable release the page has yet to
name — would carry its older latestRelease into the entry, which is the
regression the >= guard on latestReleaseTime is there to refuse, and
would drop the releases that install the newer one. A page stating an
older beta time would take that too, leaving the entry naming a beta
whose time no longer matches it, a pair complete() refuses to lay a
record over, so an installed module stays without an APK until
something else changes its name. Each channel now keeps what it holds
unless this page states something newer on it. Both release lists still
go together, since a detail record carries both.

Also here: the Store's comments lose what cannot be checked and what
dates. The site does not mark its index pages must-revalidate, and
nothing either host serves does, so the interceptor is described by the
response it answers rather than by a host that sends one. Entry counts,
null-field tallies, asset censuses, page timings and the modules named
for an odd release history are gone with them; the payload shapes they
illustrated are what the code reads, and those are stated on their own.
StoreDataSource no longer says it is where the two apps' network code
differs, which StoreRepository ended.

One gap: nothing the two hosts serve today reaches either branch of the
first paragraph, which is why no report names it.
@JingMatrix
JingMatrix merged commit 6f6b976 into JingMatrix:master Nov 15, 2026
1 check passed
LxyaoX added a commit to LxyaoX/Aevix that referenced this pull request Nov 15, 2026
Ports JingMatrix/Vector#1022 (`6f6b976e`), which rewrote where the Store's catalogue comes from.
The two hosts answer different questions and neither is enough alone:

* the mirror, `backup.modules.lsposed.org`, serves `modules.json` in one cacheable response, each
  entry carrying its newest release with the APK — but it is a periodic rebuild, so a release
  published since the last one is missing;
* the site, `modules.lsposed.org`, answers that path with a 403 and serves the index only as
  `module-list/<n>.json`, a page at a time, newest first and current — but those entries name
  `latestRelease` and carry no release, scope or collaborators, and every page answers a
  conditional request with the whole page.

So the mirror is the base and the site corrects it, and the walk stops at the first page that
agrees with the mirror entirely. An entry the site knows better loses its now-outdated releases
rather than keep an APK that is no longer the one its version names; `StoreRepository` fetches the
detail record for the installed ones so the Store never offers an update it cannot install.

Structure: the network side moves out of `manager` and into `manager-ui`, behind
`StoreRepository`, with `StoreLog` and a suspend function for the installed map as the only things
a host supplies. Aevix's `RepoRepository` is now 68 lines: the daemon as the source of installed
versions, and the manager's log routing.

Two behaviour changes follow the payload rather than the old shape:

* `releasesOn` no longer returns early when `betaReleases` is empty. A detail record keeps its
  prereleases in `releases` with `isPrerelease` set, so an empty beta array says nothing.
* the list keeps its first visible row at the top while newer rows arrive above it, which is what
  the site's corrections look like.

Signed-off-by: LxyaoX <275498321+LxyaoX@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants