Skip to content

Pre-processing railway line grouping for low zooms - #1121

Merged
hiddewie merged 23 commits into
masterfrom
osm2pgsql-build
Oct 28, 2026
Merged

hiddewie merged 23 commits into
masterfrom
osm2pgsql-build

Conversation

@hiddewie

@hiddewie hiddewie commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

Apply osm2pgsql-dev/osm2pgsql#2482 to railway lines in low zooms

In particular the SQL approach could also be used for other SQL processing which is now in the import/update script.

Changes:

  • Create materialized view line_route_count storing the route count per railway line
  • Create materialized view railway_line_low_grouped storing the railway lines for low zooms, simplified and grouped by relevant fields for display. Only a single low zooms table used for all sources, allowing every source to read the same pre-grouped table. Worldwide the grouped table takes only 54MB, while the railway lines table takes 1460MB.
  • The tile views do not group nor simplify the results, making the query simpler and more performant.
  • Merge the two train protection sources for train protection and train protection under construction for low zooms.

Followups:

  • simplify low zoom bridges/tunnels rendering

Example

Source standard_railway_line_low, tile 3, 4, 2.

Before:

Sort  (cost=443035.72..443584.75 rows=219612 width=170) (actual time=2368.105..2368.979 rows=9665.00 loops=1)
  Sort Key: (max(l.rank))
  Sort Method: quicksort  Memory: 7757kB
"  Buffers: shared hit=428105 read=831, temp read=5388 written=12832"
  ->  HashAggregate  (cost=103947.00..423551.09 rows=219612 width=170) (actual time=1666.403..2358.563 rows=9665.00 loops=1)
"        Group Key: l.feature, l.ref, l.name, l.highspeed, l.radio"
        Planned Partitions: 64  Batches: 65  Memory Usage: 141243kB  Disk Usage: 62496kB
"        Buffers: shared hit=428105 read=831, temp read=5388 written=12832"
        ->  Hash Left Join  (cost=25.05..65617.84 rows=219612 width=260) (actual time=34.056..1228.410 rows=388647.00 loops=1)
              Hash Cond: (CASE WHEN (ARRAY[l.owner] <@ l.operator) THEN l.owner ELSE l.operator[1] END = ro.name)
              Buffers: shared hit=423018 read=831
              ->  Index Scan using railway_line_way_idx on railway_line l  (cost=0.41..61755.94 rows=218819 width=332) (actual time=33.450..717.215 rows=388647.00 loops=1)
                    Index Cond: (way && '0103000020110F00000100000005000000000000000000000094107C45F81B5341000000000000000093107C45F81B634194107C45F81B534193107C45F81B634194107C45F81B534194107C45F81B5341000000000000000094107C45F81B5341'::geometry)
"                    Filter: ((service IS NULL) AND (feature = ANY ('{rail,ferry}'::text[])) AND (state = 'present'::text) AND (usage = 'main'::text))"
                    Rows Removed by Filter: 1216846
                    Index Searches: 1
                    Buffers: shared hit=404986 read=831
              ->  Hash  (cost=14.25..14.25 rows=831 width=22) (actual time=0.567..0.568 rows=831.00 loops=1)
                    Buckets: 1024  Batches: 1  Memory Usage: 53kB
                    Buffers: shared hit=464
                    ->  Index Only Scan using railway_operator_name on railway_operator ro  (cost=0.28..14.25 rows=831 width=22) (actual time=0.047..0.333 rows=831.00 loops=1)
                          Heap Fetches: 831
                          Index Searches: 1
                          Buffers: shared hit=464
Planning Time: 1.076 ms
JIT:
  Functions: 22
"  Options: Inlining false, Optimization false, Expressions true, Deforming true"
"  Timing: Generation 4.680 ms (Deform 2.642 ms), Inlining 0.000 ms, Optimization 2.808 ms, Emission 39.977 ms, Total 47.465 ms"
Execution Time: 2389.652 ms

After:

Gather  (cost=1000.00..54935.88 rows=126859 width=120) (actual time=0.467..101.217 rows=122917.00 loops=1)
  Workers Planned: 2
  Workers Launched: 2
  Buffers: shared hit=6849
  ->  Parallel Seq Scan on railway_line_low_grouped l  (cost=0.00..41249.98 rows=52858 width=120) (actual time=0.075..40.502 rows=40972.33 loops=3)
        Filter: (way && '0103000020110F00000100000005000000000000000000000094107C45F81B5341000000000000000093107C45F81B634194107C45F81B534193107C45F81B634194107C45F81B534194107C45F81B5341000000000000000094107C45F81B5341'::geometry)
        Rows Removed by Filter: 46371
        Buffers: shared hit=6849
Planning Time: 0.313 ms
Execution Time: 107.332 ms

Note that this query queries a significant part of the railway lines table, so a sequential scan is done.

A similar location (center europe, zoom level 6, tile 6, 33, 21):

Index Scan using railway_line_low_grouped_way on railway_line_low_grouped l  (cost=0.28..14822.49 rows=21986 width=120) (actual time=0.173..13.094 rows=21928.00 loops=1)
  Index Cond: (way && '0103000020110F00000100000005000000A0107C45F81B2341B814DB56F6E25741A0107C45F81B2341CA968A5F75465A4190107C45F81B3341CA968A5F75465A4190107C45F81B3341B814DB56F6E25741A0107C45F81B2341B814DB56F6E25741'::geometry)
  Index Searches: 1
  Buffers: shared hit=866
Planning Time: 0.537 ms
Execution Time: 13.829 ms

This region is much smaller so uses a direct index scan with a query response within a few milliseconds.

@hiddewie hiddewie mentioned this pull request Oct 13, 2026
@hiddewie hiddewie changed the title Test for pre-processing railway line grouping for low zooms Pre-processing railway line grouping for low zooms Oct 20, 2026
@hiddewie

Copy link
Copy Markdown
Owner Author

Report back in osm2pgsql-dev/osm2pgsql#2508

@hiddewie
hiddewie marked this pull request as ready for review October 25, 2026 08:42
…to osm2pgsql-build

# Conflicts:
#	import/sql/tile_views.sql
@hiddewie
hiddewie merged commit 3e172f4 into master Oct 28, 2026
3 checks passed
@hiddewie
hiddewie deleted the osm2pgsql-build branch October 28, 2026 20:08
@hiddewie

Copy link
Copy Markdown
Owner Author

Disk read statistics look much better since release:
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant