Vector Tile styles require icons are served in a sprite sheet. This contains all of the icons in one file. Years ago there were a few options for these, none of them great. These days, there are three common options: spreet, @basemaps/sprites, and sprite-one. The first is written in Rust while the other two are written in Javascript.

All have the same basic functionality of turning a folder of SVGs into a json+png spritesheet, and doing so at multiple resolutions. Spreet has the additional option of de-duplicating icons. If two icons are identical it will only put one copy in the PNG and reference the same image twice.

I benchmarked all three options with two sets of sprites: all the SVGs from OpenStreetMap Carto, and the OpenStreetMap Americana icons. The former is 973 icons and the latter is 248 icons. These are larger than a typical set of icons but are a good test.

	Test	spreet	sprite-one	@basemaps/sprites
osm-carto SVGs	@1x pixels	4194304	4078074	8339456
osm-carto SVGs	@1x bytes	513159	763531	837792
osm-carto SVGs	@1x bytes after oxipng	474750	649894	706845
osm-carto SVGs	@2x bytes	1442588	2176457	2411489
osm-carto SVGs	@2x bytes after oxipng	1325707	1896784	2088729
osm-americana	@1x pixels	128265	122400	151760
osm-americana	@1x bytes	75749	91870	92066
osm-americana	@1x bytes after oxipng	75497	84986	85828
osm-americana	@2x bytes	136177	213876	210650
osm-americana	@2x bytes after oxipng	132462	197687	194950

… 查看全文

A style using the Shortbread vector tiles is now live on OpenStreetMap.org. There’s been some minor issues with the roll-out, but nothing that caused me to panic. I’ve been taking a couple of days to unwind and relax, as this has been a major milestone.

A feature like this includes work across multiple projects, so people have been asking where to report issues.

1. Any issues that are present on the OpenStreetMap website but not on the vector demo page should be reported to openstreetmap-website. Some issues reported have been with attribution controls and strange panning at extremely high zooms

2. Style issues (e.g. colors, symbology, fonts, etc) are handled by the VersaTiles Colorful and VersaTiles Eclipse styles. Both can be found in a VersaTiles repository.

3. What is supposed to be in the tiles is defined in the Shortbread Vector Tile Schema 1.0. There are lists of what features should appear, and where they should appear.

4. The code responsible for what is actually in the tiles is contained in the Street Spirit repo. If something is missing that is supposed to be in Shortbread, this needs fixing. This also handles simplifcation and other parts of generating the tiles.

5. The software that serves the tiles is Tilekiln. If the wrong HTTP headers are being sent, this is where you report it.

6. The chef code pulls everything together on the OSMF servers. It’s unlikely you should report an issue directly here.

7. Some font rendering issues are part of MapLibre GL JS. This is most likely to come up in Southeast Asia.

Just a quick blog post on some coastline work I was doing.

For the OSMF Shortbread vector tiles I had to identify when coastlines has changed. The solution I came up isn’t specific to Shortbread, but is useful for anyone using the tiled ocean shapefiles.

I’m going to start by assuming that the old ocean data is in water_polygons and the new data is in loading.water_polygons. Other parts of my code already handle this. The shapefiles are loaded into tables that have the columns x int, y int, way geometry.

To start I want to find any geometries that have changed. For geometries in the new data that aren’t in the old data, I can get this with a LEFT JOIN. I want a set of geometries that includes any geometries from the new data that aren’t in the old. This set can be made by excluding any geometries in the old data that have identical x, y, and binary identical geometry to a new geometry. It’s possible this set includes extra geometries, but that’s okay.

A RIGHT JOIN would find geometries in the old data that aren’t in the new. Combining these gives a FULL OUTER JOIN. If I then collect the geometries in each shapefile tile I can compare them to find the geometries

SELECT ST_SymDifference(ST_Collect(old.way), ST_Collect(new.way)) AS dgeom
    FROM water_polygons AS old
    FULL OUTER JOIN loading.water_polygons AS new
        ON old.way = new.way and old.x = new.x and old.y = new.y
    WHERE new.way IS NULL OR old.way IS NULL
    GROUP BY COALESCE(old.x, new.x), COALESCE(old.y, new.y)

This gets me the difference in geometries for the entire world in about two minutes. But I need tiles, which is it’s own complication.

… 查看全文

Load testing vector tiles

As part of bringing the new vector tile servers into production, I had to benchmark their performance. Since there’s a cache in front of the servers, it’s challenging to benchmark them accurately. Although we’ve never had a heavy load on the vector tile servers, we’ve been running raster tile servers for years.

All tile requests on the standard layer are logged, and from those logs, I can generate a list of tiles to benchmark the vector tile servers. The logs are stored as Parquet files, which I query using Amazon Athena, a hosted Presto database.

Vector tiles and raster tiles typically have different scales at the same zoom level. To convert raster tile requests to equivalent vector tile requests, I divide the x and y coordinates by 2 and decrease the zoom level by 1. I also skip zoom 0 raster tile requests to simplify the process, as these don’t affect performance since zoom 0 is always cached.

The OSMF shortbread tiles have a maximum zoom of 14. Lower scales (higher zoom levels) are achieved by overzooming on the client side. Requests from zoom 1 to 15 should have their zoom level lowered by 1. Requests from zoom 16 to 19 need their zoom level decreased by the difference between their level and 14. I divide the x and y coordinates by 2 the appropriate number of times to match the new zoom level.

Filtering to have only cache misses gets me a request list on the backend servers.

SELECT 
    CASE WHEN z > 15 THEN 14 ELSE z - 1 END AS v_z,
    bitwise_right_shift(x, CASE WHEN z > 15 THEN z-14 ELSE 1 END) AS v_x,
    bitwise_right_shift(x, CASE WHEN z > 15 THEN z-14 ELSE 1 END) AS v_y
    z, x, y
FROM fastly_success_logs_v1
WHERE year=2025 AND month=5 AND day = 1 AND hour = 1
    AND z >= 1
    AND cachehit = 'MISS';

Unfortunately, this is the wrong list.

… 查看全文

It’s difficult to write in all map style languages. A style written in JSON, like MapLibre, has a few extra pain points because JSON is not designed for editing by humans.

Some “common” style languages are

• CartoCSS
• Mapnik XML
• MapCSS
• MapServer
• MapLibre GL/Mapbox GL

Some, like CartoCSS, are designed for human editing, while others, like Mapnik XML, serve as a lower-level language. MapLibre GL falls into this category of not being designed for editing by humans. MapLibre GL preprocessors like glug were designed to help with this, but none of them have taken off. Other style projects like openstreetmap-americana have taken a different route. Their developers have written a program in JavaScript that generates the style.

I’m taking a different route. I’m creating a language that uses minimal pre-processing of its input to produce MapLibre GL. I don’t aim to solve every difficulty with MapLibre GL, only the ones that impact me the most. The end result will be a pre-processing language

The biggest problems I encounter when writing MapLibre GL are

1. No comments

   Comments are essential so other readers understand what’s written

2. Everything has to be in one file.

   With large styles this is a burden. More than one file makes it easier to edit.

3. Having to repeat definitions instead of using a variable.

   Something like a color or symbol definition might appear a dozen times in the style. If you want to change it, you need to make sure you got all the occurrences.

4. Inability to make versions of the style in different colors.

   When you only want to change a few superficial elements of the style, you want to contain those changes to one file.

5. Not having support for more colorspaces

   I work in perceptual colorspaces like Lch. It’s a lot of converting that the computer should automate.

What issues have you found when writing MapLibre GL styles?

This blog post explains how I handle a typical bug report for the new OSMF Shortbread tiles. Here, I focus on the “island” seems to be missing from “place_labels” report from SomeoneElse

After verifying that the report is correct, I set up my editor environment. It’s useful to have an environment that syntax highlights Jinja SQL files, as well as other files. I use a Visual Studio Code-based editor with the Better Jinja plugin.

The issue is in the place_labels layer. After checking Shortbread, I see that place=island should show at zoom 10 or higher, so there is a bug. Tilekiln creates tiles by reading definitions from shortbread.yaml, so I check there for the place_labels definition.

place_labels:
    description: Holds label points for populated places.
    fields:
        kind: Value of OSM place tag
        name: *name
        name_en: *name_en
        name_de: *name_de
        population: Value of OSM population tag
    sql:
    - minzoom: 4
        maxzoom: 14
        file: shortbread_original/place_labels.04-14.sql.jinja2

This file shows that for zooms 4 to 14, the SQL for the layer is in shortbread_original/place_labels.04-14.sql.jinja2. Since this file is in shortbread_original, osm2pgsql-themepark created it, and it remains unchanged.

SELECT
        ST_AsMVTGeom(geom, {{unbuffered_bbox}}, {{extent}}, {{buffer}}) AS way,
        name,
        name_de,
        name_en,
        kind,
        population
    FROM place_labels
    WHERE geom && {{bbox}}
        AND {{zoom}} >= minzoom
    ORDER BY population desc

There aren’t any obvious bugs in the SQL. There’s no filtering out of islands, so either the data isn’t making it into the place_labels table or it has the wrong zoom. The data is loaded by osm2pgsql, and shortbread.lua tells osm2pgsql how to do that.

themepark:add_topic('shortbread/places')

… 查看全文

I’ve put up a demo page showing my work on minutely updated vector tiles. This demo is using my work for the tiles and the Versatiles Colorful stylesheet.

With this year being the year of OpenStreetMap vector maps I’ve been working on making vector tile maps that update minutely. Most maps don’t need minutely updates and are fine with daily or, at most, weekly. Minutely updates on OpenStreetMap.org are a crucial part of the feedback cycle where mappers can see their edits right away and get inspired to map more often. Typically a mapper can make an edit and see their edit when reloading after 90-180 seconds, compared to the days or weeks of most OSM-based services, or the months or years of proprietary data sources.

Updating maps once a week can be done with a simple architecture that takes the OSM file for the planet and turns it into a single file containing all the tiles for the world. This can scale to daily updates, but not much faster. To do minutely updates we need to generate tiles one-by-one, since they change one-by-one. When combined with the caching requirements for osm.org, this is something no existing software solved.

For some time I’ve been working on Tilekiln, a small piece of software which leverages the existing vector tile generation of PostGIS, the standard geospatial database. Tilekiln is written specifically to meet the unique requirements of a default layer on osm.org. Recently, I’ve been working for the OSMF at setting up minutely updated vector tiles using the Shortbread schema. A schema is a set of definitions for what goes in the vector tiles, and Shortbread is a CC0 licensed schema that anyone can use and there are existing styles for.

… 查看全文

I’ve been looking at how many tiles are changed when updating OSM data in order to better guide resource estimations, and have completed some benchmarks. This is the technical post with details, I’ll be doing a high-level post later.

Software like Tilemaker and Planetiler is great for generating a complete set of tiles, updated about once a day, but they can’t handle minutely updates. Most users are fine with daily or slower updates, but OSM.org users are different, and minutely updates are critical for them. All the current minutely ways to generate map tiles involve loading the changes and regenerating tiles when data in them may have changed. I used osm2pgsql, the standard way to load OSM data for rendering, but the results should be applicable to other ways including different schemas.

Using the Shortbread schemea from osm2pgsql-themepark I loaded the data with osm2pgsql and ran updates. osm2pgsql can output a list of changed tiles (“expired tiles”) and I did this for zoom 1 to 14 for each update. Because I was running this on real data sometimes an update took longer than 60 seconds to process if it was particularly large, and in this case the next run would combine multiple updates from OSM. Combining multiple updates reduces how much work the server has to do at the cost of less frequent updates, and this has been well documented since 2012, but no one has looked at the impact from combining tiles.

To do this testing I was using a Hezner server with 2x1TB NVMe drives in RAID0, 64GB of RAM, and an Intel i7-8700 @ 3.2 GHz. Osm2pgsql 1.10 was used, the latest version at the time. The version of themepark was equivalent to the latest version

The updates were run for a week from 2023-12-30T08:24:00Z to 2024-01-06T20:31:45Z. There were some interruptions in the updates, but I did an update without expiring tiles after the interruptions so they wouldn’t impact the results.

To run the updates I used a simple shell script

… 查看全文

The Standard Tile Layer has a lot of traffic. On August 1st, a typical day, it had 2.8 billion requests served by Fastly, about 32 thousand a second. The challenges of scaling to this size are documented elsewhere, and we handle the traffic reliably, but something we don’t often talk about is the logging. In some cases, you could log a random sample of requests but that comes with downsides like obscuring low frequency events, and preventing some kinds of log analysis. Critically, we publish data that depends on logging all requests.

We query our logs with Athena, a hosted version of Presto, a SQL engine that, among features, can query files on an object store like S3. Automated queries are run with tilelog, which publishes files daily to generate published files on usage of the standard tile layer.

As you might imagine, 2.8 billion requests is a lot of log data. Fastly offers a number of logging options, and we publish compressed CSV logs to Amazon S3. These logs are large, and suffer a few problems for long-term use because they:

1. contain personal information like request details and IPs, that, although essential for running the service, cannot be retained forever;
2. contain invalid requests, making analysis more difficult;
3. are large, being 136 GB/day; and
4. become slow to query, being compressed gzip files with the only indexing being the date and hour of the request, which is part of the file path.

To solve these problems we reformat, filter, and aggregate logs which lets us delete old logs. We’ve done the first two for some time, and are now doing the third.

… 查看全文

I was curious about the usage of Maxar over the last year, so did some quick work to see where it was used. To start, I used a Python 3 version of ChangesetMD to load the latest changesets into PostgreSQL, using the -g option to create geometries.

I then, with a bit of manual work, identified the changesets of the last year are those between 122852000 and 137769483. Using this, and knowledge of tags normally used with maxar, I created a materialized view with just the Maxar changesets

CREATE MATERALIZED VIEW maxar AS
SELECT id,
    num_changes,
    st_centroid(geom)
FROM osm_changeset
WHERE id BETWEEN 122852000 and 137769483
    AND (tags->'source' ILIKE '%maxar%' AND tags->'imagery_used' ILIKE '%maxar%');

This created a table of 2713316 changesets, which is too many to directly view, so I needed to get it by country.

I did this with the border data from country-coder

curl -OL 'https://raw.githubusercontent.com/rapideditor/country-coder/main/src/data/borders.json'
ogr2ogr -f PostgreSQL PG:dbname=changesetmd borders.json

This loaded a quick and dirty method of determining the point a country is in into the DB, allowing me to join the tables together

SELECT COALESCE(iso1a2, country), COUNT(*)
FROM maxar JOIN borders ON ST_Within(maxar.st_centroid, borders.wkb_geometry)
GROUP BY COALESCE(iso1a2, country)
ORDER BY COUNT(*) DESC;

… 查看全文

I added functionality to tilelog to generate per-country usage information for the OSMF Standard Map Layer. The output of this is a CSV file, generated every day, which contains country code, number of unique IPs that day, tiles per second, and tiles per second that were a cache miss, all for each country code.

With a bit of work, I manipulated the files to give me the usage from the 10 countries with the most usage, for the first four months of 2023.

Perhaps more interesting is looking at the usage for each country by the day of week.

… 查看全文

The Operations Working Group is looking at what it take to deprecate HTTP Basic Auth and OAuth 1.0a in favour of OAuth 2.0 on the main API in order to improve security and reduce code maintenance.

Some of the libraries that the software powering the API relies on for OAuth 1.0a are unmaintained, there is currently a need to maintain two parallel OAuth interfaces, and HTTP Basic Auth requires bad password management practices. OAuth 2.0 libraries should be available for every major language.

We do not yet have a timeline for this, but do not expect to shut off either this year. Before action is taken, we will send out more notifications. Deprecation may be incremental, e.g., we may shut off creation of new applications as an earlier step.

What can you do to help?

If you are developing new software that interacts with the OSM API, use OAuth 2.0 from the start. Non-editing software can require authentication support, e.g. software that checks if you have an OSM login.

If you maintain existing software, then look into OAuth 2.0 libraries that can replace your OAuth 1.0a ones. We do not recommend implementing support for either protocol version “by hand”, as libraries are readily available and history has shown that implementing your own support is prone to errors.

If you do not develop software that interacts with the OSM API, this change will not directly impact you. You may need to update software you use at some point.

I have been developing Street Spirit, a new style using OpenStreetMap data. It uses Maplibre GL for client side rendering of MVTs generated by Tilekiln, which supports minutely updates using the standard osm2pgsql toolchain.

To focus style development, I have set its aims as being suitable for

• use as a locator map,
• to show off what can be done with OpenStreetMap data,
• to be up-to-date with the latest OpenStreetMap data, and
• using to orient a viewer to a location they are at.

Although not complete - if a style can ever said to be complete - it is at the point where there’s enough features to give the overall feel of the map, at least for zooms 12 and higher. Lower zooms are missing many features still, particularly roads and rail and some landcover and other fills.

Because the style has a more clearly defined purpose, I’ve been able to use more of the colour pallet than many other styles, particularly compared to styles designed for overlaying other data on top of.

I’ve set up a dev instance on one of my servers, using OSM data from 2023-02-27. Have an explore around.

Some of the bigger areas that need work are

• Missing mid- and low-zoom features
• Missing fills
• A consistent set of POI icons
• More POIs

If you’re interested in contributing to the work, let me know. Contributing will require some technical knowledge in the following areas

• MapLibre GL style specification, focusing on layers and expressions, including data-driven expressions;
• YAML, in particular appropriate indentation for arrays. MapLibre GL styles tend to feature deeply nested arrays; and
• SQL for writing read-only PostGIS queries if modifying vector tiles.

… 查看全文

Dear all,

Today, v5.7.0 of the OpenStreetMap Carto stylesheet (the default stylesheet on the OSM website) has been released. Once changes are deployed on openstreetmap.org it will take couple of days before all tiles show the new rendering.

Changes include - Unpaved roads are now indicated on the map (#3399)

• Country label placement improved, particularly for countries in the north (#4616)

• Added elevation to wilderness huts (#4648)

• New index for low-zoom performance (#4617)

• Added a script to switch between script variations for CJK languages (#4707)

• Ordering fixes for piers (#4703)

• Numerous CI improvements

Thanks to all the contributors for this release, including wyskoj, tjur0, depth221, SlowMo24, altilunium, and cklein05, all new contributors.

For a full list of commits, see https://github.com/gravitystorm/openstreetmap-carto/compare/v5.6.2…v5.7.0

As always, we welcome any bug reports at https://github.com/gravitystorm/openstreetmap-carto/issues

I gathered some statistics about usage of the website and tiles in 2022Q3.

I looked at total tile.osm.org usage, tile.osm.org usage from osm.org itself, osm.org visits, and osm.org unique visitors.

Here’s the data for the top 20 countries.

country	osm.org tile requests	total tile requests	Website visits	Website unique Visitors
DE	17.79%	7.78%	8.27%	7.98%
RU	12.23%	8.49%	2.47%	2.43%
US	8.72%	9.22%	13.11%	13.56%
PL	7.69%	4.99%	3.09%	2.80%
GB	4.85%	3.68%	4.42%	4.42%
FR	4.79%	7.00%	3.91%	3.94%
NL	3.62%	3.31%	2.17%	2.09%
IT	3.49%	3.46%	4.74%	4.86%
IN	2.64%	2.66%	3.67%	3.16%
CN	2.62%	0.79%	2.65%	2.72%
AT	2.03%	0.89%	0.98%	0.91%
UA	1.78%	1.98%	1.20%	1.21%
CH	1.41%	0.71%	0.83%	0.82%
CA	1.29%	1.59%	1.36%	1.39%
BE	1.29%	1.06%	1.10%	1.03%
ES	1.27%	2.41%	2.32%	2.39%
JP	1.10%	1.54%	1.74%	1.71%
AU	1.09%	0.92%	0.88%	0.82%
SE	0.91%	0.95%	0.87%	0.88%
FI	0.89%	0.74%	0.74%	0.71%

I’ve put the full data into a gist on github

Dear all,

Today, v5.6.1 of the OpenStreetMap Carto stylesheet (the default stylesheet on the OSM website) has been released. Once changes are deployed on the openstreetmap.org it will take couple of days before all tiles show the new rendering.

Changes include

• Fixing rendering of water bodies on zooms 0 to 4

Thanks to all the contributors for this release.

For a full list of commits, see https://github.com/gravitystorm/openstreetmap-carto/compare/v5.6.0…v5.6.1

As always, we welcome any bug reports at https://github.com/gravitystorm/openstreetmap-carto/issues

Dear all,

Today, v5.6.0 of the OpenStreetMap Carto stylesheet (the default stylesheet on the OSM website) has been released. Once changes are deployed on the openstreetmap.org it will take couple of days before all tiles show the new rendering.

Changes include

• using locally installed fonts instead of system fonts, for more up to date fonts;
• changing tree and tree row colours to the same colour as areas with trees;
• rendering parcel lockers; and
• rendering name labels of bays and straights from z14 only, and lakes from z5

Thanks to all the contributors for this release including GoutamVerma, yvecai, ttomasz, and Indieberrie, new contributors.

For a full list of commits, see https://github.com/gravitystorm/openstreetmap-carto/compare/v5.5.1…v5.6.0

As always, we welcome any bug reports at https://github.com/gravitystorm/openstreetmap-carto/issues

OpenStreetMap Carto could use more help reviewing pull requests, so if you’re able to, please head over to Github and review some of the open PRs.

This is a bit less OpenStreetMap related then normal, but has to do with the Standard Tile Layer and an outage we had this month.

On July 18th, the Standard Tile Layer experienced degraded service, with 4% of traffic resulting in errors for 2.5 hours. A significant factor in the time to resolve the incident was a lack of visibility of the health status of the rendering servers. The architecture consists of a content delivery network (CDN) hosted by Fastly, backed by 7 rendering servers. Fastly, like most CDNs, offers automatic failover of backends by fetching a URL on the backend server and checking its response. If the response fails, it will shift traffic to a different backend.

A bug in Apache resulted in the servers being able to handle only a reduced number of connections, causing a server to fail the health check, diverting all load to another server. This repeated with multiple servers, sending the load between them until the first server responded to the health check again because it had zero load. Because the servers were responding to most of the manually issued health checks and we had no visibility into how each Fastly node was directing its traffic, it took longer to find the cause than it should have.

Our normal monitoring is provided by Statuscake, but this wasn’t enough here. Instead of increasing the monitoring, we wanted to make use of the existing Fastly healthchecks, which probe the servers from 90 different CDN points. Besides being a vastly higher volume of checks, this more directly monitors the health checks that matter for the service

During the incident, Fastly support provided some details on how to monitor health check status. Based on this guide, the OWG has set up an API on the tile CDN to indicate backend health, and monitoring to track this across all POPs.

… 查看全文

Dear all,

Today, v5.5.1 of the OpenStreetMap Carto stylesheet (the default stylesheet on the OSM website) has been released. Once changes are deployed on the openstreetmap.org it will take couple of days before all tiles show the new rendering.

The one change is a bugfix to the colour of gates (#4600)

For a full list of commits, see https://github.com/gravitystorm/openstreetmap-carto/compare/v5.5.0…v5.5.1

As always, we welcome any bug reports at https://github.com/gravitystorm/openstreetmap-carto/issues

Dear all,

Today, v5.5.0 of the OpenStreetMap Carto stylesheet (the default stylesheet on the OSM website) has been released. Once changes are deployed on the openstreetmap.org it will take couple of days before all tiles show the new rendering.

Changes include

• Fixed colour mismatch of car repair shop icon and text (#4535)

• Cleaned up SVG files to better align with Mapnik requirements (#4457)

• Allow Docker builds on ARM machines (e.g. new Apple laptops) (#4539)

• Allow file:// URLs in external data config and caching of downloaded files (#4468, #4153, #4584)

• Render mountain passes (#4121)

• Don’t use a cross symbol for more Christian denominations that don’t use a cross (#4587)

Thanks to all the contributors for this release, including stephan2012, endim8, danieldegroot2, and jacekkow, new contributors.

For a full list of commits, see https://github.com/gravitystorm/openstreetmap-carto/compare/v5.4.0…v5.5.0

As always, we welcome any bug reports at https://github.com/gravitystorm/openstreetmap-carto/issues