Thomas Gauvin

Thomas Gauvin

Host Protomaps as static files on Cloudflare Pages for free with Service Workers

In my last blog post, I discussed how to host Protomaps on Cloudflare, using Cloudflare Workers and R2. This is the recommended approach for minimal latency and edge caching provided by Workers, in addition to being backwards compatible with clients that expect the Z/X/Y format. You should check out that blog post if you’re looking to host protomaps on Cloudflare with the least headaches and the best performance.

This blog post is about experimenting with hosting PMTiles as static assets on Cloudflare Pages. I’m working on a simple pmtiles styling project (https://pmtiles-styling.pages.dev/), and I wanted to be able to host the entire project on Cloudflare Pages (without having to rely on server-side compute). (I work at Cloudflare so that’s why I’m using Cloudflare Pages.)

Quick recap on PMTiles

PMTiles is a file format for storing map data as a single file, optimized to be hosted on cloud object store services. Mapping libraries, such as MapLibre GL JS or Leaflet, can then make HTTP Range Requests to retrieve sections of the file. Given the PMTiles file format, these mapping libraries know which ranges to request and how to parse the received data to display the map information on screen. 

As explained in the Protomaps docs https://docs.protomaps.com/deploy/, it is possible to host static PMTiles files on a simple static server, but that server needs to support HTTP Range Requests. In this case, Cloudflare Pages (and most static website hosting platforms) do not support range requests (they also has file size limits, but we’ll get to that later).

The experiment

For https://pmtiles-styling.pages.dev/, I’m using MapLibre GL JS, and this library is directly compatible with PMTiles. I’m hosting 2 base maps, one of New York City and one of the World, with different zoom levels, such that both of these are below the 25 MiB file size limit. 

Since MapLibre will make range requests for these files, we’ll need to intercept the request on the browser and respond to it locally instead of Cloudflare Pages. Now how could we do that…? Service Workers!

What are Service Workers again?

A service worker is a script that runs in the background, separate from the main JavaScript of our website, that can intercept network requests and cache resources, typically used to improve performance and offline support (PWAs).

Given this capability, it should be possible to fetch the full static .pmtiles files once and store them in the cache, intercept HTTP Range Requests made by the mapping library, and then respond to the range requests by splicing the bytes of the full static .pmtiles files. This should work with our existing mapping libraries. 

How it works

Here’s a quick architecture diagram of how our service worker setup for PMTiles will work and how it compares to the standard usage of PMTiles and Workers. 

{% image ”./uploads/static-protomaps-on-cloudflare-architecture.png” ,“w-auto m-auto h-full max-h-[36rem] rounded-md object-contain”, “Architecture of Static Protomaps on Cloudflare Pages vs Protomaps on Cloudflare Workers and R2”, “400” %}

In the above half of this architecture diagram, we can see the typical setup of Cloudflare Workers as a map tile server, serving areas of the map from the R2 data. This allows requests for areas of the map to get quick responses since only that area needs to be loaded onto the browser (kilobytes!). It also provides us with a way to store very large (100gb +) .pmtiles files and serve more information as a user zooms in closer.

The second half contains our service worker implementation. In this scenario, upon page load, in addition to loading the React bundle, our site also loads the full static .pmtiles file into local cache (megabytes!). Then, the service worker will intercept requests and parse the .pmtiles file to build the response from a range of bytes.

Granted, this service worker experience will be less performant than using the Workers solution, since the full static PMTiles file (~15-20mb) will need to be loaded into the cache upon page load. But, since this project can tolerate longer initial load times, it’s worth the tradeoff (and there’s a way to minimize this initial load time that I’ll touch on towards the end).

Let’s get into the code!

I’m using React for this project but this will work with any frontend library (or just using plain JavaScript as well). This blog extends and builds upon the simple demo project built in my previous blog post.

First, we’ll get a PMTiles file for the map areas we want to display using the pmtiles CLI. We need to have files under the file size limit of our static site host (in this case, Cloudflare Pages has a 25MiB file size limit) so I’ll download the world map with maxzoom 5 and place it in the public folder of my Vite React project.

pmtiles extract https://build.protomaps.com/20230918.pmtiles world.pmtiles --maxzoom=5

{% image ”./uploads/static-protomaps-on-cloudflare-public-folder.png” ,“w-auto m-auto h-full max-h-80 rounded-md object-contain”, “Screenshot of public folder for PMTiles styling project”, “400” %}

Next, we’ll update the map code to reference this file as the url of the protomaps source. By placing our world.pmtiles file in the /public folder of our project, it will be served at the /world.pmtiles URL of our website. (Depending on your project, you may have to change the URL to point it to where your PMTiles file is served from.)

import React, { useEffect } from "react";
import { Protocol } from "pmtiles";
import maplibregl, { LayerSpecification } from "maplibre-gl";
import Map from "react-map-gl/maplibre";
import layers from "protomaps-themes-base";

export const MapComponent = () => {

  useEffect(() => {
    let protocol = new Protocol();
    maplibregl.addProtocol("pmtiles", protocol.tile);
    return () => {
      maplibregl.removeProtocol("pmtiles");
    };
  }, []);

  return (
    <>
      <Map
        initialViewState={ {
          longitude: -74.006,
          latitude: 40.7128,
          zoom: 12,
        } }
        mapStyle={ {
          version: 8,
          sources: {
            protomaps: {
              type: "vector",
              url: "pmtiles:///world.pmtiles", //URL is the world.pmtiles file!
              maxzoom: 5
            },
          },
          glyphs:
            "https://protomaps.github.io/basemaps-assets/fonts/{fontstack}/{range}.pbf",
          layers: layers("protomaps", "light"),
        } }
        attributionControl={false}
      />
    </>
  );
};

Running npm run dev, we’ll actually see that our map is properly loading. This is because the Vite dev server supports HTTP Range Requests, but this won’t work when deploying to a static site host/Cloudflare Pages (try it out!).

Here’s where things get interesting. We’ll now create a service-worker.js file in our public folder. Here’s the code:

// service-worker.js

//install the service worker
self.addEventListener('install', event => {
    console.log('Service Worker installing.');
    event.waitUntil(self.skipWaiting()); // Activate worker immediately
});

//activate the service worker
self.addEventListener('activate', event => {
    console.log('Service Worker activating.');
    event.waitUntil(self.clients.claim()); // Become available to all pages
});

//intercept fetch requests for pmtiles files to respond from local static file
self.addEventListener('fetch', event => {
    const request = event.request;
    console.log("Service Worker handling network request for: ", request.url);
    console.log((new URL(request.url)).pathname)

    //this is a hacky way to verify that the service worker is intercepting fetch requests, called in React before rendering the map
    if((new URL(request.url)).pathname === "/checkSw"){
        console.log("returning A-OK")
        return event.respondWith(new Response("A-OK", {
            status: 202, // Use 200 to indicate successful response
            headers: {
                'Content-Type': 'text/plain', // Set appropriate content type
                'X-Sw-Tag': 'Served by Service Worker'
            }
        }));
    }

    //if the URL is anything other than the .pmtiles file, respond normally
    if ((new URL(request.url)).pathname == "/world.pmtiles") {
        return event.respondWith(handleRangeRequest(request));
    }
    return event.respondWith(fetch(request));
});

//function to handle the request for the pmtiles file
async function handleRangeRequest(request) {
    console.log("Service Worker handling range request for: ", request.url);
    const path = (new URL(request.url)).pathname;
    const pmtilesFile = await fetchPmtilesFile(path);
    const rangeHeader = request.headers.get('Range');

    // If there is a range header, then we use the service worker to respond to the HTTP Range Request 
    // by getting the needed chunks from the locally cached static PMTiles file
    if (rangeHeader) {
        const rangeMatch = rangeHeader.match(/bytes=(\d+)-(\d+)?/);
        if (rangeMatch) {
            const start = parseInt(rangeMatch[1], 10);
            const end = rangeMatch[2] ? parseInt(rangeMatch[2], 10) : pmtilesFile.byteLength - 1;

            const chunk = pmtilesFile.slice(start, end + 1);
            return new Response(chunk, {
                status: 206,
                statusText: 'Partial Content',
                headers: {
                    'Content-Type': 'application/octet-stream',
                    'Content-Length': chunk.byteLength,
                    'Content-Range': `bytes ${start}-${end}/${pmtilesFile.byteLength}`,
                    'X-Sw-Tag': 'Served by Service Worker'
                }
            });
        }
    }

    // If no Range header, return the entire file
    return new Response(pmtilesFile, {
        status: 200,
        statusText: 'OK',
        headers: {
            'Content-Type': 'application/octet-stream',
            'Content-Length': pmtilesFile.byteLength
        }
    });
}


//function to fetch the static pmtiles file and serve it from the cache
async function fetchPmtilesFile(path) {
    const cache = await caches.open("pmtiles-file-cache");
    const cachedResponse = await cache.match(path);

    if (cachedResponse) {
        return cachedResponse.arrayBuffer();
    }

    console.log("Fetching from network");
    const response = await fetch(path);
    const responseClone = response.clone()
    const responseBuffer = await response.arrayBuffer();

    try {
        await cache.put(path, responseClone);
    } catch (e) {
        console.log("Problem writing to cache: ", e);
    }

    return responseBuffer;
}

I’ve commented the main sections with explanations of the code, and left logs so you can follow along in the developer console. Essentially, apart from installing and activating itself, this service worker will intercept requests for the .pmtiles file, and serve HTTP Range Requests for this file from bytes of the cached static .pmtiles file.

Now, we can register this Service Worker manually in the map component.

import Map from "react-map-gl/maplibre";
import { Protocol } from "pmtiles";
import maplibregl, { LayerSpecification } from "maplibre-gl";
import { useEffect, useState } from "react";
import layers, { layersWithPartialCustomTheme } from "protomaps-themes-base";

import { Theme } from "../lib/theme";

export const MapComponent = ({ customTheme }: { customTheme: Partial<Theme> }) => {

  const [swLoaded, setSwLoaded] = useState(false);
  
  //... existing useEffect to use pmtiles Protocol 

  //this is a hacky way to verify that the service worker is intercepting fetch requests, called before rendering the map
  //if we don't do this, the map will try to make a range request to Cloudflare Pages and get an error
  useEffect(() => {
    const checkServiceWorker = async () => {
      try {
        const response = await fetch('/checkSw');
        console.log(response);
        console.log(response.headers.get('X-Sw-Tag'))

        // Check if the response status is 202 and the X-Sw-Tag header is present
        if (response.status === 202 && response.headers.get('X-Sw-Tag') === 'Served by Service Worker') {
          console.log('Service worker is active');
          setSwLoaded(true);
        } else {
          console.log('Service worker is not active, reloading the page...');
          window.location.reload();
          location.reload(true); 
        }
      } catch (error) {
        console.error('Error checking service worker:', error);
        window.location.reload(); // Reload the page in case of an error
        location.reload(true); 
      }
    };

    checkServiceWorker();
  }, []);

  //manually register the service worker
  useEffect(()=> {
    if ('serviceWorker' in navigator) {
      console.log('registering service worker')
      navigator.serviceWorker.register('/service-worker.js')
          .then(registration => {
              console.log('Service Worker registered with scope:', registration.scope);
          })
          .catch(error => {
              console.log('Service Worker registration failed:', error);
          });
    }
  }, []);

  return (
    <>
      {
        swLoaded ?
        <Map
        //....
      />
      :
      <div className="fixed inset-0 z-50 bg-white">
        Loading
      </div>
      }
    </>
  );
};

(This could be optimized by loading the Service Worker as part of the project/Vite bundle.)

Now, when loading our page, if the service worker is not yet registered, it will be registered and the page will be reloaded for it to take effect. On the next page load, we’ll notice that the Service Worker is successfully loaded and registered from the console logs. We can also see that our Service Worker is handling range requests from the locally cached file.

And that’s it!

When we deploy our project to Cloudflare Pages, we can see that our map loads correctly since it is served from Service Workers!

Above is an iframe of https://static-protomaps-on-cloudflare-maxzoom-9.pages.dev/.

Taking this further

Given the size limitation of files on static website hosts, this solution might be limiting given it provides a very small zoom range and does not provide much detail when zoomed in. 2 potential solutions are available for this.

Solution 1: Superimpose multiple PMTiles maps

For my PMTiles styling app (https://pmtiles-styling.pages.dev/), I wanted to provide a world map, while retaining detailed areas of New York City in order to enable developers to see the styling of zoomed in features. So, I downloaded a separate PMTiles file for New York City (also <25MB) and superimposed this on top of the world map as a separate basemap layers in maplibre. This is actually what is embedded above, notice the difference in zoom levels between New York City and other cities:

Above is an iframe of https://static-protomaps-on-cloudflare.pages.dev/.

With this solution, we have one detailed map section of New York City and a less detailed base map of the entire world. We need to deduplicate the styling layers, which we do by appending the index of the layer (-2) for the second layer.

export const MapComponent = ({ customTheme }: { customTheme: Partial<Theme> }) => {

  //[...]

  return (
    <>
        <Map
            initialViewState={ {
            longitude: -74.006,
            latitude: 40.7128,
            zoom: 12,
            } }
            mapStyle={ {
            version: 8,
            sources: {
                protomaps: {
                type: "vector",
                url: "pmtiles:///world.pmtiles",
                },
                protomaps2: {
                type: "vector",
                url: "pmtiles:///nyc.pmtiles",
                },
            },
            glyphs:
                "https://protomaps.github.io/basemaps-assets/fonts/{fontstack}/{range}.pbf",
            layers: [
                ...layersWithPartialCustomTheme(
                "protomaps",
                "light",
                customTheme
                ).filter((e) => {
                    return !e.id.includes("background");
                }),
                ...layersWithPartialCustomTheme(
                "protomaps2",
                "light",
                customTheme
                ).filter((e) => {
                    return !e.id.includes("background");
                }).map((e) => {
                    return {
                        ...e,
                        id: `${e.id}-2` //we need to remove conflicting map layer ids, which we do by appending '-2' to each id
                    };
                }),
            ],
            } }
            mapLib={maplibregl}
            attributionControl={false}
         />
      
    </>
  );
};

Solution 2: Split a large (~2gb) PMTiles file into smaller chunks and serve HTTP Range Requests from these

Since we’re responding to HTTP Range Requests by parsing the requested range from bytes of a single .pmtiles file, we should be able to do the same across multiple .pmtiles files. So we’ll download a large .pmtiles file and split it into many different files.

pmtiles extract https://build.protomaps.com/20230918.pmtiles world-2gb.pmtiles --maxzoom=12

Now, we can split up this world-2gb.pmtiles file into 10MB chunks, which we’ll host as static files, and use to respond to the HTTP Range Requests. These will fit within the file size limits of our static website host. To split up this file, we can write a simple script to achieve this:

const fs = require('fs');
const path = require('path');

const inputFilePath = './world-2gb.pmtiles'; // Path to the large file
const outputDir = './world.pmtiles'; // Directory to save the smaller files
const chunkSize = 10 * 1024 * 1024; // 10MB in bytes

// Ensure the output directory exists
if (!fs.existsSync(outputDir)) {
    fs.mkdirSync(outputDir, { recursive: true });
}

// Function to split the file
function splitFile() {
    fs.stat(inputFilePath, (err, stats) => {
        if (err) {
            console.error('Error getting file stats:', err);
            return;
        }

        console.log('File size:', stats.size);
        console.log('Chunk size:', chunkSize);

        let bytesRead = 0;
        let partNumber = 1;
        let startByte = 0;

        const readStream = fs.createReadStream(inputFilePath, { highWaterMark: chunkSize });
        let writeStream;

        readStream.on('data', (chunk) => {
            if (!writeStream) {
                const endByte = startByte + chunkSize - 1;
                writeStream = fs.createWriteStream(path.join(outputDir, `${startByte}-${endByte}.bin`));
            }

            writeStream.write(chunk);
            bytesRead += chunk.length;
            startByte += chunk.length;

            // Close the current file and move to the next part if necessary
            if (bytesRead >= chunkSize) {
                writeStream.end();
                partNumber++;
                bytesRead = 0;
                writeStream = null; // Reset for the next chunk
            }
        });

        readStream.on('end', () => {
            if (writeStream) {
                writeStream.end(); // Ensure the last chunk is written
            }
            console.log('File splitting complete.');
        });

        readStream.on('error', (err) => {
            console.error('Error reading file:', err);
        });
    });
}

splitFile();

This will create a folder named world.pmtiles, with files named {start_byte}-{end_byte}.bin. We can then place this into the public folder of our React project such that each of these static assets are hosted.

{% image ”./uploads/static-protomaps-on-cloudflare-bins.png” ,“w-auto m-auto h-full rounded-md object-contain”, “Screenshot of chunked PMTiles in public folder of Vite React project”, “800” %}

Then, we’ll need to update our Service Worker to adapt to the chunks format of our PMTiles file.

// service-worker.js
self.addEventListener('install', event => {
    //unchanged from above
});

self.addEventListener('activate', event => {
    //unchanged from above
});

self.addEventListener('fetch', event => {
    //unchanged from above
});

async function fetchPmtilesFile(path) {
    //unchanged from above
}


const chunkSize = 10 * 1024 * 1024; // 10MB in bytes

function getFilePathsFromRange(path,range){
    const start = parseInt(range[1], 10);
    const end = range[2] ? parseInt(range[2], 10) : pmtilesFile.byteLength - 1;

    const firstFileIndex = Math.floor(start / chunkSize);
    const lastFileIndex = Math.floor(end / chunkSize);

    let chunkFilePaths = [];
    //from the first file to the last file, inclusive
    //return an array of file paths (file paths are first byte - last byte, where the index represents the multiple of the chunk size)  
    for (let i = firstFileIndex; i <= lastFileIndex; i++) {
        const startByte = i * chunkSize;
        const endByte = startByte + chunkSize - 1;
        const filePath = `${path}/${startByte}-${endByte}.bin`;
        chunkFilePaths.push(filePath);
    }

    return chunkFilePaths;
}

var _appendBuffer = function(buffer1, buffer2) {
    var tmp = new Uint8Array(buffer1.byteLength + buffer2.byteLength);
    tmp.set(new Uint8Array(buffer1), 0);
    tmp.set(new Uint8Array(buffer2), buffer1.byteLength);
    return tmp.buffer;
  };

async function handleRangeRequest(request) {
    //this is the folder name. by convention, the pmtiles file will be the name of the folder
    const path = (new URL(request.url)).pathname;
    const fileByteLength = 10485759; //hardcoded for now. this should be the last byte range of your last chunk

    const rangeHeader = request.headers.get('Range');
    const rangeMatch = rangeHeader.match(/bytes=(\d+)-(\d+)?/);
    const start = parseInt(rangeMatch[1], 10);
    const end = parseInt(rangeMatch[2], 10);

    const chunkFilePaths = getFilePathsFromRange(path,rangeMatch);

    const pmtilesChunkFiles = [];
    for (let i = 0; i < chunkFilePaths.length; i++) {
        const chunkFile = await fetchPmtilesFile(chunkFilePaths[i]);
        pmtilesChunkFiles.push(chunkFile);
    }

    let chunkStart = start % chunkSize;
    let chunkEnd = end % chunkSize;

    let chunk = new Uint8Array(0);
    for (let i = 0; i < pmtilesChunkFiles.length; i++) {
        //add to the chunk either the entire file or the part of the file that is requested
        //if it is the first file, only add the part of the file that is requested
        //if it is the last file, only add the part of the file that is requested
        //if it is a file in between, add the entire file
        if (i === 0) {
            chunkEnd = chunkEnd > chunkSize ? chunkSize - 1 : chunkEnd;

            const tempChunk = pmtilesChunkFiles[i].slice(chunkStart, chunkEnd + 1);
            chunk = _appendBuffer(chunk, pmtilesChunkFiles[i].slice(chunkStart, chunkEnd + 1));
        }
        else if (i === pmtilesChunkFiles.length - 1) {
            chunkStart = 0;
            chunk = _appendBuffer(chunk, pmtilesChunkFiles[i].slice(chunkStart, chunkEnd + 1));
        }
        else {
            chunk = _appendBuffer(chunk, pmtilesChunkFiles[i]);
        }
    }

    const byteSize = chunk.length;

    if (rangeHeader) {
        const start = parseInt(rangeMatch[1], 10);
        const end = rangeMatch[2] ? parseInt(rangeMatch[2], 10) : fileByteLength - 1;

        return new Response(chunk, {
            status: 206,
            statusText: 'Partial Content',
            headers: {
                'Content-Type': 'application/octet-stream',
                'Content-Length': byteSize,
                'Content-Range': `bytes ${start}-${end}/${fileByteLength}`,
                'X-Sw-Tag': 'Served by Service Worker'
            }
        });
    }

    // If no Range header, return the entire file
    return new Response(pmtilesFile, {
        status: 200,
        statusText: 'OK',
        headers: {
            'Content-Type': 'application/octet-stream',
            'Content-Length': pmtilesFile.byteLength
        }
    });
}

With this implementation, the service worker now parses the headers of the HTTP Range Requests to determine the bytes that are requested. Then, it fetches the appropriate chunks to respond to that request, and will append all of the bytes needed to respond to the HTTP Range Request.

Make sure to update the fileByteLength to the max byte of the PMTiles file (the end byte of the last file in the folder), depending on the file size you have. Also, make sure to update the chunk size if you change this..

And now, we can verify that the Service Worker is properly working, caching the various chunks and files that make up the entire 2gb PMTiles file. Specifically, from the Application tab of the Chrome developer tools, we can see the Cache storage with the various files used.

{% image ”./uploads/static-protomaps-on-cloudflare-cache-bins.png” ,“w-auto m-auto h-full rounded-md object-contain”, “Screenshot of chunked PMTiles in public folder of Vite React project”, “800” %}

If you end up trying different file sizes and PMTiles maxzooms, make sure to change the fileByteLength in the Service Worker and the maxzoom property of the MapLibre component source.

Above is an iframe of https://large-static-protomaps-on-cloudflare-maxzoom-12.pages.dev/.

Optimizing file sizes

This project was mainly an exploration. Using this approach will provide less optimal load times since the full PMTiles file must be stored in cache before Range Requests can be served from Service Workers. Moreover, it limits the amount of data that can be stored. Practically speaking, splitting a large file across many smaller chunks will also result in long deploy times .

However, it is possible to minimize the initial load time by chunking PMTiles into smaller chunks (<1mb) to make it load faster, which can be especially crucial for smaller devices and on slower connections. The chunked approach can also help load larger maps.

Conclusion

This was fun. It’s incredible how much progress there’s been in the map ecosystem over the past handful of years, and this takes me way back to when I was using MapBox in my startup days. Protomaps has made it so simple to self-host map servers, and I highly recommend checking out their website (https://protomaps.com/).

Let me know if you end up trying this out! (Send me a DM on Twitter/X!)