Internationalizing Docs Pages with Astro

Here at Defined Networking we use Astro to build our website and we love the speed and flexibility it gives us. When we chose to open-source our documentation for the Nebula project which our company is based on, we decided to support internationalization so that contributors could translate the docs into their own languages.

This post is an explanation of the approach we took to build the new site, some considerations we made on the URLs of localized pages, and the ways in which Astro made it all possible.

Introduction to Astro

Astro is a static site generator (SSG), which means that all of the pages on the site need to be created up-front before the site is published. One thing that makes it different from similar site frameworks like Gatsby, Next.js, or SvelteKit is that Astro allows you to build components using whichever javascript framework (or frameworks!) you prefer, whether that’s React, Preact, Vue, or Svelte, and does not automatically hydrate javascript on every page load.

Instead, Astro creates pure html and css, and gives you control to opt specific components into interactivity and determine when they should download their javascript. Astro calls this ”partial hydration”, and it results in lean, fast loading websites. And the developer experience is top-notch, with instant hot reloads, scoped-styles, built in typescript support, and more.

Internationalization Objectives

Internationalization (shortened to i18n) is a complex topic and we are by no means experts in it. That said, we had a few requirements for our site that we think will give a smooth experience for readers of other languages:

1. Some pieces of each page should be consistent between all of the languages (images, names of configuration options, etc), and it should be easy for contributors to add new translations.
2. There should be a language picker at the top of the page to change the language.
3. The address in the URL should be localized, too.
4. If localized content is not available for any given page, fallback English content should be provided.

The rest of this post will detail how we accomplished each of these objectives.

1. Content Management

To add some structure and consistency to the docs pages, and to help keep translations in sync with each other, we decided to use a Content Management System, or CMS. The one we chose to use is Netlify CMS, which is a git-based CMS that stores all of the data in files in our repository. It also includes an i18n feature that allows us to mark individual fields as localizable or not, ensuring that things like the names of configuration settings stay the same between all the translations.

We won’t go into details here about how to set up Netlify CMS, but the end result is that we have two main “collections”, one for the locales and another for the docs pages. These are saved as files in the /src/data/ directory of our repo like this:

src/
 ┣ components/
 ┣ data/
 ┃ ┣ docs/
 ┃ ┃ ┣ en/
 ┃ ┃ ┃ ┗ introduction.md
 ┃ ┃ ┗ de/
 ┃ ┃   ┗ introduction.md
 ┃ ┣ settings/
 ┃ ┃ ┗ locales.json
 ┣ layouts/
 ┣ pages/
 ┣ scripts/
 ┗ styles/

Note that in this example there is one “introduction” doc in two locales, English (en) and German (de). The file is named the same across all locales, using the English title of the documentation page. We’ll see what is inside those files in a moment, but lets look in the locales.json file first:

{
  "locales": [
    {
      "language": "English",
      "locale": "en"
    },
    {
      "language": "Deutsch",
      "locale": "de"
    }
  ]
}

The list of locales is fed back into Netlify CMS’s i18n configuration, so that adding a new language is accomplished by first adding a new locale, then editing a page and choosing the new locale from a dropdown to start a new translation.

This file is also used in a few places in the webpage, most importantly in the language picker, which is why we use the native name of the language.

2. Language Picker

There are a number of themes / starter kits that you can choose from when creating a new site with Astro, and the docs theme comes with a handy language selection component. This allows a reader to manually change the language of the site to one which they prefer, and from that point on, all navigation should occur within that chosen locale.

The component is a select box with options for each of the locales in locales.json. When a new option is chosen, some javascript runs to change the url to the same page with the new locale. So, if the current page is defined.net/en/introduction, changing to Deutsch would navigate to defined.net/de/introduction. This works fine, but native German speakers wouldn’t expect to see the english word introduction in the url, which leads us to our next objective.

3. Localize the URL

The web address you see at the top of your browser window matters, especially when you’re sharing a link with someone else. It gives important hints about what the page contains, and for a localized website to feel truly complete, the URL should also be translated. So, in our “introduction” example, we might want the address for the German page to be defined.net/de/einleitung. This presents some challenges:

1. Where should this translation be stored? (The portion of the url we are talking about here is commonly called a “slug”, which is often a url-safe version of a page’s title).
2. How can we ensure that the correct address is shown when changing between languages?

Let’s take a look at each of these.

Storing the internationalized slug

We chose to add an i18n-enabled field into the CMS to store the slug for each page, which allows a unique value to be set for each locale. The other fields we are currently using are title, summary (to show in unfurls and social media cards), and then of course the main body of the page. When the CMS saves the page into markdown files, the fields are put into a yaml format in the file’s frontmatter, like this:

---
title: Introduction
slug: introduction
summary: Nebula is a scalable overlay networking tool
  with a focus on performance, simplicity and security.
  It enables you to seamlessly connect computers
  **anywhere** in the world. Nebula is portable,
  and runs on Linux, MacOS, Windows, iOS, and Android.
---

# Introduction to Nebula

...rest of the page

And the German version is similar:

---
title: Einleitung
slug: einleitung
summary: Nebula ist ein skalierbares Overlay-Netzwerktool
  mit Fokus auf Leistung, Einfachheit und Sicherheit.
  Es ermöglicht Ihnen die nahtlose Verbindung von Computern
  **überall auf der Welt. Nebel ist tragbar,
  und läuft auf Linux, MacOS, Windows, iOS und Android.
---

# Einleitung in Nebula

So, the base filenames are both the same (introduction.md), but we have unique slug values that are internationalized and the files are in a folder named for the locale. With those pieces of information, we have what we need to ensure that the correct slug is used when switching between languages.

Changing Locale

The strategy we took to make sure we could change from any language to any other language is for the select box to navigate to the new locale with an English slug (e.g. /de/introduction), and then set up redirects from those pages to the localized slugs (/de/einleitung). Here are the important parts of the language select box:

type Props = {
  /** The current locale */
  lang: string;
  /** The English slug for the current page */
  slug: string;
};

const LanguageSelect = ({ lang, slug }: Props) => {
  return (
    <select
      value={lang}
      onChange={(e) => {
        const newLang = e.target.value;
        window.location.pathname = `/${newLang}/${slug}`;
      }}
    >
      {Object.keys(KNOWN_LANGUAGES).map((key) => {
        return (
          <option key={key} value={KNOWN_LANGUAGES[key]}>
            {key}
          </option>
        );
      })}
    </select>
  );
};

KNOWN_LANGUAGES is derived from our locales.json file, and is shaped like:

{
  "English": "en",
  "Deutsch": "de"
}

For the next step, redirecting from /de/introduction to /de/einleitung, we need to start creating pages that will be served up to the browser.

Creating a redirect page

We could manually create a page for each combination of locale and slug, but this would quickly get out-of-hand. Even with one docs page and two locales, this would already require three files:

src/
 ┗ pages/
   ┣ en/
   ┃ ┗ introduction.astro
   ┗ de/
     ┣ introduction.astro
     ┗ einleitung.astro

The more content and locales we add, the more files we would need, and each of these would be nearly the same, either exposing the content from the corresponding file in our data/ directory, or redirecting to the localized slug.

Let’s look at what that redirect file might contain. In a .astro file, we can run build-time javascript inside a frontmatter block between --- lines, and then use the result in a jsx-like markup format. So maybe we could keep a mapping of slugs somewhere, and de/introduction.astro would be:

---
import { slugMap } from "../../slug-map";

const redirect = slugMap.introduction;
---

<head>
  <meta http-equiv="refresh" content={`0; url=${redirect}`} />
</head>

which, when built would result in a /dist/de/introduction/index.html file:

<html>
  <head>
    <meta http-equiv="refresh" content="0; url=/de/einleitung" />
  </head>
  <body></body>
</html>

Great! This will automatically redirect the browser to the url that we want. This does potentially come at the cost of some SEO, and a _redirects file that your hosting provider understands might be better in that regard, but that would be another file that needs to be manually updated for each page and each locale, and one of our goals is to make adding new locales as easy as possible.

Turning Markdown Data into a Page

Now let’s take a look inside /de/einleitung.astro to see how we can pull the data from the CRM into the page.

---
import Layout from "../../layouts/MainLayout.astro";

const pageData = Astro.fetchContent("../../data/docs/de/introduction.md");
---

<Layout content={pageData}>
  {pageData.astro.html}
</Layout>

The Layout component handles setting up the <head> of the page with our css, adding parts of the website that live on every page like the header and footer, and setting the title and social media meta tags based on the frontmatter from the markdown file, which we get as a result of calling Astro.fetchContent. Astro also converts the main markdown body of introduction.md into html, which we pass as a child to the layout for it to be included in the final markup of the page.

This all works great, and /en/introduction.astro would look almost the exact same, except that we’d be fetching content from the english file Astro.fetchContent('../../data/docs/en/introduction.md').

If we don’t want to manually update a redirect file and create new files for each page every time a new translation or docs page is added, we can use Astro’s dynamic routing functionality.

Dynamic Routing in a Static Generated Site?

This might sound like a contradiction, but this is not “dynamic” at runtime, but rather during the build. It is a flexible way of creating a lot of different pages from a single file using special bracket notation in the filename. So, instead of the explicit files we used earlier, we can create a single file named /src/pages/[locale]/[slug].astro. Then inside the file, we need to export a special function that will tell Astro what files to create. From the docs:

Pages that use dynamic routes must export a getStaticPaths() function which will tell Astro exactly what pages to generate.

In practice, for us this looks like:

---
// Utility functions to parse filenames to get their locale and English slug
import { getLanguageFromFilename, getSlugFromFilename } from "../../languages";
import Layout from "../../layouts/MainLayout.astro";

export async function getStaticPaths() {
  const allData = Astro.fetchContent("../../data/docs/**/*.md");
  const paths = allData.map(({ astro, url, file, ...page }) => {
    // This is the two-character code ('en' or 'de'), from the file path
    const locale = getLanguageFromFilename(file.pathname);
    // This is the English version of the slug
    const canonicalSlug = getSlugFromFilename(file.pathname);

    return {
      params: {
        locale: locale,
        // This is the translated version of the slug
        slug: page.slug,
      },

      props: {
        page: {
          ...page,
          // This is provided to the language picker as a `slug` prop,
          // so that it can redirect as we described above
          canonicalSlug,
          html: astro.html,
        },
      },
    };
  });
  // Filter out any paths with an undefined locale or slug.
  return paths.filter(({ params }) => !!params.locale && !!params.slug);
}

// This will be the props for this specific page, which was added above
const { page } = Astro.props;
---

<Layout content={page}>
  {page.html}
</Layout>

Now, when Astro runs its build process, it will use the params of each object we returned from getStaticPaths() to generate the following files:

dist/
 ┣ en/
 ┃ ┗ introduction.astro
 ┗ de/
   ┗ einleitung.astro

We’re getting close! We’re still missing the redirect file, though (/de/introduction.astro). We can make some tweaks to our /src/pages/[locale]/[slug].astro file to create all the redirect files that we need:

---
import Layout from '../../layouts/MainLayout.astro';
import {
  getLanguageFromFilename,
  getSlugFromFilename,
  DEFAULT_LOCALE
} from '../../languages';

export async function getStaticPaths() {
  /**
   * This builds up a set of params using the filename
   * (which is always in English) as the slug,
   * and adds a redirect prop to the proper internationalized slug.
   */
  function getRedirects(data) {
    return data
      .map(({ astro, url, file, ...page }) => {
        const locale = getLanguageFromFilename(file.pathname);
        const canonicalSlug = getSlugFromFilename(file.pathname);

        return {
          params: {
            locale,
            slug: canonicalSlug
          },
          props: {
            redirect: `/${locale}/${page.slug}`,
          },
        };
      })
      // Don't need redirects for the English (default) locale
      .filter(p => p.params.locale !== DEFAULT_LOCALE)
  }

  const allData = Astro.fetchContent('../../data/docs/**/*.md');

  const paths = /* same as before */;
  const redirects = getRedirects(allData);

  // Filter out any results with an undefined locale or slug.
  return [...paths, ...redirects].filter(({params}) =>
    !!params.locale && !!params.slug
  );
}

const { page, redirect } = Astro.props;
---

{redirect
  ? <head><meta http-equiv="refresh" content={`0; url=${redirect}`}></head>
  : <Layout content={page}>
      {page.html}
    </Layout>
}

Perfect! Now astro will build the files we need:

dist/
 ┣ en/
 ┃ ┗ introduction.astro
 ┗ de/
   ┣ introduction.astro
   ┗ einleitung.astro

And when the language picker is changed to Deutsch and navigates to /de/introduction, the page will redirect to /de/einleitung, and if it’s changed back to English, it’ll go straight to /en/introduction.

4. Fallback content

Now for the last objective, that fallback (English) pages should be shown if there is no matching translated version. We can expand on our getStaticPaths function yet again to create the missing pages. For example, let’s say we add another locale, Español: 'es', but don’t create a translation for the introduction. The data/docs/ folder would look like this:

docs/
┣ en/
┃ ┗ introduction.md
┣ de/
┃ ┗ introduction.md
┗ es/

As it stands, our Astro.fetchContent() won’t process any Spanish files because there aren’t any, so changing the language to Español in the language picker would result in a 404 page. That’s not a great experience, and we’d rather the visitor see English content, but have the url be /es/introduction, so that other pages that perhaps are translated would still be linked in the sidebar, etc. We can follow the same kind of pattern as the redirects to create the fallback pages like this:

---
import Layout from '../../layouts/MainLayout.astro';
import {
  getLanguageFromFilename,
  getSlugFromFilename,
  DEFAULT_LOCALE
} from '../../languages';

export async function getStaticPaths() {

  /**
   * Create any missing files for a locale, using to the default locale's content
   */
  function getFallbacks(data) {
    return data
      // Loop over the English (default) locale files
      .filter(p => getLanguageFromFilename(p.file.pathname) === DEFAULT_LOCALE)
      .map(({ astro, url, file, ...page }) => {
        const canonicalSlug = getSlugFromFilename(file.pathname);

        // Create any missing fallbacks
        return KNOWN_LANGUAGE_CODES
          .filter(
            // limit to non-default locales...
            locale => locale !== DEFAULT_LOCALE
            // ...which do not already have a file for this slug in this locale
            && !data.find(d =>
              getLanguageFromFilename(d.file.pathname) === locale
              && d.file.pathname.endsWith(`${canonicalSlug}.md`)
            )
          )
          .map(locale => ({
            params: {
              locale, // non-default locale, like `es`
              slug: canonicalSlug, // default slug, like `introduction`
            },
            props: {
              locale,
              page: {
                ...page,
                canonicalSlug,
                html: astro.html
              }
            }
          }))
      })
      // Flatten array of arrays into a single array
      .flat();
  }

  const allData = Astro.fetchContent('../../data/docs/**/*.md');

  const paths = /* same as before */;
  const redirects = /* same as before */;
  const fallbacks = getFallbacks(allData);

  // Filter out any results with an undefined locale or slug.
  return [...paths, ...redirects, ...fallbacks].filter(({params}) =>
    !!params.locale && !!params.slug
  );
}

const { page, redirect } = Astro.props;
---

{redirect
  ? <head><meta http-equiv="refresh" content={`0; url=${redirect}`}></head>
  : <Layout content={page}>
      {page.html}
    </Layout>
}

Finally, when Astro builds our site, it will create all the files we want:

dist/
 ┣ en/
 ┃ ┗ introduction.astro
 ┣ de/
 ┃ ┣ introduction.astro (redirects to einleitung)
 ┃ ┗ einleitung.astro
 ┗ es/
   ┗ introduction.astro (English fallback content)

Wrap-up

Using the strategies here, we have created a docs site that will support localization and allow contributions from developers in the nebula community. We are still finishing up a few details before using it to replace our current docs at www.defined.net/nebula, but if you’d like to see the techniques from this blog post being used in a real project, you can poke through the repo at https://github.com/DefinedNet/nebula-docs. Feel free to open an issue or submit a PR if you have feedback or ideas!

On that note, if working with cutting-edge technologies like Astro and Nebula seems like fun to you, check out our jobs postings; we’re currently looking for a frontend engineer to join the team.