Skip to content

Add additional detail about the site deployment pipeline #812

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add additional detail about the site deployment pipeline #812

Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
55f5146
Add additional detail about the site deployment pipeline
willingc Aug 1, 2025
ee23f3b
fix headings
willingc Aug 1, 2025
acab48d
Add explicit statement for served files location
willingc Aug 1, 2025
7499831
Merge branch 'main' into update-deploy
willingc Aug 15, 2025
e6d6a12
Add link to action as suggested by melissawm
willingc Aug 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
141 changes: 85 additions & 56 deletions docs/developers/contributing/documentation/docs_deployment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,65 +3,94 @@
# Documentation and website deployment

```{note}
This guide is intended for napari core contributors and is not required reading
for regular contributors. If you're looking for information on how to contribute
This guide targets core team members doing deployments. Curious contributors may
find it interesting. If you're looking for information on how to contribute
to the documentation, see [](contributing-docs).
```

The napari documentation and website sources are spread over three repositories,
and these are connected and used to generate the documentation and website
through several CI workflows detailed below.

1. [`napari/napari`](https://github.com/napari/napari)
- **Workflow file:** [`build_docs.yml`](https://github.com/napari/napari/blob/main/.github/workflows/build_docs.yml)
- **job:** `build-and-upload`
- Pulls in sources from `napari/docs` and builds docs locally. Uploads
artifacts to this repository (`napari/napari`).
- This is triggered on any push to a branch named 'docs*' or tag named 'v*',
to `napari/napari`. The artifact can be used to check the documentation
but is not used for docs deployment.
- **Workflow file:** [`deploy_docs.yml`](https://github.com/napari/napari/blob/main/.github/workflows/deploy_docs.yml)
- **job:** `build-napari-docs`
- Triggers [`build_and_deploy.yml`](https://github.com/napari/docs/blob/main/.github/workflows/build_and_deploy.yml)
workflow at the [napari/docs](https://github.com/napari/docs) repo.
Waits for results and reports it.
- This is triggered on any commit to the `main` branch on
`napari/napari` (and consequently triggers a new deployment of the
`napari.org` website). When the commit is tagged, the `build_and_deploy.yml`
workflow will deploy to the version folder e.g., '{{ napari_version }}/'.

2. [`napari/docs`](https://github.com/napari/docs)
- **Workflow file:** [`build_and_deploy.yml`](https://github.com/napari/docs/blob/main/.github/workflows/build_and_deploy.yml)
- **job:** `build-and-upload`
- Pulls in sources from `napari/napari`, builds docs, then uploads as an
artifact named 'html' to this repository (`napari/docs`).
- This is triggered on every Pull Request and shows up as a "Build & Deploy PR
Docs / Build & Upload Artifact" check on the PR.
- **job:** `deploy`
- Downloads the artifact from the `build-and-upload` job and deploys the html
to GitHub pages at the `gh-pages` branch of
[napari/napari.github.io](https://github.com/napari/napari.github.io/tree/gh-pages).
- Always deploys to the `dev/` folder on `napari.github.io` (version
"latest" on the website).
- Deployment is triggered by any commit to the `main` branch on `napari/docs`
(and consequently triggers a new deployment of the `napari.org`
website.)

Note that these files are not identical to the `napari/napari` version.

3. [`napari/napari.github.io`](https://github.com/napari/napari.github.io)
- Contains built documentation files (.html) for all versions in the
`gh-pages` branch. Auto-deploys to `gh-pages` on every commit (which in
turn happens on every commit to the `main` branches of either
`napari/napari` or `napari/docs`).
- For every commit to the `gh-pages` branch of this repo, the
[`unversioned_pages.yml` action](https://github.com/napari/napari.github.io/blob/gh-pages/.github/workflows/unversioned_pages.yml)
is triggered and copies the latest contents of the index page and the dev
folder over the "stable" version of the documentation. This ensures that
both the landing page of the website and the developer documentation,
including the contributing guide, are always up to date.

## Notes
Deploying the napari website, which includes our documentation, requires the
orchestration of workflows across multiple repositories. The following
repos provide logic for the deployment pipeline:
- [`napari/napari-sphinx-theme`](https://github.com/napari/napari-sphinx-theme)
- [`napari/docs`](https://github.com/napari/docs)
- [`napari/napari`](https://github.com/napari/napari)
- [`napari/napari.github.io`](https://github.com/napari/napari.github.io)

The files served to the web are found in the `gh-pages` branch of the `napari/napari.github.io` repo.

## Site deployment pipeline

### From source files to html artifact ([`napari/docs`](https://github.com/napari/docs))

The deployment pipeline begins with the source files found in `napari/docs`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or napari/napari in the case of docstrings and examples.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to clarify. I want to make sure that I understand the actual approach since it's a bit confusing to have docs build/deploy actions spread across 3 repos.

Source files

  • rst/md files in napari/docs
  • docstrings in napari/napari source files
  • examples in napari/napari examples directory

napari/docs actions

What are they specifically doing related to deployment?

napari/napari actions

What are they doing related to deployment?

The reason that I took the workflow approach was I was having difficulty understanding which repo was doing what. There's a bunch of non-typical workflows happening and it's difficult for someone who has plenty of docs experience to follow what is happening with our workflow tooling.

While we can do a local build in napari/docs, it's unclear how the local build differs from the deployed site:

  • versioned / unversioned pages
  • actions and workflow responsibilities across repos
  • differences in Sphinx template locations than a standard Sphinx build

It's totally cool that we're doing something different. My hope is that we can document it for greater clarity for current and future contributors. ❤️

The `napari/docs` repo contains the website and documentation source files.
It uses the `napari/napari-sphinx-theme` to scaffold structure and styling.

**Workflow file:** [`build_and_deploy.yml`](https://github.com/napari/docs/blob/main/.github/workflows/build_and_deploy.yml)
- **job:** `build-and-upload`
- Pulls in sources from `napari/napari`, builds docs, then uploads as an
artifact named 'html' to this repository (`napari/docs`).
- This is triggered on every Pull Request and shows up as a "Build & Deploy PR
Docs / Build & Upload Artifact" check on the PR.
- **job:** `deploy`
- Downloads the artifact from the `build-and-upload` job and deploys the html
to GitHub pages at the `gh-pages` branch of
[napari/napari.github.io](https://github.com/napari/napari.github.io/tree/gh-pages).
- Always deploys to the `dev/` folder on `napari.github.io` (version
"latest" on the website).
- Deployment is triggered by any commit to the `main` branch on `napari/docs`
(and consequently triggers a new deployment of the `napari.org`
website.)

```{note}
The html artifact files created by the `build_and_deploy.yml` workflow are not
identical to the html files built by the `napari/napari` workflows.
Comment on lines +46 to +47
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for my understanding, what is the difference? I may be missing something here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs/.github/workflows/build_and_deploy.yml

Line 4 in 4d5e4ff

# Note this workflow is also triggered by

This was taken from the existing documentation on https://napari.org/stable/developers/contributing/documentation/docs_deployment.html in this line

docs/docs/developers/contributing/documentation/docs_deployment.md

Line 50 in 4d5e4ff

Note that these files are not identical to the `napari/napari` version.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'm not sure what this means? Might be worth removing, I don't remember why this is here and I don't see a reason why the artifact files would be different...

```
### From html artifact to deployment trigger ([`napari/napari`](https://github.com/napari/napari))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like the workflow-focused organization, but I'm not sure if this is clear. The build_docs workflow can be triggered by both the napari/napari or napari/docs repos, and in each case it will clone the other corresponding repo, build and deploy to napari.gihub.io, so I think these items don't cover all options.

The following situations are possible:

  • A commit to napari/napari is made.
    • The build_docs.yml workflow in napari/napari is triggered
    • The napari/docs repo is cloned to napari/napari actions environment
    • html artifacts are generated and pushed to napari.github.io
  • A commit to napari/docs is mae
    • The build_and_deploy.html workflow in napari/docs is triggered
    • The napari/napari repo is cloned to napari/docs actions environment
    • html artifacts are generated and pushed to napari.github.io

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is good context and unclear from the existing docs and the workflows in the repos.

A few questions:

The first situation when changes to napari/napari are made.

  1. Are the docs always built?
  2. Or are the docs built and deployed only on changes to examples and/or docstrings?
  3. Are the raw html artifacts that are pushed to napari.github.io, then acted upon by the actions in napari.github.io?

The second situation when commit is made to napari/docs

  1. Are the html files created by the napari/docs actions?
  2. Or does a napari/docs action trigger another action to run in napari/napari?
  3. Same question as Update contributor docs #3 above

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first situation when changes to napari/napari are made.

  1. Are the docs always built? Yes
  2. Or are the docs built and deployed only on changes to examples and/or docstrings? No
  3. Are the raw html artifacts that are pushed to napari.github.io, then acted upon by the actions in napari.github.io? Yes

The second situation when commit is made to napari/docs

  1. Are the html files created by the napari/docs actions? Yes
  2. Or does a napari/docs action trigger another action to run in napari/napari? No, napari/napari gets cloned into the actions environment in napari/docs and the docs are built to generate the html artifacts
  3. Same question as 3. above Yes, same process as with napari/napari, html artifacts are pushed to napari.github.io and the workflows there act on the artifacts

We have considered many different workflows in the past, such as having one single action that would run to build the docs when commits are made to either repo, but if that action lives on one repo it would be harder to detect when changes are made in the other repo, for example. We try to keep the two actions that build docs (in napari/napari and napari/docs) synchronized as much as possible though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify point 3 in both situations:

  • napari/napari action can push raw html artifacts to napari.github.io
  • napari/docs action can puah raw html artifacts to napari.github.io
  • These actions should produce the same raw html artifacts if acting on identical source files. In other words, given the same source files, each action would produce raw html artifacts which are identical.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes exactly - that is my understanding at least!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @melissawm. I will rework this PR later in the week. Thanks for the additional context. I have a couple of ideas to keep it workflow centric but to handle the case that either repo can trigger the push to napari.github.io

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

napari/napari action can push raw html artifacts to napari.github.io

Just to be clear, this happens on merge to main by triggering the building and deployment in napari/docs. This napari/napari workflow triggers the napari/docs build&deploy:
https://github.com/napari/napari/blob/main/.github/workflows/deploy_docs.yml

The napari/napari build_docs workflow is essentially unused at this point. For PRs we favor the CircleCI build, for the previews, so the build_docs is basically for manual triggers of an artifact build.


**Workflow file:** [`build_docs.yml`](https://github.com/napari/napari/blob/main/.github/workflows/build_docs.yml)
- **job:** `build-and-upload`
- Pulls in sources from `napari/docs` and builds docs locally. Uploads
artifacts to this repository (`napari/napari`).
- Any push to a branch named 'docs*' or tag named 'v*', to `napari/napari`
triggers this workflow. The artifact can be used to check the documentation
but is not used for docs deployment.

**Workflow file:** [`deploy_docs.yml`](https://github.com/napari/napari/blob/main/.github/workflows/deploy_docs.yml)
- **job:** `build-napari-docs`
- Triggers [`build_and_deploy.yml`](https://github.com/napari/docs/blob/main/.github/workflows/build_and_deploy.yml)
workflow at the [napari/docs](https://github.com/napari/docs) repo.
Waits for results and reports it.
- This is triggered on any commit to the `main` branch on
`napari/napari` (and consequently triggers a new deployment of the
`napari.org` website). When the commit is tagged, the `build_and_deploy.yml`
workflow will deploy to the version folder e.g., '{{ napari_version }}/'.

### From deployment trigger to a live site [`napari/napari.github.io`](https://github.com/napari/napari.github.io)

- Contains built documentation files (.html) for all versions in the
`gh-pages` branch. Auto-deploys to `gh-pages` on every commit (which in
turn happens on every commit to the `main` branches of either
`napari/napari` or `napari/docs`).
- For every commit to the `gh-pages` branch of this repo, the
[`unversioned_pages.yml` action](https://github.com/napari/napari.github.io/blob/gh-pages/.github/workflows/unversioned_pages.yml)
is triggered and copies the latest contents of the index page and the dev
folder over the "stable" version of the documentation. This ensures that
both the landing page of the website and the developer documentation,
including the contributing guide, are always up to date.

## Unversioned pages

The website and documentation is designed to support the concept of
**unversioned pages**. These pages will display the same content for any
documentation version and offers the reader the
most up-to-date and consistent information, such as community norms, project
roadmaps, and governance.
These pages do add a bit more complexity as there is a need to move and copy files
so they are the same across version. The `unversioned_pages.yml` action
does this work.

## CircleCI for tests and previews

CircleCI is set up in both `napari/napari` and `napari/docs`, but doesn't deploy
documentation. It serves to test the docs build and to provide documentation
Expand Down