Skip to content

Add docs for updating external dependencies #1280

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
+33 −1
Open

Add docs for updating external dependencies #1280

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
6b6bbad
Add docs for updating external dependencies
sethmlarson Feb 27, 2024
6646cf8
Apply suggestions from code review
sethmlarson Feb 29, 2024
cefc01d
Add PCbuild/build.bat --regen option to docs
sethmlarson Feb 29, 2024
a154e01
Apply suggestions from code review
sethmlarson Mar 4, 2024
cc0709a
Update developer-workflow/sbom.rst
sethmlarson Mar 4, 2024
b32b691
Add warning for non-core developers and external dependencies
sethmlarson Mar 4, 2024
3588722
Apply suggestions from code review
sethmlarson Mar 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion developer-workflow/sbom.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ After gathering this information:

* ``name`` for the project name.
* ``SPDXID`` which will be ``"SPDXRef-PACKAGE-{name}"``.
* ``licenseConcluded`` for the SPDX license identifier of the project license.
* ``licenseConcluded`` must be ``NOASSERTION``.
* ``versionInfo`` for the version of the project.
* ``downloadLocation`` should be an HTTPS URL for the project download as an archive.
* ``checksums[0].checksumValue`` and ``.algorithm`` will be the SHA-256
Expand Down Expand Up @@ -107,3 +107,19 @@ When removing a dependency:
that correct package is removed from the SBOM.
5. Commit the changes to :cpy-file:`Misc/sbom.spdx.json` and
:cpy-file:`Tools/build/generate_sbom.py`.

Updating external dependencies (cpython-source-deps)
----------------------------------------------------

Dependencies for Windows CPython builds are `stored in a separate repository <https://github.com/python/cpython-source-deps>`_
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some binaries are also stored in https://github.com/python/cpython-bin-deps, though generally they should also have sources in the source-deps repo. Is this distinction important here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do any of the cpython-bin-deps get shipped along with the CPython artifacts? If they're derived from the cpython-source-deps repository I think we should be okay.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the only one that isn't derived from cpython-source-deps is vcruntime140.dll, which comes from our repo to make sure we always get the latest one and not whichever GHA build machine we're on.

and then fetched during builds of CPython for Windows in the script :cpy-file:`PCbuild/get_externals.bat`.

In this script the libraries to fetch are designated by ``{name}-{version}`` git refs being added to the ``libraries`` variable.
SBOM tooling in the CPython repository matches these git refs in order to build the :cpy-file:`Misc/externals.spdx.json`
SBOM file. When updating external dependencies for a CPython branch:

1. Push the update to the ``cpython-source-deps`` repository and create a new git tag.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe worth noting that this can only be done by a core committer, and we don't accept PRs for it (because we need to verify the sources have come from the right source and are unmodified, and our trust boundary for this is "has the commit bit").

Also might be worth noting that sometimes there's a build step involved and the core committer will then push a tag to cpython-bin-deps that will actually be used in the build. Tcl/Tk, libffi and OpenSSL are all in this group.

In practice for contributors, what this usually means is that they should post an issue requesting the updated version, wait for a core dev to say the tags are ready, and then the contributor can continue with the following steps.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've addressed this comment in b32b691. Do you think we should cover the cpython-bin-deps part here as well?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not in the same note, but it ought to be documented somewhere. At the very least, we should mention the cpython-bin-deps repo at least once so that someone reading this knows to look there.

2. Update the entry for the project in ``get_externals.bat``.
3. Run ``make regen-sbom`` in the CPython source repository.
4. Verify the metadata (like version, download location) in ``externals.spdx.json`` SBOM is updated as expected with ``git diff``
5. Commit the changes and have them merged together.