Generation Software Bill of Materials (SBOM) #4757

gamlerhart · 2025-03-20T21:22:47Z

Motivation: In some companies, the development
team has to produce Software Bill of Materials (SBOM) for their project for compliance reasons:
To track dependencies and licenses across their organisation. Provide a Module that produces SBOMs
in JSON format.

Changes in the core: Extended the .getArtifact
to return the coursier.Resolution as well.
This is then used to get the license information.

Outside the core: Add a SBOM contrib module

Generate the most basic CycloneDX SBOM files Supporting Java modules for a start
Provide a basic upload to the Dependency Track server

gamlerhart · 2025-03-21T11:08:03Z

I've these high level questions

I implemented the SBOM json from scratch, so that we can use the UPickle etc and do not add libraries:
The alternative is to use the CycloneDX 'model' library, that mostly implements the JSON and some hashing.
But that then adds this extra library etc: More stuff to download, more stuff in the classpath etc.
So, the question is: What is preferred in general: Avoiding external libraries when possible? Or go for maximum comparability and include more external libraries?
I adding this to the 'contrib' section the right place? The alternative seems to have it as a complete external library. However, that adds extra maintenance burdens: Pushing it to Maven repos, compiling it, versioning it. So, having it in-sync seems better.
I've extended the returned data from the Coursier .artifacts method. I think that is ok, because that method wasn't yet published in 0.12.9?

(update: Android tests are fixed)

lefou

I've these high level questions

I implemented the SBOM json from scratch, so that we can use the UPickle etc and do not add libraries:
The alternative is to use the CycloneDX 'model' library, that mostly implements the JSON and some hashing.
But that then adds this extra library etc: More stuff to download, more stuff in the classpath etc.
So, the question is: What is preferred in general: Avoiding external libraries when possible? Or go for maximum comparability and include more external libraries?

I'd say it depends on how stable the CyconeDX format it. We don't have an issue with downloading additional dependencies, as long as we encapsulate them in an isolated classloader and properly manage their use (e.g. share same classloader for multple modules via a worker module).

But the model added here seems small enough and the benefit of being directly cacheable by Mill as a task result is something I find useful.

I adding this to the 'contrib' section the right place? The alternative seems to have it as a complete external library. However, that adds extra maintenance burdens: Pushing it to Maven repos, compiling it, versioning it. So, having it in-sync seems better.

Contrib is exactly for when you don't want to host it yourself.

I've extended the returned data from the Coursier .artifacts method. I think that is ok, because that method wasn't yet published in 0.12.9?

I'd like to have @alexarchambault thoughts on that. I think we don't want to return tuples from Resolver.artifacts (until we can use named tuples). Maybe we want add a new method with a better name instead.

I added some comments below.

contrib/sbom/readme.adoc

contrib/sbom/src/mill/contrib/sbom/CycloneDXModule.scala

scalalib/src/mill/scalalib/CoursierModule.scala

gamlerhart · 2025-04-04T20:00:37Z

I'll stick with the upickl JSON for a start. That integrates well with Mill eco system. I consider the CyclonDX library only if there is too painful to directly write the Json.

alexarchambault · 2025-04-07T16:40:41Z

main/util/src/mill/util/Jvm.scala

-        case Right(res) =>
-          Result.Success(res)
+        case Right(artifacts) =>
+          Result.Success(ArtifactResolution(resolution, artifacts))


Maybe coursier.Fetch.Result could be used for that, like

Suggested change

Result.Success(ArtifactResolution(resolution, artifacts))

Result.Success(Fetch.Result(resolution, artifacts.fullDetailedArtifacts0, artifacts.fullExtraArtifacts))

And we could rename CoursierModule#artifacts to CoursierModule#fetch then

Used Fetch.Result and renamed.

@alexarchambault Ping. Updated to Fetch.Result

gamlerhart · 2025-05-21T05:07:34Z

Rebased & Tests passing =)

lihaoyi · 2025-08-26T05:37:55Z

@gamlerhart sorry we've overlooked this. Could you rebase this one more time, and flesh out the PR description with a more verbose explanation of what this is all about? I'm not familiar with SBOMs myself and need help understanding what's going on here

gamlerhart · 2025-08-27T18:15:57Z

Hi everyone. I'll try to get to it and rebase it.

@lihaoyi At this stage the PR is the most minimal possible SBOM. The idea is to get the 'foot in the door' for SBOM support. And then see what people actually need and extend it.

Eg. Missing atm:

Having different hash flavors (md5, sha1, sha2 etc) for include library. I added SHA2 only.
Having a report for the build tool itself
Support for non-maven dependencies.

Anyway, that is something to add if there is demand =).

lefou · 2025-08-27T20:48:04Z

@gamlerhart To get an idea, how such an SBOM looks like, could you attach an SBOM generated with this PR for Mill itself?

gamlerhart · 2025-09-02T20:11:22Z

Ok...I'm not sure how to create a SBOM for Mill itself with a dev build:

What I tried:

I've mill checked out in a second location
I add //| mvnDeps: ["com.lihaoyi::mill-contrib-sbom:$MILL_VERSION"] to the build.mill in that directory, to add the contribution library
I then run with this branch /mill dist.run /home/roman/mill-test-checkout resolve dist._

Then I get this error:

============================== resolve dist._ ==============================
[build.mill-60/65] compile
[build.mill-60] [info] compiling 22 Scala sources to /home/roman/dev-temp/mill/out/mill-build/compile.dest/classes ...
[build.mill-60] [error] -- [E046] Cyclic Error: /home/roman/dev-temp/mill/mill-build/src/millbuild/MillScalaModule.scala:12:67 
[build.mill-60] [error] 12 |trait MillScalaModule extends ScalaModule with MillJavaModule with ScalafixModule { outer =>
[build.mill-60] [error]    |                                                                   ^
[build.mill-60] [error]    |                             Cyclic reference involving val <import>
[build.mill-60] [error]    |
[build.mill-60] [error]    |                              Run with -explain-cyclic for more details.
[build.mill-60] [error]    |
[build.mill-60] [error]    | longer explanation available when compiling with `-explain`
[build.mill-60] [error] -- [E46] /home/roman/dev-temp/mill/libs/javalib/package.mill:123:53
[build.mill-60] [error] 123 │  object worker extends MillPublishScalaModule with BuildInfo {
[build.mill-60] [error]     │                                                    ^
[build.mill-60] [error]     │Cyclic reference involving val <import>
[build.mill-60] [error]     │
[build.mill-60] [error]     │ Run with -explain-cyclic for more details.
[build.mill-60] [error] -- [E8] /home/roman/dev-temp/mill/website/package.mill:3:12
[build.mill-60] [error] 3 │import org.jsoup._
[build.mill-60] [error]   │           ^^^^^
[build.mill-60] [error]   │value jsoup is not a member of org
[build.mill-60] [error] -- [E8] /home/roman/dev-temp/mill/runner/meta/package.mill:4:21
[build.mill-60] [error] 4 │import mill.contrib.buildinfo.BuildInfo
[build.mill-60] [error]   │                    ^^^^^^^^^

So, I think my plain dist.run doesn't work with the mill build itself.

lihaoyi · 2025-09-03T02:58:13Z

@gamlerhart since Mill has a meta-build in mill-build/build.mill, you need to add your dependency there instead of in the build header

lefou · 2025-09-03T08:24:29Z

Make a local release of your development snapshot:

> MILL_STABLE_VERSION=1 mill dist.installLocalCache
...
[7152] /home/lefou/.cache/mill/download/1.0.4-37-a03078

Add the plugin to the Mill build

diff --git a/mill-build/build.mill b/mill-build/build.mill
@@ -15,6 +15,7 @@
     // TODO: implement empty version for ivy deps as we do in import parser
     mvn"com.lihaoyi::mill-contrib-buildinfo:${mill.api.BuildInfo.millVersion}",
     mvn"com.goyeau::mill-scalafix_mill1:0.6.0",
-    mvn"org.jsoup:jsoup:1.21.2"
+    mvn"org.jsoup:jsoup:1.21.2",
+    mvn"com.lihaoyi::mill-contrib-sbom:${mill.api.BuildInfo.millVersion}"
   )
 }

Use the just released Mill version via MILL_VERSION:

> MILL_VERSION="1.0.4-37-a03078" mill ...

or edit the build.mill:

diff --git a/build.mill b/build.mill
@@ -1,4 +1,4 @@
-//| mill-version: 1.0.4-26-a6e4c1
+//| mill-version: 1.0.4-37-a03078
 //| mill-jvm-opts: ["-XX:NonProfiledCodeHeapSize=250m", "-XX:ReservedCodeCacheSize=500m"]
 //| mill-opts: ["--jobs=0.5C"]

gamlerhart · 2025-09-03T21:03:49Z

Thanks for the help =). That worked:

The SBOM for Mill itself looks like this:
sbom.json

Or for example when viewing it in a tool like dependency track:

Reminder: It is a foot in the door state:

There are tons and tons of data that can be potentially filled. Depends on the feedback on what people actually need.
Plus: Afaik there is ways to represent sub-modules etc, but the generated SBOM is the flat list. Same: Will see if people actually need that.

From my side: I probably more interested to also get the JavaScript/npm dependencies next. As having JVM backend + NPM in the frontend is so common.

Motivation: In some companies, the development team has to produce Software Bill of Materials (SBOM) for their project for compliance reasons: To track dependencies and licenses across their organisation. Provide a Module that produces SBOMs in JSON format. Changes in the core: Extended the .getArtifact to return the coursier.Resolution as well. This is then used to get the license information. Outside the core: Add a SBOM contrib module - Generate the most basic CycloneDX SBOM files Supporting Java modules for a start - Provide a basic upload to the Dependency Track server

gamlerhart marked this pull request as ready for review March 21, 2025 11:08

gamlerhart force-pushed the sbom-for-java-deps branch from cdc2cdd to b740cdc Compare March 31, 2025 04:02

lefou reviewed Apr 1, 2025

View reviewed changes

gamlerhart force-pushed the sbom-for-java-deps branch from b740cdc to bdb28c2 Compare April 4, 2025 19:56

alexarchambault reviewed Apr 7, 2025

View reviewed changes

gamlerhart requested a review from alexarchambault April 9, 2025 14:04

gamlerhart force-pushed the sbom-for-java-deps branch from cc09aea to e5737a7 Compare April 27, 2025 19:18

gamlerhart force-pushed the sbom-for-java-deps branch from e5737a7 to ff4bd95 Compare May 20, 2025 20:13

gamlerhart requested a review from lefou May 21, 2025 06:47

lefou mentioned this pull request Jun 2, 2025

Support for GitHub dependency graph #5242

Open

lihaoyi force-pushed the main branch 2 times, most recently from 1d3b959 to 3ba698a Compare July 10, 2025 04:00

gamlerhart marked this pull request as draft August 27, 2025 20:08

gamlerhart force-pushed the sbom-for-java-deps branch from ff4bd95 to b770fc6 Compare August 27, 2025 20:20

gamlerhart force-pushed the sbom-for-java-deps branch 4 times, most recently from e68e4df to 006f36c Compare September 2, 2025 19:46

gamlerhart marked this pull request as ready for review September 2, 2025 20:50

gamlerhart force-pushed the sbom-for-java-deps branch from 006f36c to c6d59b9 Compare September 3, 2025 20:57

gamlerhart force-pushed the sbom-for-java-deps branch from c6d59b9 to d30555c Compare September 4, 2025 20:40

	Result.Success(ArtifactResolution(resolution, artifacts))
	Result.Success(Fetch.Result(resolution, artifacts.fullDetailedArtifacts0, artifacts.fullExtraArtifacts))

Uh oh!

Generation Software Bill of Materials (SBOM) #4757

Are you sure you want to change the base?

Generation Software Bill of Materials (SBOM) #4757

Uh oh!

Conversation

gamlerhart commented Mar 20, 2025 • edited by lefou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gamlerhart commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lefou left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gamlerhart commented Apr 4, 2025

Uh oh!

alexarchambault Apr 7, 2025

Choose a reason for hiding this comment

Uh oh!

alexarchambault Apr 7, 2025

Choose a reason for hiding this comment

Uh oh!

gamlerhart Apr 7, 2025

Choose a reason for hiding this comment

Uh oh!

gamlerhart Apr 14, 2025

Choose a reason for hiding this comment

Uh oh!

gamlerhart commented May 21, 2025

Uh oh!

lihaoyi commented Aug 26, 2025

Uh oh!

gamlerhart commented Aug 27, 2025

Uh oh!

lefou commented Aug 27, 2025

Uh oh!

gamlerhart commented Sep 2, 2025

Uh oh!

lihaoyi commented Sep 3, 2025

Uh oh!

lefou commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gamlerhart commented Sep 3, 2025

Uh oh!

Uh oh!

gamlerhart commented Mar 20, 2025 •

edited by lefou

Loading

gamlerhart commented Mar 21, 2025 •

edited

Loading

lefou left a comment •

edited

Loading

lefou commented Sep 3, 2025 •

edited

Loading