Examine performing compilation against API JARs

[Sipkab]

As suggested on Reddit, the compilation performance could be improved by compiling against API JARs instead of full implementation JARs.

This can be achieved by taking the impl JARs and performing API extraction on them. Every non-anonymous class should be in it (even private ones) and every non-private members. Module-info and package-infos as well.

The private and package-private classes should be present, as there may be declarations that expose them as:

public class Main {
	private static class Inner {

	}

	public static <T extends Inner> T f() {
		return null;
	}
}

In the second iteration of the issue, we may perform deep analysis of the accessability of non-public elements, however, it may be unfeasible, as the analysis may incure greater performance cost than the improvement.

Solution

This shouldn't be performed by the compiler task itself. This is the responsibility of the code that sets up the input classpath. Therefore a solution could be to introduce a new build task that performs the API extraction. Something along the lines of:

saker.java.compile(
    ClassPath: task.that.extracts.api(lib/mylib.jar)
)

Another thing that could be examined is if repositories are able to directly provide compilation input JARs. If a JAR is uploaded to some common repository, it could perform the API extraction themselves, and provide it alongside of the implementation JAR. Therefore, if the client wants to compile for the given JAR, it could download the API instead of the implementation. However, if they want to use it, they can download the implementation as usual.

This issue may be moved from this repository.

[sviperll]

As a follow up from the reddit discussion I'd like to add. That

• yes, this feature is about class path and compilation class path only
• API-jars can be generated with ASM from real ones
• compilation class path should include API-jars for direct dependencies only without transitive dependencies (since API-jars should not depend on anything not exposed in the API and if a dependency is exposed in API it has to be a direct dependency for API to be usable)
• API-jars should be cached in the repository, at least they should be cached in local cache (like $HOME/.m2 for Maven)

[Sipkab]

with ASM from real ones

Definitely. We already use ASM to perform other tasks in the build system so this should be easy.

API-jars for direct dependencies only without transitive dependencies

Note that determining the set of JARs that need to be present for the compilation is not the scope of this issue. This is the responsibility of the dependency resolution algorithm/repository. E.g. Maven, or saker.nest

For saker.nest, we already do this by making it possible to declare a dependency compile-transitive: false. This way, if you want to retrieve only the API JARs for a given classpath, the non-API JARs will be excluded. More info: https://saker.build/nest.repository.support/doc/dependencyresolution/filters.html#compile-filter

For Maven, I'm not sure at the moment how it handles this use-case, maybe something with dependency scopes?

API-jars should be cached

I agree in this. It makes no sense to perform API extraction in every build. There are two scenarios that come up:

1. Clean builds

For clean builds, the API extracted versions of a JAR should only be used if it is directly available from an already populated cache. If it is not, then the extraction shouldn't be performed as part of the build, as that would most likely only slow it down.

2. Incremental builds

For incremental builds, the extraction can be performed, and cached in the build directory. In the next builds, it will be faster, as the previously made JAR can be reused without performing the whole operation again.

Repository caching

The repositories could make the API versions of a JAR available by performing the extraction ahead of time, or on-demand, but caching them in the repository somewhere.

like $HOME/.m2 for Maven

I personally wouldn't be a fan of caching the extracted JARs in the Maven repository. It may or may not disturb other build tools in some way that we don't forsee. I think caching the JARs in a saker.build specific location is more appropriate.

[sviperll]

For incremental builds, the extraction can be performed, and cached in the build directory. In the next builds, it will be faster, as the previously made JAR can be reused without performing the whole operation again.

There is one more angle to look into this. API-jars can be sometime identical between artifact's versions. I don't know personally if this use-case is worth catering for, but in case when a user updates a dependency version and the new version has the same API-jar as the old version we can prevent rebuilding anything.

[Sipkab]

if this use-case is worth catering for

It is. We already perform some compilation avoidance based on the signatures of the classpath classes, and this use-case integrates well in the current implementation of the Java compiler task.

It should be noted that this is a rare use-case. I think it is acceptable (and sometimes advisory) to perform a clean build when some dependency is updated for the project. Incremental builds tend to shine the most when the changes are mininal, while updating a dependency is considered to be a greater one IMO.