Rework the protobuf ecosystem

[suhailskhan]

Description

Current State of Protobuf in MacPorts

protobuf3-cpp currently represents the de-facto Protocol Buffers distribution in MacPorts. It underpins a relatively wide array of ports as it is a common build dependency. Protobuf is also commonly used in the development of networking-heavy software.

The last release of Protobuf 3, and the release that this port is pinned at, is v21.12, which dates back to December 2022. The grpc port cannot be updated to the latest release because protobuf3-cpp is not a suitable dependency. The same is true of protobuf-c, a port of a project independent of Protobuf. These two ports themselves are dependencies for a relatively large set of other ports. Also, dependencies aside, from the perspective of developers that use Protobuf for the development of their own software, the absence of a convenient port similar to protobuf3-cpp that actually tracks the latest release, may be somewhat of a pain point which is avoidable. The core issue that this PR aims to solve is the issue of moving away from this aging Protobuf port, and into a new default Protobuf port that does not need to be pinned at a certain version.

There is also protobuf3-cpp-upstream, which is meant to track the latest release, but it is installed under ${prefix}/local/libexec/protobuf3-cpp-upstream, and is not used as a dependency for those ports that depend on Protobuf. protobuf5-cpp was recently introduced, and is meant to be pinned at Protobuf 5 (v26-v29 releases). It too, is installed under ${prefix}/local/libexec.

Aside from these, there is protobuf-cpp, which is a port of Protobuf 2, which should now be considered legacy.

protobuf-cpp and protobuf3-cpp have corresponding ports containing bindings for programming languages other than C++, such as py-protobuf/py-protobuf3, protobuf-java/protobuf3-java, and others. I had decided that dealing with these ports falls outside of the scope of this PR, except for the two Python ports, which in many cases are strongly intertwined with the C++ ports. For instance, these ports depend on protobuf-cpp/protobuf3-cpp, and there are a few ports that depend on both the Python and C++ ports together.

Proposal

The first issue to tackle is the reworking of the main Protobuf ports themselves. While we are at it, this is an opportunity to rework the naming conventions for these ports; more on that later.

Again, here are the current ports in the tree:

• protobuf-cpp - pinned at v2.x.x
• protobuf3-cpp - pinned at v3.x.x (v3.21.12, equivalent to v21.12 from the releases page)
• protobuf3-cpp-upstream - tracks latest release, although out-of-date for a few months (v3.29.2; should have been called v5.29.2 according to the docs; the 3 is just hardcoded in the portfile, and in the name of the port as well)
• protobuf5-cpp - pinned at v5.x.x (last 5.x.x release will be 5.29.x, equivalent to v29.x from the releases page, hence it being pinned in this manner)

The first two among these are installed in ${prefix}, while the rest are in ${prefix}/libexec/${name}.

Now, here is what will happen with the merging of this PR:

• protobuf-cpp -> protobuf2-cpp - move to libexec
• protobuf3-cpp - no change in name; move to libexec
• protobuf3-cpp-upstream no need for this anymore
• protobuf5-cpp -> protobuf29 - switch to vX.x versioning like protobuf
• protobuf - new port; tracks latest release, uses vX.x versioning instead of vX.x.x; installed in ${prefix}

The primary reason we are treating the naming and versioning of protobuf differently from the ports we currently have, is that it brings about a great deal of simplicity. The C++ implementation, for all intents and purposes, is the Protobuf—the non-C++ ports just contain language bindings. Thus, we can drop the “cpp” from the name. This also simplifies things from a maintainer’s perspective, as there is no need to manually prepend the version number of the C++ bindings in protobuf’s portfile (e.g. v3 for protobuf3-cpp, or v5 for protobuf5-cpp).

As for the Python bindings, py-protobuf will be moved to py-protobuf2, and py-protobuf will track the latest release of the Python bindings from PyPI. The idea is to delete py-protobuf2 and py-protobuf3 from the tree, and thus keep only the new py-protobuf, as it currently appears that virtually all dependent ports build against this new port without issue. This is currently pending, and will either happen as part of this PR or in the near future after it is merged. On this same note, the same should be done for protobuf2-cpp and protobuf3-cpp, but an ideal timeline for that is a little less clear at the moment.

After doing the above, we can finally swap the protobuf3-cpp dependency used in many of its dependent ports for protobuf, and update grpc and protobuf-c to their latest releases.

A handful of these dependency swaps were moved to new PRs because of how much time it would have taken this PR’s CI checks to finish, causing them to hit their 6-hour time limit before they could. These PRs are as follows:

• python ports: use protobuf and/or py-protobuf #28442
• atuin: use protobuf #28444
• ports: update references to protobuf in comments #28445
• protobuf-c: update to 1.5.2; use protobuf #28447
• apache-arrow: use protobuf #28448
• qgis3: use protobuf #28449
• opencv4: use protobuf #28450
• qt513-qtwebengine: use protobuf #28454
• mysql8: use protobuf #28485

In summary, protobuf is the way to install Protocol Buffers and the protoc compiler with MacPorts, and ports that depend on Protobuf should use this port as a dependency.

Type(s)

• bugfix
• enhancement
• security fix

Tested on

macOS 15.4.1 24E263 arm64
Xcode 16.3 16E140

Verification

Have you

• followed our Commit Message Guidelines?
• squashed and minimized your commits?
• checked that there aren't other open pull requests for the same change?
• checked your Portfile with port lint?
• tried existing tests with sudo port test?
• tried a full install with sudo port -vst install?
• tested basic functionality of all binary files?
• checked that the Portfile's most important variants haven't been broken?

[skip notification]

[mascguy]

The intent of this PR is as follows:

• Move protobuf-cpp to protobuf2-cpp and segregate it; update dependent ports to accommodate for the change
• Segregate protobuf3-cpp
• Create a new, unsegregated protobuf-cpp port that tracks the latest stable release, replacing the segregated protobuf3-cpp-upstream port
• Ports that currently depend on protobuf3-cpp most likely should work with the the new protobuf-cpp port, so modify these ports to use it instead of protobuf3-cpp

So far, this PR only implements the first bullet point. The remaining ones are only planned, and have not been done yet. This is only a proposal for one possible approach, I am open to scrapping this and taking a different approach based on feedback.

I definitely support this.

One of our older tickets has plenty of details regarding the challenges for some of our older ports:

https://trac.macports.org/ticket/57117

But as long as we take our time, and work through the details, this would be a nice improvement.

I'm going to try and pull these changes down over the next few days, and review how everything fits together.

And finally, thank you so much for taking the initiative on this!

[suhailskhan]

By the way:

I had initially begun by opting to make the least possible changes to it, and had not added a conflict declaration, until we determine whether or not this port should even stay.

It seems I must have forgotten that py-protobuf3 and the new py-protobuf port actually do have conflict declarations for each other. See here and here.

[mascguy]

I am indeed on the west coast. Prospectively, I should be available during those time slots. Worst case, we would need to push it to next weekend, during which I should also be available. But Tuesday or Thursday evening/night should not be an issue.

How do you suggest we coordinate? Perhaps through the IRC channel?

IRC might work. But this week might be a problem, based on the latest update from the macports-dev mailing list:

From: Ryan Carsten Schmidt
To: macports-dev
Subject: Re: Buildbot offline due to severe weather
Date: Mon, 9 Jun 2025

Now another storm is approaching and I've shut buildbot down again preventatively. Storms are forecast every day for the next six days so there will probably be more downtime ahead.

[suhailskhan]

On Jun 9, 2025, at 04:45, Ryan Carsten Schmidt wrote:

Now another storm is approaching and I've shut buildbot down again preventatively. Storms are forecast every day for the next six days so there will probably be more downtime ahead.

And we're back.

[suhailskhan]

Buildbot was taken down again since my last comment, but apparently it is up now and will stay up without interruption.

On Jun 12, 2025, at 00:58, Ryan Carsten Schmidt wrote:

Buildbot is going down again. Rain is forecast for the next six hours.

Buildbot is back online and there's no further rain in the forecast.