Skip to content
New issue

Have a question about this project?Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of serviceand privacy statement.We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vendor protobuf in static builds. #17774

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

Ferroin
Copy link
Member

@Ferroin Ferroin commented May 29, 2024

Summary

There are issues detecting a system copy of Protobuf properly on newer versions of Alpine, and we’re functionally bundling whatever copy we link to for static buildsanyway,so just use a vendored copy instead of a system copy.

This is a prerequisite for updating the base image used for building static builds to Alpine 3.20.

Test Plan

CI passes for this PR.

There are issues detecting a system copy of Protobuf properly on newer
versions of Alpine, and we’re functionally bundling whatever copy we
link to for static builds _anyway_, so just use a vendored copy instead
of a system copy.
@github-actions github-actions bot added the area/packaging Packaging and operating systems support label May 29, 2024
Abseil’s build infrastructure is fundamentally broken on
non-glibc systems in a number of ways, so borrow the patches used by
Alpine/Void/gRPC to resolve these issues.
It was already a required dep for the FLB build, but we now also need it
for vendoring Abseil.
@Ferroin Ferroin marked this pull request as ready for review May 30, 2024 14:59
@Ferroin Ferroin requested review from vkalintirisand a team ascode owners May 30, 2024 14:59
@ilyam8
Copy link
Member

ilyam8 commented May 30, 2024

This doubles static builds time (ci checks) - 50 => 100 mins 😢

@Ferroin
Copy link
Member Author

Ferroin commented May 30, 2024

This doubles static builds time (ci checks) - 50 => 100 mins 😢

I wish there was an alternative, but I was unable to find one. Even bundling a known working copy of the FindProtobuf CMake module and it’s associated requirements wasn’t enough to get things working properly.

Copy link
Collaborator

@vkalintiris vkalintiris left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Fork the repo, commit the changes and fetch the specific SHA.

@Ferroin
Copy link
Member Author

Ferroin commented May 31, 2024

Fork the repo, commit the changes and fetch the specific SHA.

Can you please provide concrete reasoning as to why that approach is superior to carrying a patch file here and applying it when fetching the sources other than not requiring a local copy of the patch command?

Experience has taught me that we have a rather bad track record of not managing repos used for such an approach well.

@vkalintiris
Copy link
Collaborator

Can you please provide concrete reasoning as to why that approach is superior to carrying a patch file here and applying it when fetching the sources other than not requiring a local copy of the patch command?

Handling everything in a standardized way.

@Ferroin
Copy link
Member Author

Ferroin commented Jun 3, 2024

Handling everything in a standardized way.

Standards are great and all, when the benefits of using them actually outweigh the downsides. And the primary benefit to handling it the way you propose (avoiding dependence on external infrastructure) is of questionable validity for us, because there is no external infrastructure involved in this, and from a realistic perspective anything that may cause the official Abseil repo to suddenly disappear on GitHub is either almost certain to hit all the forks, or is about as likely as us accidentally deleting our own fork, and thus is not particularly relevant to the argument.

On the other hand, forking Abseil just for this adds a significant amount of extra complexity to testing and deploying newer versions of it for our build system, makes it more complicated to keep track of what version we’reactuallyusing (which has nasty implications for both security auditing and for just generally keeping things up to date), and is different from how we handle essentially everything else.

@vkalintiris
Copy link
Collaborator

Handling everything in a standardized way.

Standards are great and all, when the benefits of using them actually outweigh the downsides. And the primary benefit to handling it the way you propose (avoiding dependence on external infrastructure) is of questionable validity for us, because there is no external infrastructure involved in this, and from a realistic perspective anything that may cause the official Abseil repo to suddenly disappear on GitHub is either almost certain to hit all the forks, or is about as likely as us accidentally deleting our own fork, and thus is not particularly relevant to the argument.

It'sinternalinfrastructure I don't agree with. Relying on CMake's patching, maintaining a custom script to apply said patches, an extra build environment dependency (patch), source code with no versioning. It also changes the way we'll have to work with code. We usegitfor tracking code changes.

On the other hand, forking Abseil just for this adds a significant amount of extra complexity to testing and deploying newer versions of it for our build system

Merge from upstream and pushing to our fork?

makes it more complicated to keep track of what version we’reactuallyusing (which has nasty implications for both security auditing and for just generally keeping things up to date), and is different from how we handle essentially everything else.

That's simply not true. The repo/sha will be in the commits that update them.

@Ferroin
Copy link
Member Author

Ferroin commented Jun 4, 2024

Relying on CMake's patching,

This is functionality that has existed in CMake for years and is generally considered perfectly stable.

And all that it is internally is aadd_custom_command()call that ensures the specified patch command gets called at the right time.

maintaining a custom script to apply said patches,

This could be completely removed in this case if you think it’s really so onerous to maintain a script that just iteratively calls patch on each patch file in the directory of patches one at a time. As far as script maintenance goes though, this is so trivial that the CMake code driving it is more complicated.

an extra build environment dependency (patch),

If this is a deal breaker we can usegit applyinstead. Doing so would break some cases of a pre-populated source tree though, and it’s actuallylessreliable than callingpatchbecause it’s impacted by git configuration.

That said, this is not an even remotely onerous dependency. It has no external dependencies and is already pulled in on a number of our build environments anyway (and we officially don’t care about end users building Netdata, so you can’t claim them as a problem here).

source code with no versioning.

No, there is reproducible versioning here, the version is exactly equivalent to that of the commit in our repo that was used to generate the code, because every single piece of it is covered by the commit.

Merge from upstream and pushing to our fork?

Assuming of course that there are no merge conflicts that need specially resolved. And if such conflicts do arise, we kind of need to actually know what the hell we’re doing in detail.

And that either forces complicated local testing, or punts testing until after things have been merged.

OTOH, by the time we actually get around to updating to a newer version of Abseil, if there are issues with the patch that’s currently in the repo with that version, the upstream copies of it in Alpine, Void, and gRPC will have long since been updated to deal with that, so updating then reduces to copying the updated patch, updating the commit SHA, and then doing any testing without having to change anything else.

That's simply not true. The repo/sha will be in the commits that update them.

Cool, can you tell me exactly what parts ofhttps://github /netdata/libbpfare ours and what aren’t without spending 20 minutes digging through git history?

Figuring this info out is possible, but it’s nowhere near as easy as it is if you’re maintaining patches instead of a fork. And thisdoesmatter, quite a bit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/packaging Packaging and operating systems support
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants