-
Notifications
You must be signed in to change notification settings - Fork 5.9k
New issue
Have a question about this project?Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of serviceand privacy statement.We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Vendor protobuf in static builds. #17774
base: master
Are you sure you want to change the base?
Conversation
There are issues detecting a system copy of Protobuf properly on newer versions of Alpine, and we’re functionally bundling whatever copy we link to for static builds _anyway_, so just use a vendored copy instead of a system copy.
Abseil’s build infrastructure is fundamentally broken on non-glibc systems in a number of ways, so borrow the patches used by Alpine/Void/gRPC to resolve these issues.
It was already a required dep for the FLB build, but we now also need it for vendoring Abseil.
This doubles static builds time (ci checks) - 50 => 100 mins 😢 |
I wish there was an alternative, but I was unable to find one. Even bundling a known working copy of the FindProtobuf CMake module and it’s associated requirements wasn’t enough to get things working properly. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Fork the repo, commit the changes and fetch the specific SHA.
Can you please provide concrete reasoning as to why that approach is superior to carrying a patch file here and applying it when fetching the sources other than not requiring a local copy of the patch command? Experience has taught me that we have a rather bad track record of not managing repos used for such an approach well. |
Handling everything in a standardized way. |
Standards are great and all, when the benefits of using them actually outweigh the downsides. And the primary benefit to handling it the way you propose (avoiding dependence on external infrastructure) is of questionable validity for us, because there is no external infrastructure involved in this, and from a realistic perspective anything that may cause the official Abseil repo to suddenly disappear on GitHub is either almost certain to hit all the forks, or is about as likely as us accidentally deleting our own fork, and thus is not particularly relevant to the argument. On the other hand, forking Abseil just for this adds a significant amount of extra complexity to testing and deploying newer versions of it for our build system, makes it more complicated to keep track of what version we’reactuallyusing (which has nasty implications for both security auditing and for just generally keeping things up to date), and is different from how we handle essentially everything else. |
It'sinternalinfrastructure I don't agree with. Relying on CMake's patching, maintaining a custom script to apply said patches, an extra build environment dependency (
Merge from upstream and pushing to our fork?
That's simply not true. The repo/sha will be in the commits that update them. |
This is functionality that has existed in CMake for years and is generally considered perfectly stable. And all that it is internally is a
This could be completely removed in this case if you think it’s really so onerous to maintain a script that just iteratively calls patch on each patch file in the directory of patches one at a time. As far as script maintenance goes though, this is so trivial that the CMake code driving it is more complicated.
If this is a deal breaker we can use That said, this is not an even remotely onerous dependency. It has no external dependencies and is already pulled in on a number of our build environments anyway (and we officially don’t care about end users building Netdata, so you can’t claim them as a problem here).
No, there is reproducible versioning here, the version is exactly equivalent to that of the commit in our repo that was used to generate the code, because every single piece of it is covered by the commit.
Assuming of course that there are no merge conflicts that need specially resolved. And if such conflicts do arise, we kind of need to actually know what the hell we’re doing in detail. And that either forces complicated local testing, or punts testing until after things have been merged. OTOH, by the time we actually get around to updating to a newer version of Abseil, if there are issues with the patch that’s currently in the repo with that version, the upstream copies of it in Alpine, Void, and gRPC will have long since been updated to deal with that, so updating then reduces to copying the updated patch, updating the commit SHA, and then doing any testing without having to change anything else.
Cool, can you tell me exactly what parts ofhttps://github /netdata/libbpfare ours and what aren’t without spending 20 minutes digging through git history? Figuring this info out is possible, but it’s nowhere near as easy as it is if you’re maintaining patches instead of a fork. And thisdoesmatter, quite a bit. |
Summary
There are issues detecting a system copy of Protobuf properly on newer versions of Alpine, and we’re functionally bundling whatever copy we link to for static buildsanyway,so just use a vendored copy instead of a system copy.
This is a prerequisite for updating the base image used for building static builds to Alpine 3.20.
Test Plan
CI passes for this PR.