[squid-dev] RFC submodule repositories

[Alex Rousskov]

On 8/1/22 22:32, Amos Jeffries wrote:
> On 1/08/22 03:09, Alex Rousskov wrote:
>> On 7/31/22 00:29, Amos Jeffries wrote:
>>> When PR #937 merges we will have the ability to shuffle old helpers 
>>> into a separate repository that users can import as a submodule to 
>>> build with their Squid as-needed.

>> What will we optimize by introducing such a significant
>> complication as git submodules?


> ( So every change to Squid has to be justified as an optimization now. 
> Right... )

The verb "optimize" is generally defined as "to make the best or most 
efficient use". In computing, that definition is relaxed to mean any 
improvement (not necessarily the "best"). Both definitions would work in 
this context, but I used "optimize" in the latter ("to improve") sense. 
I hope we can all agree that Squid changes should improve Squid.


> We "optimize" future code maintenance workload with the ability to drop 
> outdated helpers which are still required by a small group of users but 
> no longer want to be actively developed by the core dev team.

N.B. Unless something else changes, simply moving "dropped" helper code 
to a submodule would require reviewing dropped helper update PRs in an 
aggregate "take it or leave it" mode. No dropped helper code change 
requests would be appropriate for such "updated dropped helper H to 
version V" pull requests. I do not know whether that will improve core 
developer lives, but it is likely to require making more judgement calls 
than we have to make today, and jusdgement calls can be especially 
controversial/painful.

How would moving "dropped" helper code to a submodule help core 
developers? AFAICT, just moving a dropped helper into a submodule 
creates more work for core developers because they now have one more 
thing to worry about. Git submodules (and git in general!) have very 
little impact on making "./bootstrap.sh && ./configure && make && sudo 
make install" produce the desired outcome -- with or without submodules, 
we still need to make the whole thing build and work correctly. Thus, 
any improvements must be rooted in some other, yet unspecified Project 
changes besides moving some helper code into submodules. What other 
changes will make that extra core developer work worth it?

In other words, it is possible that the RFC associates "git submodules" 
with some unspecified changes outside the Squid git repository 
structure[1] that RFC readers cannot guess. Identifying any important 
changes the RFC associates with git submodules may help us make progress.

[1] For example, one common submodule use case is having different 
users/permissions for different parts of the Project -- essentially 
having multiple core developer teams with varying levels of access to 
critical Project resources, but all collaborating under one Project 
roof. It is often impractical to achieve that kind of access controls 
within one git/GitHub repository. I am not saying this example applies 
to this RFC -- I am just illustrating how submodules may facilitate 
important changes outside of low-level git mechanisms themselves.


> Case in point being CERN who still require our bundled LanManager 
> helper(s) for some internal needs. That requires a lot of deprecated and 
> rarely touched code being maintained for only one (albeit important) 
> user case.
>   That code could all be shuffled to a separate repository outside the 
> official Squid release, but maintained by developers that support CERN 
> needs.

Sure, but we clearly do not need git submodules to "shuffle LanManager
helper(s) code to a separate repository". Why submodules? The RFC 
implies, without elaborating, that submodules are the right solution to 
some problem. Several RFC reviewers correctly point out that submodules 
are associated with significant "cons". I am asking what the "pros" are. 
It is a very reasonable request IMO.

Agreeing on the _problem_ definition may also help shorten the number of 
iterations required to reach consensus. What do we really mean when we 
say that "dropped" helpers are "not actively developed by core 
developers"? To me, the lack of core developer involvement implies that 
PRs against Squid master may break "dropped" helpers and that "dropped" 
helper needs are assigned much lower weight compared to "native" Squid 
improvement needs, but others are likely to interpret that phrase 
differently.


>>> What (if any) updates do we need to make to Anubis and other 
>>> infrastructure so support git submodules ?
>>
>> That answer probably depends, in part, on what support guarantees we 
>> are going to issue for those submodules and on what our submodule 
>> update policy will be.

> IMO we should maintain at least one helper officially as a submodule to 
> ensure the git submodule mechanisms remain viable for distributors as a 
> modern replacement for the old squid-2 way of building their custom 
> helpers.

The above paragraph does not supply the information required for me to 
answer your original question. To make progress, let's temporary assume, 
for simplicity sake, that all "dropped" helpers are moved to "officially 
maintained" submodules. Now, will "./bootstrap...make install" sequence 
be allowed to ignore submodule existence? Ignore build failures related 
to submodules? Treat submodule code specially in some other way?

If the answer to those questions is "no", then the automation/CI changes 
(that I can think of) would revolve around adding commands to fetch 
submodules before running that sequence (e.g., "git submodules 
update..."). Doing so should be easy in our current Semaphore CI. I am 
not sure about Jenkins.

Alex.