[squid-dev] [RFC] The situation with helpers/

[Amos Jeffries]

On 14/01/2016 4:43 a.m., Alex Rousskov wrote:
> On 01/13/2016 06:12 AM, Kinkie wrote:
>> On Tue, Jan 5, 2016 at 6:01 PM, Alex Rousskov
>> <rousskov at measurement-factory.com> wrote:
>>> On 01/04/2016 08:58 PM, Amos Jeffries wrote:
>>>> The SBuf and CharacterSet patch auditing has brought up the questions of
>>>> why helpers/ is separate from src/, built first, what cross-dependency
>>>> there can be, and even why we are bundling C++ helpers to begin with.
>>>>
>>>> So lets clarify it formally once and for all.
>>>>
>>>> Q1: why is helpers/ built before src/ ?
>>>
>>> A1: Primarily for historical and paranoia reasons.
>>>
>>>
>>>> Q2: why is helpers/ separate from src/ ?
>>>
>>> A2: Primarily for historical and paranoia reasons.
>>>
>>>
>>>> should we shuffle them back to src/*/helpers/ directories on a
>>>> per-component basis ?
>>>
>>> Yes w.r.t. src/. There is no known good reason to enact artificial hard
>>> barriers between parts of Squid code. All Squid code should be in src/.
> 
>> Could or could not. For sure it would help with dependency tracking
>> where needed, and there are cases where it is needed.
> 
> Yes, that is one reason I said "should", not "could".
> 
> 
>>>> * it does somewhat hinder anyone wanting to manually build a single
>>>> helper. Though I have not heard of anyone doing so in many years.
>>>
>>> The move does not preclude anyone from building a helper. The fact that
>>> a few more things may be built along the way is a minor detail IMO.
>>> Patches to improve dependency tracking (by migrating away from recursive
>>> Makefiles??) are welcomed.
>>
>> Ugh. see src/Makefile.am for the monster file this would yield.
> 
> And see src/Makefile.am for the monster file the current
> lets-track-complex-dependencies-manually approach has created. I do not
> know exactly what the best solution to the current src/Makefile.am
> problem is. I know that is a big problem. I suspect the solution
> involves more than simply migrating away from recursive Makefiles. I do
> not want to discuss that problem in this thread. That is why I said
> exactly what I said, including the question marks.

Small side-track (sorry):

My experiments in fixing that makefile dependencies situation have
resulted in a few conclusions. Though nowhere near complete the
lp:~yadi/squid/makefile-deps branch has the latest version of what I
think it should look like.

* First stage, bar none is to simply stop adding new foo_DEPENDENCIES
lines to any .am file. We can/should/need all do that right now.

* Second stage is to remove existing *_DEPENDENCIES where possible. I
have found that needs to be done with care, bulk erasing or moving the
entris to LDADD lists is not always correct. The branch did it and a few
test binaries had inconsistent builds until their SOURCES were
corrected. Still not sure I got them all yet.

* Third stage is to reduce the SOURCES and LDADD lists to the actual
minimal set needed (maybe increase to full set needed). Stubs helps
there a lot, but a) some new ones are needed, b) some existing are
incorrect, and c) more/better use of the ones we have improves automake
results.

* Fourth stage I am experimenting with in the branch now is
non-recursive unit tests. By using src/*/UnitTest.am files we can have
the tests defined in their respective sudir, but actually built and run
non-recursive by the src/Makefile.

It is taking a lot of slow work to make happen. But so far shown some
good results with smaller unit tests and better dependency tracking
overall despite the mixed non-/recursive layout.

automake itself is migrating upstream to a model where it assumes
recursive Makefiles and deals better with those sometimes than
non-recursive. eg. the whole subdirs feature in 1.12+. So I do not think
moving away from recursive matters.

IMO the recursive design is better. We have to take care where we put
things in either case. But with recursive we can reduce the focus to
only the dirs, not every individual file in the sources.


> 
>>>> * it does mean we should probably discard the goal of having the helpers
>>>> be demo code examples for people to base their custom helpers on.
> 
>>> The location of the helper code is pretty much irrelevant for the above
>>> goal. However, if we want to provide demo helpers that are easy to
>>> understand and customize, then we probably should not use C++ to write
>>> those demo helpers.
> 
>> I don't agree. We should use whatever tool makes sense, depending on
>> the availability of libraries, complexity, volunteer time etc.
> 
> Not sure what you do not agree with:
> 
> * Your statement does not contradict what I said about the location of
> the helper code being pretty much irrelevant.
> 
> * If you are focusing on the second part of my statement, then please
> note that it is possible, perhaps even likely that the "tools" that
> "make sense" in our situation are not appropriate for some goals. When
> that happens, the goals have to be re-evaluated to match reality so that
> unrealistic goals do not misdirect our decisions.
> 
> 
>>> * If we want to provide the smallest set of default helpers for the
>>> widest possible set of deployment environments, then internal
>>> dependencies are preferred to the external ones and, hence, C++ is the
>>> best language choice.
>>>
>>> * If we want to provide default production-quality helpers for the most
>>> demanding deployment environments, then internal dependencies are
>>> irrelevant, and C++ is the best language choice. Please note that I do
>>> not know whether it is wise to declare performance the primary goal,
>>> given the performance deficiencies of the helper interface itself and
>>> the current Squid Project resources.
>>> Any open source helper can be examined, customized, and cloned, of
>>> course. Our decision with regard to programming language and
>>> dependencies may make it easier or more difficult, but not impossible.
>>> It is a question of the primary optimization target: ease of
>>> understanding/customizability versus supported platforms versus performance.
>>
>> I believe our goals should be, in descending order of importance:
>> 1- cover a reasonable set of needs
>> 2- provide a gallery of examples for administrators who have special
>> needs not covered by the provided set of helpers. 
> 
> Goal #2, as stated, is not a goal but a side-effect: Whatever helpers we
> provide to satisfy goal #1 will automatically become a "gallery of
> examples".
> 
> 
>> Corollaries of this objective are:
>> 2a- bundled helpers should aim at being as readable as possible.
> 
> Any code should aim at being as readable as possible so this statement
> is not a corollary and does not really bring anything new to the debate.
> 
> 
>> 2b- bundled helpers should try to avoid relying on Squid internal code
>> within reason. We can expect administrators embarking in the task to
>> be competent programmers, asking them to be also competent Squid
>> coders is maybe too much to ask.
> 
> Relying on Squid libraries/APIs is not that much different than relying
> on external libraries/APIs. Thus, I disagree that a reasonable use of
> Squid code in helpers forces helper writers to become competent Squid
> coders. It is a question of how much use is "reasonable" given our
> still-being-debated goals, of course.
> 

Our Squid code is somewhat warped by its own transitional nature.
Improving, but not even close to being done yet. While competence in the
Squid code style may not be necessary in all cases, it usually is not as
easy to deal with as other code might be, just because of its
Squid-specific oddness.

I am more worried about the volatile nature of it though. Linking the
helpers to src/* means they are now even more dependent code that needs
to be managed whenever things are touched.

So the definition of "reasonable use" is one that has a lot of
importance. That is why I raised the "heavyweight" vs other Squid
components distinction earlier.

Amos