[squid-dev] Strategy about build farm nodes

[Alex Rousskov]

On 5/17/21 3:32 PM, Francesco Chemolli wrote:
> On Mon, May 17, 2021 at 8:32 PM Alex Rousskov wrote:
> 
>     On 5/17/21 2:17 AM, Francesco Chemolli wrote:
>     > $ make all push
> 
>     Does that "make push" command automatically switch Jenkins CI to using
>     the new/pushed containers? Or is that a separate manual step?

> Right now that's automatic.
> I'm pondering the best way to have a staging step; Docker supports
> versioning of images but we're using that feature to implement
> multiarch. In order to rely on that I'll need to change the naming
> conventions we rely on.

I think you should be free to use whatever versioning/naming scheme you
think works best for supporting safe Jenkins CI upgrades.

Ideally, the new docker images (and any other changes) should be tested
(against official branches) _before_ the official tests are switched to
use them. I hope you find a way to implement that without a lot of work.
If you need to tap into Foundation resources, please say so!


> The overarching objectives are:
> - focus on where most users probably are
> - allow iterating quickly giving some signal on tracked branches
> - allow landing quickly with more signal
> - be slow but exhaustive on every other platform we can cover. Do not
> block landing new code on less common or harder to test on platforms,
> but still rely on devs to own their changes there, too.

What exactly do you mean by "own their changes there"? The rest is
general/vague enough to allow for many interpretations I can agree with,
but that last bit seems rather specific, so I wanted to ask...


>     I think that full round of PR tests (i.e. one initial set plus one
>     staging set) should not exceed ~24 hours, _including_ any wait/queuing
>     time.

> Is everyone okay with such a slow turnaround?

Well, you should not overload the question by calling a 24-hour delay
for an all-green signal slow :-).

It is all relative, of course: Most Squid PRs take a lot longer to
review and fix. There are open source projects where PRs wait for a week
or more, so 24 hours for a full round (and usually just an hour or two
for the first signal!) would feel very fast for them. Etc.

Yes, 24 hours is not "interactive", but it is acceptable for the vast
majority of PRs AFAICT, and you are providing dockers for those who want
to test in a more interactive mode.


>     I think you are proposing to make some tests optional. Which ones?
> 
> 
> Not the tests, but the platforms.

>From my test results consumer point of view, they are
[platform-specific] [build] tests.


> Things like gentoo, fedora rawhide, freebsd, openbsd.
> Testing is slower there, so results will lag and we need to batch,
> running a full test suite every week or so

OK, so you propose to make slow Jenkins platform-specific tests (i.e.
Jenkins tests on platforms where tests run slowly today) optional and
those platforms are gentoo, fedora rawhide, freebsd, openbsd. Got it!

What happens when such a slow test fails and the PR author ignores the
failure of that optional test and/or the PR is already closed? The
answer to that question is the key to evaluating any proposal that
declares any tests optional IMO...


>     FWIW, I do not think reducing the number of #ifdefs will solve our
>     primary CI problems.


> Each test build is right now 4 full builds and test suites:
> autodetected, minimal, maximum, everytthing-in-except-for-auth

And all of that takes less than 10 minutes on decent hardware which we
can rent or buy! To me, it feels like we are creating an unsolvable
problem here (i.e. testing a lot of things quickly on a raspberry Pi)
and then trying extra hard to solve that unsolvable problem. We should
either stop testing a lot of things or we should use appropriate
resources for testing a lot of things quickly.

Yes, we can reduce testing time by removing #ifdefs, but that is a lot
of work and does not really scale. #ifdefs should be removed primarily
for other reasons.


Cheers,

Alex.