[squid-users] Store-ID documentation could be a little clearer.

[Amos Jeffries]

On 24/11/2015 1:38 p.m., 1508 wrote:
> Hello,
> 
> Thank you for your replies.  I spent a long time typeing this and I would be
> grateful if you can read it all at least twice slowly before sending a
> reply.
> 
> A reminder... Give yourself a smug smile if you find a spelling mistake, my
> screen reader is used to my Typonese and my seeing eye dog can't proof read.  
> 
> Yes, I am almost blind but not daft... 
> 
> I also said 
> 
> I am not trying to pick any holes... You both are far cleverer than me. Vi
> is rocket science, Nano is my friend. I am trying to establish some facts to
> make an accurate bit of documentation... I want to do something to pay back
> many peoples efforts.
> 
> Anyway, E (Sorry I cant type the rest of your name forgive me), I looked at
> your article you found on google. I prefer the man pages first then the
> programs web pages and documentation. Bear in mind I use a screen reader and
> it takes ages to listen to stuff.
> 
> 
> I would like to create a working example so I intend to use the sourceforge
> example in the database. Id pick something that is reproduceable from
> Sourceforge to help the new user check the database and script are working.
> 

I think I get what you are trying to do. But we do things a little
differently in the Squid documentation.

Features/ wiki pages are for documenting the Squid feature and teaching
people about it. *NOT* providing working configurations. There are
usually just too many moving parts for the latter.

We have ConfigExamples/ wiki pages for narrow configuration how-to's
like I think you are proposing.

As far as I can see the Features/StoreID page already contains the full
and accurate information aboute StoreID feature itself. It may appear to
be missing a lot of info about setting up helpers, but that is because
this is a plugin interface feature.

The helpers and everything about setting any of them up is unrelated to
StoreID itself. The expectation is that there would be a helper for
every type of DB or storage engine anyone can dream up for putting the
StoreID data into, or for generating and calculating it on the fly.

> 
> Amos, I am not being critical, one article you gave me said database entries
> are separated by whitespace, the man page says:
> 
>  so I went with the man page.

There seems to be a misunderstanding here.

The Features/StoreID/DB page contains complete and accurate information
about the patterns registry we run. This is a simple set of pattern
pairs which are known to be checked and confirmed safe to use. It is
also *just* a flat-file DB. Many different helpers could use it or the
info provided.

The helper we provide is intended to be capable of reading the data D/L
from the wiki, if not there is a bug to be resolved. Possibly in the
helper docs or its internal regex. But it is not restrcted to those
datasets, nor required to use them.

In the general case, it is expected that the wiki pattern sets be
transformed into whatever DB format the helper being used needs. So the
wiki documents what format the examples displayed take, not what formats
it could potentially be mapped to. (Also partially in a way to clarify
what the HTML mangled view of the datases should be read as, you will
notice the line wrapping gets screwed up in the web view).

The helper-specific man page should document what format the DB used by
that helper takes. It is best to ensure that custom entries being added
for the helper DB meet the relevant helpers DB format. Even if D/L'ing
the dataset(s) from our wiki, for use in our provided flat-file helper.

> 
> Now I know configuration files can be anywhere depending on what you are
> running, windoze or Linux or coal fired abacus.  The thing is it would not
> matter where the files are if we know the filenames and give an example on
> how to find them eg:  in windows or  in Linux.
> 
> So the outline of my bit of documentation would be along the lines of (it is
> not set in stone just something I cobbled up on my brailler) 
> 

The Features/StoreID page is the central documentation for introducing
the feature and describing all this. I've split your descriptive test
and referenced what we have. It you notice carefully the orders match.

PS. the overall layout is a templated style, so all Feature pages should
have the same section layout to make learning Squid features an easy-ish
process (though older ones need updating sometimes).

> You want to use StoreID. 

Firstly we document what StoreID *is* (the "Details" section), what it
does and what the pros and cons of using it are (the "Known Issues"
section).
 No assuming they already know and want it. ToC is available if they
want to skip that part.

> OK you will need a few things like Squid,

This is assumed, the reader is on the Squid website reading about Squid
functionality/features. They may not already have Squid, but assume they
are fully aware that it will be needed.

> perl 

Perl is a basic system requirement of having Squid installed. No need to
state that.

NP: It is also not a requirement of StoreID, but of the specific helper.
Other helpers exist ...

> the rewrite script, and a database file

The "Available Helpers" section, lists both the Perl helper, with its
proper name and lists its DB requirement, with reference to where to D/L
the *many* different available DB files we have already.

It also lists references to other helpers that exist (Ruby ones in this
case).


NP: we then address a common mistake about StoreID being able to cache
YouTube content. Once upon a time it could, nowdays it can't do so
directly. YT devs changed their system.
 [ the text there is a bit confuzled and could do with an edit to
clarify its meaning and grammar. ]


> and an entry in the squid.conf file.

The "Squid Configuration" section.

Note though that being in Features/* page this is a generalized example
designed to explain the directives involved with this feature. It often
uses directives unnecessarily in order to display usage. cut-n-paste
ability is just a bonus, but even then doing so will usually leave one
with a overly-complex configuration that does not work very well.

To repeat:
  Features/* is educational material not run-time configuration.


> You can find the rewrite script by doing ...... command on linux or .... on
> Windows (or .... on another OS if somebody has the command to tell me.) 

There are dozens of ways to do this per-OS. Squid has hundreds of
features. Multiply.

Repeating all that info in each Feature (or ConfigExample) description
page is a huge amount of text and very rarely has anything to do with
the feature or config itself. Usually its needed for Troubleshooting or
bug fixing text notes only.

The page is there to educate people about the specific feature of Squid,
not to teach sysadmin how to use their OS tools. We tell them what the
helper name is to look for, and leave the reader to use whatever method
they are familiar with already to find it quickly.


Because StoreID is a plugin interface we have a section documenting how
to write a custom helper and what I/O protocol the StoreID plugins are
required to communicate to Squid with when using this interface. So that
people interested can create their own custom ones as needed.

That is the entirety of the StoreID feature in Squid. Note how what you
want to add matches up wth what is there already. Which was my point
earlier, you dont need to re-document all that. :-)

Everything else is use-case specific. Including the rest of what you
propose writing. Which means a ConfigExample/CachingWithStoreId page for
that use-case. All about how to setup and configure a specific helper
plugin that *uses* StoreID interface for caching the contents in the
wiki provided pattern registry / DB / dataset.


I welcome your interest to write a ConfigExample page for using the
provided helper. Please take a short while to look at how the other
ConfigExample pages are structured (to avoid me or Francesco having to
re-edit it all). When creating a new example the page template contains
most of what you need, so just fill in the sections with your text.

Some hints on how to improve it inline below:

> You need to create a database file called .... and put (sourceforge example
> in it) and save the file to ..... on Linux or ..... on windows. 

DB filename and location is completely optional. It is passed as the
command-line parameter of the helper.

> Make sure the entries on the database are sepearated by a .....
> (tab/whitespace). 

repeating a snippet or referencing the man page text would be best.
There might need to be a paragraph on how to run a dataset converter
over the wiki contents to ensure its tab-delimited.

> When you have done this you can type ...........(example command to test the
> script) in Linux or ............... in Windows to test your database works. 
> If you see ERR then something is not right. 
> If you see ............... congratulations. 

> You MAY have to tell squid set up the cache direcories if you have not done
> it already with .... and then start Squid with ................ (give
> examples like init.d or sysctrl etc for fedora ubuntu and other popular
> linuxes windows abacus etc...) 

That is all just noise. OS-specific pages and/or FAQ elsewhere contain
details on how to run and manage Squid as a proxy. Assume the admin
knows what they are doing before they go near StoreID, if not they are
playing with explosives over an open fire.

The "Details" section of the ConfigExample page should list the use-case
for when this example is actually of any use. Squid needing to be
actually caching stuff is part of or implied by that use-case.


> Now go to your web browser and set up the proxy settings to the ip address
> of your squid server and the correct port. 

That is completely irrelevant to StoreID. Or even to testing it. The
above test of the helper is all that is necessary. Assume that the proxy
is already working for clients or admin testing use before this feature
is even attempted.

There are some related issues around Squid not being able to run the
helper: ie if they get helper binary or DB file permissions wrong it
wont start, or perhapse helper-specific error messages that get logged
to cache.log.
The "Troubleshooting" section should enumerate those errors, how to
diagnose them and what/how to fix.


> Try the ........(example) in your web browser to see if the page arrives and
> check the ... log file to see it was dealt with correctly (miss 1st time
> then, hit after a few retries) linux cat ...log | grep .... or windows use
> snaketail
> 

This does not test what you seem to think it does. You can run it
successfully on a default Squid "out-of-the-box" without any storeid_*
related configuration setup.


> I can polish the documentation once WE CAN WORK TOGETHER to get the
> information correct. Please give me a chance to put something back I dont
> want any credit and you can licence it any way you wish.

If you need a wiki account please register for one, then let me know
(privately is fine) what username you chose for yourself so I can assign
editing privileges.

Amos