[squid-users] [PATCH] Squid 3.5.19 SMP under OpenBSD - setsockopt for UDS

[Stephen Borrill]

On 27/06/2016 12:35, Silamael wrote:
> On 27.06.2016 13:19, Amos Jeffries wrote:
>> On 27/06/2016 9:16 p.m., Silamael wrote:
>>> Hi,
>>>
>>> I'm playing around with the SMP feature on OpenBSD 5.9 and noticed that
>>> Squid does not run due to hard coded limits for the receive and send
>>> buffer sizes of Unix Domain Sockets. In contrary to other OSes these
>>> limits cannot be adjusted by a sysctl.
>>
>> Is that a new regression in 5.9?
>> Can you provide a reference please?
> 
> Haven't tested it with a prior version than 5.9.
> But as the buffer limits haven't changed for a long time I don't think
> that's a regression in 5.9.
> 
>>
>> We have had Squid working just fine with SMP on OpenBSD before.
> 
> That's interesting. Without the patch with no disk cache at all
> configured and 4 workers, Squid seem to run a short time until the
> children terminate due to a registration timeout.
> 
>>
>>> The attached patch adds some setsockopt() calls to comm.cc which sets
>>> the buffer sizes to 256kb.
>>
>> Please submit patchs to the squid-dev mailing list.
>>
>> Please also include an example of the errors that are visible to users
>> with your squid-dev submission so we can refer people complaining about
>> specific error messages to the appropriate fix.
> 
> The error I got was a write failed on the UDS socket. Error code was
> EMSGSIZE. Squid tried to write 4320 bytes on the socket.
> 
>>
>> And detail of where the 2K/4K limit you mention is located. So we can
>> verify if you are looking at the right limitation and what else it affects.
>>  AFAIK, the UDS limitations are about the size of objects which are used
>> over UDS sockets. Reading 256KB into a 4KB object does not have pretty
>> results. Likewise a blocking queue of up to 64 time-critical SMP actions
>> awaiting the first one getting its synchronous UDS response does not
>> have good side effects.
> 
> The limits can be found in /usr/src/sys/kern/uipc_usrreq.c:
> u_long unpdg_sendspace = 2*1024;
> u_long unpdg_recvspace = 4*1024;
> 
> Quick check against OpenBSD 5.2 showed no difference.
> The value of 256kb is just taken from the Squid SMP FAQ page.

Yes, namely:

Squid workers exchange Unix Domain Sockets (UDS) messages (not to be
confused with UDP messages or System V IPC messages). These messages
should be smaller than 16KB in size, but even that creates problems in
some environments because of very low default UDS message size and
buffer size limits.

Usually, the limits can be adjusted using sysctl but exact control names
are not documented and vary from one OS to another. Here is one known
variant with recommended minimum settings (please add more if you know
them!):


net.local.dgram.recvspace: 262144
net.local.dgram.maxdgram: 16384

>> We no longer accept #ifdef construction in squid coding guidelines. Use
>> #if instead. If you can please also identify what other *BSD are needing
>> this and add them to the #if condition.
>>  Though if this is accepted the change may be relevant to all OS and not
>> needing a wrapper at all.
> 
> Ok, I can change the #ifdef into an #if. But I don't have other BSDs for
> testing.

I can help you there :-)

>From NetBSD:
FATAL: kid2 registration timed out
Squid Cache (Version 3.5.19): Terminated abnormally.
CPU Usage: 0.169 seconds = 0.074 user + 0.095 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 55


/usr/src/sys/kern/uipc_usrreq.c is the same as OpenBSD:

/*
 * Both send and receive buffers are allocated PIPSIZ bytes of buffering
 * for stream sockets, although the total for sender and receiver is
 * actually only PIPSIZ.
 * Datagram sockets really use the sendspace as the maximum datagram size,
 * and don't really want to reserve the sendspace.  Their recvspace should
 * be large enough for at least one max-size datagram plus address.
 */
#define PIPSIZ  4096
u_long  unpst_sendspace = PIPSIZ;
u_long  unpst_recvspace = PIPSIZ;
u_long  unpdg_sendspace = 2*1024;       /* really max datagram size */
u_long  unpdg_recvspace = 4*1024;