[squid-dev] Developing customized Cache Selection algorithm from Round Robin, Least Load

[Du, Hongfei]

Hi Alex
Thanks for the comments, really helpful, actually by the time I read this email in the morning, I have managed to find some things exactly using search methodology as you mentioned, please find my findings inline.

Best Regards,

Humphrey

/*

Hongfei Du
Staff Engineer (UK Software)
InterDigital UK, Inc.
Shoreditch Business Center
64 Great Eastern Street
London,  EC2A 3QR
T: +44 207.749.9140
Hongfei.Du at InterDigital.com
www.InterDigital.com<http://www.interdigital.com>

[cid:image5363d4.BMP at 8e96b82c.42b8be99]

This e-mail is intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential and/or otherwise protected from disclosure to anyone other than its intended recipient. Unintended transmission shall not constitute waiver of any privilege or confidentiality obligation. If you received this communication in error, please do not review, copy or distribute it, notify me immediately by email, and delete the original message and any attachments. Unless expressly stated in this e-mail, nothing in this message or any attachment should be construed as a digital or electronic signature.


-----Original Message-----
/*From: Alex Rousskov [mailto:rousskov at measurement-factory.com]
/*Sent: Thursday, August 20, 2015 5:16 AM
/*To: squid-dev at lists.squid-cache.org
/*Cc: Du, Hongfei <Hongfei.Du at InterDigital.com>
/*Subject: Re: [squid-dev] Developing customized Cache Selection algorithm from
/*Round Robin, Least Load
/*
/*On 08/19/2015 05:03 AM, Du, Hongfei wrote:
/*
/*> We are in an attempt to extend Squid Cache selection algorithm to
/*> develop a solution for controlled storage of subscriber’s content in
/*> Squid Caches, and thereafter apply access rule on these contents, a
/*> few questions to start with:
/*>
/*> ·        As we probably has to rewrite new algorithm and recompile it,
/*> so does anyone know where(or which file) is the existing Round Robin
/*> or Least Load algorithm defined in source codes?
/*
/*Search for the storeDirSelectSwapDir variable in src/store_dir.cc. To learn more
/*about customizing cache dir selection code, run:
/*
/*  $ bzr grep Config.store_dir_select_algorithm
/*
/*or an equivalent search command.
/*
[Du, Hongfei(Humphrey)] Using search commands, I found the following points:
- grep store_dir_select_algorithm, first hit at src/SquidConfig.h
- LL and RR is declared at store_dir.cc using static STDIRSELECT
- storeDirSelectSwapDir is really the one playing the trick(i.e. returns where the content is to be put into ), and it uses dirn as index with Config.cacheSwap.n_configured(I guess this is the total number of caches predefined in the disk?) I attached RR algorithm definition, really short yet helpful to learn the logics:

static int
storeDirSelectSwapDirRoundRobin(const StoreEntry * e)
{
   // e->objectLen() is negative at this point when we are still STORE_PENDING
   ssize_t objsize = e->mem_obj->expectedReplySize();
   if (objsize != -1)
       objsize += e->mem_obj->swap_hdr_sz;

   // Increment the first candidate once per selection (not once per
   // iteration) to reduce bias when some disk(s) attract more entries.
   static int firstCandidate = 0;
   if (++firstCandidate >= Config.cacheSwap.n_configured)
       firstCandidate = 0;

   for (int i = 0; i < Config.cacheSwap.n_configured; ++i) {
       const int dirn = (firstCandidate + i) % Config.cacheSwap.n_configured;
       const SwapDir *sd = dynamic_cast<SwapDir*>(INDEXSD(dirn));

       int load = 0;
       if (!sd->canStore(*e, objsize, load))
           continue;

       if (load < 0 || load > 1000) {
           continue;
       }

       return dirn;
   }

   return -1;
}


/*
/*> ·        Is there straight forward method to tell/instruct squid to
/*> store content(e.g. an URL)  from network in a predefined specific disk
/*> folder rather than using the selection algorithm itself?
/*
/*You can use cache_dir min-size and max-size options to control cache_dir
/*selection to some extent.
/*
/*You can use the "cache" directive in squid.conf to control what gets stored and
/*served from the cache. This directive accepts Squid ACLs, so it can be applied to
/*specific URLs, but it cannot allow or prohibit individual cache_dir usage.
/*
/*I am not aware of any other configurable interface at the individual URL level.
/*There is no configurable mapping of URLs/StoreIDs to cache_dirs (and beyond).
/*The storeDirSelectSwapDir variable mentioned above points to one of the two
/*functions that do the mapping (selected at configuration time based on
/*store_dir_select_algorithm in squid.conf).
/*
/*
/*The details of the cache_dir selection code are rather ugly, with several half-
/*broken interfaces augmented by weird storage module-dependent
/*implementations. Do not expect a smooth sailing.
/*
[Du, Hongfei(Humphrey)] We tested many access control functions with squid.conf, and managed to response correctly from squidclient URL requests, works perfectly via monitoring terminals, the key issue now is to find a way to instruct squid to understand request from client at URL level before mapping to dir accordingly. Not sure if anyone has been successful of implementing similar thing before?
/*
/*Good luck,
/*
/*Alex.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: image5363d4.BMP
Type: image/bmp
Size: 13078 bytes
Desc: image5363d4.BMP
URL: <http://lists.squid-cache.org/pipermail/squid-dev/attachments/20150820/f95a77b6/attachment-0001.bin>