Difference: MyOwnSquid (1 vs. 45)

Revision 452019-05-14 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"
Here is what is on this page:
Line: 28 to 28
 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
Changed:
<
<
so it's not much of a change.)
>
>
so it's not much of a change.) squid-3 and later do not have the hierarchy stoplist, so you can skip that.
  For squid-2.6 or earlier, the "hierarchy stoplist" line still needs to be changed, but instead of the
Line: 95 to 95
 cache_store_log none logformat awstats %>a %ui %un [%{%d/%b/%Y:%H:%M:%S}tl.%03tu %{%z}tl] "%rm %ru HTTP/%rv" %Hs %h %{cvmfs-info}>h" "%{Referer}>h" "%{User-Agent}>h"
Changed:
<
<
and then use "awstats" as the second parameter to the "acces_log" option.

Since the log files can get very big, we run a cron job every night to rotate the log files, keeping 10 days worth. The cron job runs a script that looks like:

>
>
and then use "awstats" as the second parameter to the "access_log" option, for example:
 
Changed:
<
<
#!/bin/bash

SQUID_DIR=/nthome/bjb/frontier/frontier-cache/squid FNCRON_DIR=/nthome/bjb/frontier/frontier-cache/utils/cron

$SQUID_DIR/sbin/squid -k rotate 2>&1 >> $FNCRON_DIR/daily.log

>
>
access_log /var/log/squid/access.log awstats
 
Changed:
<
<
You could also use a logrotate.d script.

Daily might not be enough, however. On heavily used squids we run another cron 4 times an hour that checks to see if access.log is greater than a chosen size and if so does an extra rotation.

>
>
Since the log files can get very big very fast, we run a cron job 4 times an hour that checks to see if access.log is greater than a chosen size and if so does an extra rotation, in addition to nightly rotations.
 

Enabling monitoring

Revision 442018-12-06 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"
Here is what is on this page:
Line: 8 to 8
  Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger.
Changed:
<
<
NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.27 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. If your squid is feeding other squids then definitely use our distribution because squid-3.5.27 and later 3.5.X versions still have a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. If you want to support IPv6 with multiple workers then also use our version because we have a fix for that which isn't yet in any squid version.
>
>
NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.28 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. If your squid is feeding other squids then definitely use our distribution because all upstream squid versions still have a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. We also discourage using 4.X versions prior to 4.4 because there has been no significant testing of it on the grid. Also, do not use rock cache because it does not work properly with If-Modified-Since.
  If you want to use squid-2, all versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project.
Changed:
<
<
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833; it was fixed again in squid-3.5.27. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952, and #4616 that are not yet in any upstream version, and has a workaround for bug #4575. A fix for #4767 was included in frontier-squid-3.5.27 but it was fixed in upstream 3.5.28.

Squid-4 is out of beta testing now but it is also susceptible to many of the above problems so it is not yet recommended to be used at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler, and we will probably support it on EL7 only).

>
>
Squid-3 & Squid-4 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Fortunately squid-4 is much more similar to squid-3, with only the addition of C++11 as a requirement. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). That problem with rock cache was resolved in squid-4.2, but then an additional problem documented in squid bug #4890 was discovered with rock cache + collapsed forwarding + If-Modified-Since and is not yet fixed, so we still cannot use rock cache with squid-4. A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833; it was fixed again in squid-3.5.27. Also, although squid bug #7 was mostly fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid version. Our version has a workaround for bug #4575 which caused many wasted DNS lookups and which has not yet been fixed in any upstream squid version, although the workaround can be applied by configuration (instructions below). Frontier-squid-3 had patches for #3952 and #4616 that didn't get into any upstream squid-3.x version, but they did get into squid-4.3. A fix for #4767 which was a problem with IPv6 on multiple workers was included in frontier-squid-3.5.27 but it was fixed in upstream 3.5.28 and 4.2.
  So for the Frontier application:

Revision 432018-11-14 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"
Added:
>
>
Here is what is on this page:

 

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger.

Revision 422018-08-23 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Line: 8 to 8
  If you want to use squid-2, all versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project.
Changed:
<
<
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833; it was fixed again in squid-3.5.27. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952, and #4616 that are not yet in any upstream version, and has a workaround for bug #4575. #4767 was fixed in 3.5.28.
>
>
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833; it was fixed again in squid-3.5.27. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952, and #4616 that are not yet in any upstream version, and has a workaround for bug #4575. A fix for #4767 was included in frontier-squid-3.5.27 but it was fixed in upstream 3.5.28.
  Squid-4 is out of beta testing now but it is also susceptible to many of the above problems so it is not yet recommended to be used at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler, and we will probably support it on EL7 only).

Revision 412018-08-10 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger.

Changed:
<
<
NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.27 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. If your squid is feeding other squids then definitely use our distribution because squid-3.5.27 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. If you want to support IPv6 with multiple workers then also use our version because we have a fix for that which isn't yet in any squid version.
>
>
NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.27 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. If your squid is feeding other squids then definitely use our distribution because squid-3.5.27 and later 3.5.X versions still have a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. If you want to support IPv6 with multiple workers then also use our version because we have a fix for that which isn't yet in any squid version.
  If you want to use squid-2, all versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project.
Changed:
<
<
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833; it was fixed again in squid-3.5.27. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952, #4616, and #4767 that are not yet in any upstream version, and has a workaround for bug #4575.
>
>
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833; it was fixed again in squid-3.5.27. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952, and #4616 that are not yet in any upstream version, and has a workaround for bug #4575. #4767 was fixed in 3.5.28.
  Squid-4 is out of beta testing now but it is also susceptible to many of the above problems so it is not yet recommended to be used at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler, and we will probably support it on EL7 only).

Revision 402018-07-06 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Line: 8 to 8
  If you want to use squid-2, all versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project.
Changed:
<
<
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833; it was fixed again in squid-3.5.27. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952, #4616, and #4767 that are not yet in any upstream version. Squid-4 and Squid-5 are still in beta testing and are also susceptible to many of the above problems so they should not be used yet at this time. (The are also difficult for us to distribute on EL6 because they require a c++11 compiler, and will probably need to be on EL7 only).
>
>
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833; it was fixed again in squid-3.5.27. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952, #4616, and #4767 that are not yet in any upstream version, and has a workaround for bug #4575.

Squid-4 is out of beta testing now but it is also susceptible to many of the above problems so it is not yet recommended to be used at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler, and we will probably support it on EL7 only).

  So for the Frontier application:
Line: 67 to 69
  This option combines requests agressively so that a file is retrieved only once from the origin server. This is a very good idea for computer farms so make sure it is on.
Changed:
<
<
For version squid-3.2 and later, in order to enable frontier-client versions older than 2.8.21 (March 2018) to clear certain server errors from the cache, it is important to include the following config parameters as described in squid bug #4809:
>
>
For version squid-3.2 and later, in order to enable frontier-client versions older than 2.8.21 (which is not yet released as of July 2018) to clear certain server errors from the cache, it is important to include the following config parameters as described in squid bug #4809:
 
acl PragmaNoCache req_header Pragma no-cache
send_hit deny PragmaNoCache
Added:
>
>
To avoid many DNS lookups as described in squid bug #4575, set the following:
url_rewrite_extras XXX
store_id_extras XXX
 Finally, if your squid is a squid-2 version and might possibly feed other squids then set this:
ignore_ims_on_miss on

Revision 392018-03-14 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Line: 8 to 8
  If you want to use squid-2, all versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project.
Changed:
<
<
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833; it was fixed again in squid-3.5.27. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952, #4616, and #4767 that are not yet in any upstream version. Squid-4 and Squid-5 are still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (The are also difficult for us to distribute on EL6 because they require a c++11 compiler, and will probably need to be on EL7 only).
>
>
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833; it was fixed again in squid-3.5.27. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952, #4616, and #4767 that are not yet in any upstream version. Squid-4 and Squid-5 are still in beta testing and are also susceptible to many of the above problems so they should not be used yet at this time. (The are also difficult for us to distribute on EL6 because they require a c++11 compiler, and will probably need to be on EL7 only).
  So for the Frontier application:

Revision 382018-03-08 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Line: 67 to 67
  This option combines requests agressively so that a file is retrieved only once from the origin server. This is a very good idea for computer farms so make sure it is on.
Changed:
<
<
For version squid-3.2 and later, in order to enable frontier-client versions older than 2.8.21 (January 2018) to clear certain server errors from the cache, it is important to include the following config parameters as described in squid bug #4809:
>
>
For version squid-3.2 and later, in order to enable frontier-client versions older than 2.8.21 (March 2018) to clear certain server errors from the cache, it is important to include the following config parameters as described in squid bug #4809:
 
acl PragmaNoCache req_header Pragma no-cache
send_hit deny PragmaNoCache

Revision 372018-01-24 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger.

Changed:
<
<
NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.27 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. Our version also has an important not-yet integrated fix for bug #4809 which prevents the frontier client from clearing some cached errors. If your squid is feeding other squids then definitely use our distribution because squid-3.5.27 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. If you want to support IPv6 with multiple workers then also use our version because we have a fix for that which isn't yet in any squid version.
>
>
NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.27 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. If your squid is feeding other squids then definitely use our distribution because squid-3.5.27 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. If you want to support IPv6 with multiple workers then also use our version because we have a fix for that which isn't yet in any squid version.
  If you want to use squid-2, all versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project.
Changed:
<
<
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833; it was fixed again in squid-3.5.27. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952, #4616, #4767, and #4809 that are not yet in any upstream version. Squid-4 and Squid-5 are still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (The are also difficult for us to distribute on EL6 because they require a c++11 compiler, and will probably need to be on EL7 only).
>
>
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833; it was fixed again in squid-3.5.27. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952, #4616, and #4767 that are not yet in any upstream version. Squid-4 and Squid-5 are still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (The are also difficult for us to distribute on EL6 because they require a c++11 compiler, and will probably need to be on EL7 only).
  So for the Frontier application:
Line: 67 to 67
  This option combines requests agressively so that a file is retrieved only once from the origin server. This is a very good idea for computer farms so make sure it is on.
Added:
>
>
For version squid-3.2 and later, in order to enable frontier-client versions older than 2.8.21 (January 2018) to clear certain server errors from the cache, it is important to include the following config parameters as described in squid bug #4809:
acl PragmaNoCache req_header Pragma no-cache
send_hit deny PragmaNoCache
 Finally, if your squid is a squid-2 version and might possibly feed other squids then set this:
ignore_ims_on_miss on

Revision 362018-01-19 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger.

Changed:
<
<
NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.27 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. If your squid is feeding other squids then definitely use our distribution because squid-3.5.27 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. If you want to support IPv6 with multiple workers then also use our version because we have a fix for that which isn't yet in any squid version.
>
>
NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.27 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. Our version also has an important not-yet integrated fix for bug #4809 which prevents the frontier client from clearing some cached errors. If your squid is feeding other squids then definitely use our distribution because squid-3.5.27 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. If you want to support IPv6 with multiple workers then also use our version because we have a fix for that which isn't yet in any squid version.
  If you want to use squid-2, all versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project.
Changed:
<
<
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833; it was fixed again in squid-3.5.27. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952, #4616, and #4767 that are not yet in any upstream version. Squid-4 is still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler, and will probably need to be on EL7 only).
>
>
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833; it was fixed again in squid-3.5.27. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952, #4616, #4767, and #4809 that are not yet in any upstream version. Squid-4 and Squid-5 are still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (The are also difficult for us to distribute on EL6 because they require a c++11 compiler, and will probably need to be on EL7 only).
  So for the Frontier application:

Revision 352018-01-17 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Line: 114 to 114
  because it is not on by default in squid-3.
Changed:
<
<
You may also need to open firewall and/or iptables holes for the addresses on the HOST_MONITOR line above.
>
>
You may also need to open firewall and/or iptables holes for the addresses on the HOST_MONITOR line above. If your firewall administrators don't like opening the whole IP address ranges please look at the instructions in the frontier-squid install documentation for justification and an alternative. In any case leave the squid.conf acl more open so only one place needs to change when the monitoring machine IP addresses change.
 

Double checking

Revision 342017-10-18 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger.

Changed:
<
<
NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.27 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. If your squid is feeding other squids then definitely use our distribution because squid-3.5.27 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. If you want to support IPv6 with multiple workers then also use our version because we have a fix for that which isn't yet in an squid version.
>
>
NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.27 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. If your squid is feeding other squids then definitely use our distribution because squid-3.5.27 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. If you want to support IPv6 with multiple workers then also use our version because we have a fix for that which isn't yet in any squid version.
 
Changed:
<
<
If you want to use squid-2, all versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project.
>
>
If you want to use squid-2, all versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project.
  Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833; it was fixed again in squid-3.5.27. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952, #4616, and #4767 that are not yet in any upstream version. Squid-4 is still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler, and will probably need to be on EL7 only).

Revision 332017-09-01 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger.

Changed:
<
<
NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.22 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. squid-3.5.23 promptly broke one of those crucial features. If your squid is feeding other squids then definitely use our distribution because squid-3.5.22 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. If you want to use squid-2, all versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project.
>
>
NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.27 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. If your squid is feeding other squids then definitely use our distribution because squid-3.5.27 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. If you want to support IPv6 with multiple workers then also use our version because we have a fix for that which isn't yet in an squid version.
 
Changed:
<
<
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952 and #4616 that are not yet in any upstream version. Squid-4 is still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler, and will probably need to be on EL7 only).
>
>
If you want to use squid-2, all versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project.

Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833; it was fixed again in squid-3.5.27. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952, #4616, and #4767 that are not yet in any upstream version. Squid-4 is still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler, and will probably need to be on EL7 only).

  So for the Frontier application:

Revision 322017-07-12 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Line: 6 to 6
  NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.22 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. squid-3.5.23 promptly broke one of those crucial features. If your squid is feeding other squids then definitely use our distribution because squid-3.5.22 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. If you want to use squid-2, all versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project.
Changed:
<
<
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. Even our most current version has a known problem with sometimes causing duplicated queries when used as a Frontier launchpad reverse proxy; this is the last bug preventing frontier-squid-3 from becoming our default for WLCG distribution. Squid-4 is still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler).
>
>
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. The current frontier-squid also has patches for #3952 and #4616 that are not yet in any upstream version. Squid-4 is still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler, and will probably need to be on EL7 only).
  So for the Frontier application:

Revision 312017-01-11 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Line: 6 to 6
  NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.22 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. squid-3.5.23 promptly broke one of those crucial features. If your squid is feeding other squids then definitely use our distribution because squid-3.5.22 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. If you want to use squid-2, all versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project.
Changed:
<
<
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it causes deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. Even our most current version has a known problem with sometimes causing duplicated queries when used as a Frontier launchpad reverse proxy; this is the last bug preventing frontier-squid-3 from becoming our default for WLCG distribution. Squid-4 is still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler).
>
>
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it caused deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. Even our most current version has a known problem with sometimes causing duplicated queries when used as a Frontier launchpad reverse proxy; this is the last bug preventing frontier-squid-3 from becoming our default for WLCG distribution. Squid-4 is still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler).
  So for the Frontier application:

Revision 302016-12-21 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger.

Changed:
<
<
NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.22 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. If your squid is feeding other squids then definitely use our distribution because squid-3.5.22 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. If you want to use squid-2, all versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project.
>
>
NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.22 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. squid-3.5.23 promptly broke one of those crucial features. If your squid is feeding other squids then definitely use our distribution because squid-3.5.22 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. If you want to use squid-2, all versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project.
 
Changed:
<
<
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it causes deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. Even our most current version has a known problem with sometimes causing duplicated queries when used as a Frontier launchpad reverse proxy; this is the last bug preventing frontier-squid-3 from becoming our default for WLCG distribution. Squid-4 is still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler).
>
>
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it causes deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22, but squid-3.5.23 broke the fix for bug #2833. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. Even our most current version has a known problem with sometimes causing duplicated queries when used as a Frontier launchpad reverse proxy; this is the last bug preventing frontier-squid-3 from becoming our default for WLCG distribution. Squid-4 is still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler).
  So for the Frontier application:

Revision 292016-12-07 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Line: 65 to 65
  This option combines requests agressively so that a file is retrieved only once from the origin server. This is a very good idea for computer farms so make sure it is on.
Changed:
<
<
Finally, if your squid might possibly feed other squids then set this:
>
>
Finally, if your squid is a squid-2 version and might possibly feed other squids then set this:
 
ignore_ims_on_miss on
Changed:
<
<
The default for that option prevents caching when an upstream squid sends an If-Modified-Since request and the object isn't already cached.
>
>
The default for that option prevents caching when an upstream squid sends an If-Modified-Since request and the object isn't already cached. This is fixed in squid-3 and is not an option.
  4) Recommended log file changes

Revision 282016-11-08 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger.

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.22 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. If your squid is feeding other squids then use our distribution because squid-3.5.22 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it.
>
>
NOTE: WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.22 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. If your squid is feeding other squids then definitely use our distribution because squid-3.5.22 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it. If you want to use squid-2, all versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project.
 
Changed:
<
<
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it causes deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. A pre-release version of frontier-squid-3 is now available by following the instructions at InstallSquid3. It is based on squid-3.5.21 which has at least one additional bug fix that Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs are in our version and were included in squid-3.5.22. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. Even our most current version has a known problem with sometimes causing duplicated queries when used as a Frontier launchpad reverse proxy; this is the last bug preventing frontier-squid-3 from becoming our default for WLCG distribution. Squid-4 is still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler).
>
>
Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it causes deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. squid-3.5.21 included a fix for squid bug #4428 which Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs were included in squid-3.5.22. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. Even our most current version has a known problem with sometimes causing duplicated queries when used as a Frontier launchpad reverse proxy; this is the last bug preventing frontier-squid-3 from becoming our default for WLCG distribution. Squid-4 is still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler).
  So for the Frontier application:
Line: 91 to 91
  You could also use a logrotate.d script.
Changed:
<
<
Daily might not be enough, however. On heavily used squids we run another cron hourly that checks to see if access.log is greater than a chosen size and if so does an extra rotation.
>
>
Daily might not be enough, however. On heavily used squids we run another cron 4 times an hour that checks to see if access.log is greater than a chosen size and if so does an extra rotation.
 

Enabling monitoring

Revision 272016-11-07 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger.

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.22 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. If your squid is feeding other squids then use our distribution because squid-3.5.22 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.22 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. If your squid is feeding other squids then use our distribution because squid-3.5.22 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it.
  Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it causes deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. A pre-release version of frontier-squid-3 is now available by following the instructions at InstallSquid3. It is based on squid-3.5.21 which has at least one additional bug fix that Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs are in our version and were included in squid-3.5.22. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. Even our most current version has a known problem with sometimes causing duplicated queries when used as a Frontier launchpad reverse proxy; this is the last bug preventing frontier-squid-3 from becoming our default for WLCG distribution. Squid-4 is still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler).

Revision 262016-10-12 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger.

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use any version squid-3.X (which is the default on EL6) other than our own distribution as it is known to be missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have as long of a history of production use. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it causes deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. A pre-release version of frontier-squid-3 is now available by following the instructions at InstallSquid3. It is based on squid-3.5.21 which has at least one additional bug fix that Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs are in our version but have not been incorporated into any version of squid-3 from the squid project (they are or will be soon in squid-4). Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version. Even our most current version has a known problem with sometimes causing duplicated queries when used as a Frontier launchpad reverse proxy; this is the last bug preventing frontier-squid-3 from becoming our default for WLCG distribution. Squid-4 is also susceptible to many of the above problems so it should also not be used yet at this time. (Squid-4 is difficult for us to distribute on EL6 because it requires a c++11 compiler).
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid-2 might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use any version of squid-3.X prior to squid-3.5.22 (including the default squid on EL6 and EL7) as it was missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. If your squid is feeding other squids then use our distribution because squid-3.5.22 still has a problem with the default ufs cache related to squid bug #7 and we have included a simple patch for it.

Squid-3 Gory Details: Squid-3 is a complete rewrite of squid in C++ instead of C so it took a very long time to catch up with all the features and stability of squid-2. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it causes deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. A pre-release version of frontier-squid-3 is now available by following the instructions at InstallSquid3. It is based on squid-3.5.21 which has at least one additional bug fix that Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs are in our version and were included in squid-3.5.22. Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version which is not yet in any upstream squid-3.5 version. Even our most current version has a known problem with sometimes causing duplicated queries when used as a Frontier launchpad reverse proxy; this is the last bug preventing frontier-squid-3 from becoming our default for WLCG distribution. Squid-4 is still in beta testing and is also susceptible to many of the above problems so it should not be used yet at this time. (Squid-4 is also difficult for us to distribute on EL6 because it requires a c++11 compiler).

  So for the Frontier application:

Revision 252016-09-16 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger.

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use any version squid-3.X (which is the default on EL6) other than our own distribution as it is known to be missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it has problems (see below)) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have as long of a history of production use. Collapsed_forwarding was added to squid-3.5, but when not using rock cache it causes deadlocks with multiple workers, and in any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324) . A fix for squid bug #2831 is in squid-3.5.16. A pre-release version of frontier-squid-3 is now available for Frontier-only installations by following the instructions at InstallSquid3. It has a workaround for the non-rock cache collapsed forwarding deadlock problem in a patch, and also a patch for squid bug #4428. We hired a consultant to fix squid bug #2833 to make collapsed forwarding work with the Frontier application's use of Last-Modified headers, and the patch is in our version but has not been incorporated into any version of squid-3 from the squid project (it is now in squid-4). Even in our most current version, collapsed forwarding does not work for applications that do not send Last-Modified headers such as CVMFS, as tracked in squid bug #4471; this is the last bug preventing frontier-squid-3 from becoming our default distribution, although it only impacts CVMFS once every 3 days since that's the length of time that CVMFS stratum 1s set for cache object expiration. Squid-4 is also susceptible to many of the above problems so it should also not be used yet at this time. (Squid-4 is difficult for us to distribute on EL6 because it requires a c++11 compiler).
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use any version squid-3.X (which is the default on EL6) other than our own distribution as it is known to be missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it had such significant problems that we had to hire a consultant to fix them, see below) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have as long of a history of production use. Collapsed_forwarding was added to squid-3.5, but it only worked for the initial loading of cached items, not for revalidating or reloading anything already cached but expired, and when not using rock cache it causes deadlocks with multiple workers. In any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324). A fix for squid bug #2831 is in squid-3.5.16. A pre-release version of frontier-squid-3 is now available by following the instructions at InstallSquid3. It is based on squid-3.5.21 which has at least one additional bug fix that Frontier needs. We hired a consultant to fix bugs #2833, #4311, and #4471 which describe problems with collapsed forwarding. The patches for those bugs are in our version but have not been incorporated into any version of squid-3 from the squid project (they are or will be soon in squid-4). Also, although squid bug #7 was fixed in squid-3.5, a small subset of the problem showed up again and we have a patch for that in our version. Even our most current version has a known problem with sometimes causing duplicated queries when used as a Frontier launchpad reverse proxy; this is the last bug preventing frontier-squid-3 from becoming our default for WLCG distribution. Squid-4 is also susceptible to many of the above problems so it should also not be used yet at this time. (Squid-4 is difficult for us to distribute on EL6 because it requires a c++11 compiler).
  So for the Frontier application:

Revision 242016-08-09 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Changed:
<
<
Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).
>
>
Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger.
 
Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use any version squid-3.X (which is the default on EL6) other than our own distribution as it is known to be missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it has problems (see below)) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have as long of a history of production use. Collapsed_forwarding was added to squid-3.5, but when not using rock cache it causes deadlocks with multiple workers, and in any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324) . A fix for squid bug #2831 is in squid-3.5.16. A pre-release version of frontier-squid-3 is now available for Frontier-only installations by following the instructions at InstallSquid3. It has a workaround for the non-rock cache collapsed forwarding deadlock problem in a patch. We hired a consultant to fix squid bug #2833 to make collapsed forwarding work with the Frontier application's use of Last-Modified headers, and the patch is in our version but has not been incorporated into any version of squid-3 from the squid project (it is expected to be in squid-4). Even in our most current version, collapsed forwarding does not work for applications that do not send Last-Modified headers such as CVMFS, as tracked in squid bug #4471; this is the last bug preventing frontier-squid-3 from becoming our default distribution, although it only impacts CVMFS once every 3 days since that's the length of time that CVMFS stratum 1s set for cache object expiration. Squid-4 is also susceptible to many of the above problems so it should also not be used yet at this time.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use any version squid-3.X (which is the default on EL6) other than our own distribution as it is known to be missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it has problems (see below)) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have as long of a history of production use. Collapsed_forwarding was added to squid-3.5, but when not using rock cache it causes deadlocks with multiple workers, and in any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324) . A fix for squid bug #2831 is in squid-3.5.16. A pre-release version of frontier-squid-3 is now available for Frontier-only installations by following the instructions at InstallSquid3. It has a workaround for the non-rock cache collapsed forwarding deadlock problem in a patch, and also a patch for squid bug #4428. We hired a consultant to fix squid bug #2833 to make collapsed forwarding work with the Frontier application's use of Last-Modified headers, and the patch is in our version but has not been incorporated into any version of squid-3 from the squid project (it is now in squid-4). Even in our most current version, collapsed forwarding does not work for applications that do not send Last-Modified headers such as CVMFS, as tracked in squid bug #4471; this is the last bug preventing frontier-squid-3 from becoming our default distribution, although it only impacts CVMFS once every 3 days since that's the length of time that CVMFS stratum 1s set for cache object expiration. Squid-4 is also susceptible to many of the above problems so it should also not be used yet at this time. (Squid-4 is difficult for us to distribute on EL6 because it requires a c++11 compiler).
  So for the Frontier application:

Revision 232016-07-26 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding, a least prior to 3.5 and even then it has problems (see below)) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have as long of a history of production use. Collapsed_forwarding has been added to squid-3.5, but when not using rock cache it causes deadlocks with multiple workers and Frontier cannot use rock cache because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324) . A fix for squid bug #2831 is in squid-3.5.16. A pre-release version of frontier-squid-3 is now available for testing by following the instructions at InstallSquid3. It has a workaround for the non-rock cache collapsed forwarding deadlock problem in a patch. It also has a fix for squid bug #4471; that bug makes collapsed forwarding basically useless for even CVMFS. Frontier still suffers from squid bug #2833 and that is why frontier-squid-3 is not yet released for general production use.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we especially recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use any version squid-3.X (which is the default on EL6) other than our own distribution as it is known to be missing crucial features (such as collapsed_forwarding, at least prior to 3.5 and even then it has problems (see below)) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have as long of a history of production use. Collapsed_forwarding was added to squid-3.5, but when not using rock cache it causes deadlocks with multiple workers, and in any case Frontier cannot use rock cache in squid-3 because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324) . A fix for squid bug #2831 is in squid-3.5.16. A pre-release version of frontier-squid-3 is now available for Frontier-only installations by following the instructions at InstallSquid3. It has a workaround for the non-rock cache collapsed forwarding deadlock problem in a patch. We hired a consultant to fix squid bug #2833 to make collapsed forwarding work with the Frontier application's use of Last-Modified headers, and the patch is in our version but has not been incorporated into any version of squid-3 from the squid project (it is expected to be in squid-4). Even in our most current version, collapsed forwarding does not work for applications that do not send Last-Modified headers such as CVMFS, as tracked in squid bug #4471; this is the last bug preventing frontier-squid-3 from becoming our default distribution, although it only impacts CVMFS once every 3 days since that's the length of time that CVMFS stratum 1s set for cache object expiration. Squid-4 is also susceptible to many of the above problems so it should also not be used yet at this time.
  So for the Frontier application:

Revision 222016-06-15 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Line: 52 to 52
  The first item lets us cache objects up to 1 GB in size (which is a lot more than most web pages - the default is only 4 MB). To date, we have often cached tarballs up to 300 MB in size, and more might be needed. The second item lets us use the cache_mem we gave it above (the default is only 8 KB).
Added:
>
>
In order to avoid not caching any objects that squid first reads when they're less than 60 seconds from expiring (as described in squid bug #4531) set this parameter:
minimum_expiry_time 0
 An important option for squid-2.6 and later is:
collapsed_forwarding on 

Revision 212016-04-18 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding, a least prior to 3.5 and even then it has problems (see below)) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have as long of a history of production use. Collapsed_forwarding has been added to squid-3.5, but when not using rock cache it causes deadlocks with multiple workers and Frontier cannot use rock cache because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324) . A fix for squid bug #2831 is in squid-3.5.16. A pre-release version of frontier-squid-3 is now available for testing by following the instructions at InstallSquid3. It has a workaround for the non-rock cache collapsed forwarding deadlock problem in a patch. It also has a fix for squid bug #4471 which makes collapsed forwarding basically useless for even CVMFS. Frontier still suffers from squid bug #2833 and that is why frontier-squid-3 is not yet released for general production use.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding, a least prior to 3.5 and even then it has problems (see below)) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have as long of a history of production use. Collapsed_forwarding has been added to squid-3.5, but when not using rock cache it causes deadlocks with multiple workers and Frontier cannot use rock cache because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324) . A fix for squid bug #2831 is in squid-3.5.16. A pre-release version of frontier-squid-3 is now available for testing by following the instructions at InstallSquid3. It has a workaround for the non-rock cache collapsed forwarding deadlock problem in a patch. It also has a fix for squid bug #4471; that bug makes collapsed forwarding basically useless for even CVMFS. Frontier still suffers from squid bug #2833 and that is why frontier-squid-3 is not yet released for general production use.
  So for the Frontier application:

Revision 202016-04-05 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding, a least prior to 3.5 and even then it has problems (see below)) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have as long of a history of production use. Collapsed_forwarding has been added to squid-3.5, but when not using rock cache it causes deadlocks with multiple workers and Frontier cannot use rock cache because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324) . A pre-release version of frontier-squid-3 is now available for testing by following the instructions at InstallSquid3. It has a workaround for the non-rock cache collapsed forwarding deadlock problem in a patch. It also has a fix for squid bug #4471 which makes collapsed forwarding basically useless for even CVMFS. Frontier still suffers from squid bug #2833 and that is why frontier-squid-3 is not yet released for general production use.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding, a least prior to 3.5 and even then it has problems (see below)) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have as long of a history of production use. Collapsed_forwarding has been added to squid-3.5, but when not using rock cache it causes deadlocks with multiple workers and Frontier cannot use rock cache because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324) . A fix for squid bug #2831 is in squid-3.5.16. A pre-release version of frontier-squid-3 is now available for testing by following the instructions at InstallSquid3. It has a workaround for the non-rock cache collapsed forwarding deadlock problem in a patch. It also has a fix for squid bug #4471 which makes collapsed forwarding basically useless for even CVMFS. Frontier still suffers from squid bug #2833 and that is why frontier-squid-3 is not yet released for general production use.
  So for the Frontier application:

Revision 192016-03-23 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. Collapsed_forwarding has been added to squid-3.5, but when not using rock cache it causes deadlocks with multiple workers and Frontier cannot use rock cache because it is still susceptible to squid bug #7 (and it is more succinctly stated in squid bug #4324) . A pre-release version of frontier-squid-3 is now available for testing by following the instructions at InstallSquid3. It has a workaround for the non-rock cache collapsed forwarding deadlock problem in a patch. If you are only using one worker, or multiple workers with rock cache, you could probably get an unpatched squid-3.5.x version to work with collapsed forwarding for CVMFS, but even with one worker Frontier would still suffer from squid bug #2833.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding, a least prior to 3.5 and even then it has problems (see below)) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have as long of a history of production use. Collapsed_forwarding has been added to squid-3.5, but when not using rock cache it causes deadlocks with multiple workers and Frontier cannot use rock cache because it is still susceptible to squid bug #7 (the problem is more succinctly stated in squid bug #4324) . A pre-release version of frontier-squid-3 is now available for testing by following the instructions at InstallSquid3. It has a workaround for the non-rock cache collapsed forwarding deadlock problem in a patch. It also has a fix for squid bug #4471 which makes collapsed forwarding basically useless for even CVMFS. Frontier still suffers from squid bug #2833 and that is why frontier-squid-3 is not yet released for general production use.
  So for the Frontier application:

Revision 182015-11-19 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Line: 68 to 68
 
strip_query_terms off
cache_store_log none
Changed:
<
<
logformat squid %>a %ui %un [%{%d/%b/%Y:%H:%M:%S +0000}tl] "%rm %ru HTTP/%rv" %Hs %h"
>
>
logformat awstats %>a %ui %un [%{%d/%b/%Y:%H:%M:%S}tl.%03tu %{%z}tl] "%rm %ru HTTP/%rv" %Hs %h %{cvmfs-info}>h" "%{Referer}>h" "%{User-Agent}>h"
 
Added:
>
>
and then use "awstats" as the second parameter to the "acces_log" option.
  Since the log files can get very big, we run a cron job every night to rotate the log files, keeping 10 days worth. The cron job runs a script that looks like:

Revision 172015-10-28 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. Collapsed_forwarding has been added to squid-3.5, but when not using rock cache it causes deadlocks with multiple workers and Frontier cannot use rock cache because it is still susceptible to squid bug #7 (and it is more succinctly stated in squid bug #4324) . A pre-release version of frontier-squid-3 is now available for testing by following the instructions at InstallSquid3. It has a workaround for the collapsed forwarding deadlock problem in a patch. If you are only using one worker you could probably get an unpatched squid-3.5.x version to work with collapsed forwarding for CVMFS, but Frontier would still suffer from squid bug #2833.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. Collapsed_forwarding has been added to squid-3.5, but when not using rock cache it causes deadlocks with multiple workers and Frontier cannot use rock cache because it is still susceptible to squid bug #7 (and it is more succinctly stated in squid bug #4324) . A pre-release version of frontier-squid-3 is now available for testing by following the instructions at InstallSquid3. It has a workaround for the non-rock cache collapsed forwarding deadlock problem in a patch. If you are only using one worker, or multiple workers with rock cache, you could probably get an unpatched squid-3.5.x version to work with collapsed forwarding for CVMFS, but even with one worker Frontier would still suffer from squid bug #2833.
  So for the Frontier application:

Revision 162015-10-21 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. Collapsed_forwarding has been added to squid-3.5, but when not using rock cache it causes deadlocks with multiple workers and Frontier cannot use rock cache because it is still susceptible to squid bug #7 . A pre-release version of frontier-squid-3 is now available for testing by following the instructions at InstallSquid3. It has a workaround for the collapsed forwarding deadlock problem in a patch. If you are only using one worker you could probably get an unpatched squid-3.5 version to work.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. Collapsed_forwarding has been added to squid-3.5, but when not using rock cache it causes deadlocks with multiple workers and Frontier cannot use rock cache because it is still susceptible to squid bug #7 (and it is more succinctly stated in squid bug #4324) . A pre-release version of frontier-squid-3 is now available for testing by following the instructions at InstallSquid3. It has a workaround for the collapsed forwarding deadlock problem in a patch. If you are only using one worker you could probably get an unpatched squid-3.5.x version to work with collapsed forwarding for CVMFS, but Frontier would still suffer from squid bug #2833.
  So for the Frontier application:

Revision 152015-09-15 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and because of these problems it has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. Collapsed_forwarding has been added to squid-3.5 which was first made available in a pre-release on 31 October 2015 and has not yet had thorough testing for grid applications. If you use squid-3.5 it may work, but you will be a pioneer so you may encounter other bugs. A pre-release version of frontier-squid-3 is now available for testing by following the instructions at InstallSquid#Installation and adding the yum install option --enablerepo=cern-frontier-debug.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that were never released by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. Collapsed_forwarding has been added to squid-3.5, but when not using rock cache it causes deadlocks with multiple workers and Frontier cannot use rock cache because it is still susceptible to squid bug #7 . A pre-release version of frontier-squid-3 is now available for testing by following the instructions at InstallSquid3. It has a workaround for the collapsed forwarding deadlock problem in a patch. If you are only using one worker you could probably get an unpatched squid-3.5 version to work.
  So for the Frontier application:

Revision 142014-11-25 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using a squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and because of these problems it has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. Collapsed_forwarding has been added to squid-3.5 which was first made available in a pre-release on 31 October 2015 and has not yet had thorough testing for grid applications. If you use squid-3.5 it may work, but you will be a pioneer so you may encounter other bugs.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and because of these problems it has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. Collapsed_forwarding has been added to squid-3.5 which was first made available in a pre-release on 31 October 2015 and has not yet had thorough testing for grid applications. If you use squid-3.5 it may work, but you will be a pioneer so you may encounter other bugs. A pre-release version of frontier-squid-3 is now available for testing by following the instructions at InstallSquid#Installation and adding the yum install option --enablerepo=cern-frontier-debug.
  So for the Frontier application:
Line: 96 to 96
 

all in the appropriate places in squid.conf.

Added:
>
>
If you are using a version of squid-3 you will also need to set
snmp_port 3401
because it is not on by default in squid-3.
 You may also need to open firewall and/or iptables holes for the addresses on the HOST_MONITOR line above.

Double checking

Revision 132014-11-05 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"
Changed:
<
<

Using squid version other than the frontier-squid distribution

>
>

Using a squid version other than the frontier-squid distribution

  Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).
Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and because of these problems it has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. Collapsed_forwarding has been added to the development version of squid-3.5 but it has not been released yet. If you use squid-3.5 and can guarantee that its only clients will be Frontier or possibly CVMFS (especially not another squid), it may work, but you will be a pioneer so you may encounter other bugs.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and because of these problems it has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. Collapsed_forwarding has been added to squid-3.5 which was first made available in a pre-release on 31 October 2015 and has not yet had thorough testing for grid applications. If you use squid-3.5 it may work, but you will be a pioneer so you may encounter other bugs.
  So for the Frontier application:

Revision 122014-08-28 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using squid version other than the frontier-squid distribution

Line: 84 to 84
  Daily might not be enough, however. On heavily used squids we run another cron hourly that checks to see if access.log is greater than a chosen size and if so does an extra rotation.
Changed:
<
<

>
>

Enabling monitoring

  That's basically it, except for one important thing. For Frontier we monitor our squids remotely using SNMP. This is not turned on by default in squid, so to use it you have to turn it on at compilation time. If you want to use SNMP there are a few more settings needed in the squid.conf, especially the ACL access for whatever machines are allowed to read the SNMP information. Scientific Linux/Redhat Enterprise Linux squid rpms should already have SNMP enabled at compilation time. Therefore, it should be possible to enable monitoring by adding something like the following to your squid.conf which allow requests from the main CERN network and the CERN Hungary data center:
Line: 98 to 98
 all in the appropriate places in squid.conf. You may also need to open firewall and/or iptables holes for the addresses on the HOST_MONITOR line above.
Changed:
<
<
One thing you can do is make a dummy installation of our tarball. It can be installed anywhere by any user. Then do a diff of your squid.conf with our squid.conf. For startup and shutdown procedures, you are on your own (unless of course you are using someone else's package).

The compilation options we currently use are (Squid-2.6STABLE18 or later):

>
>

Double checking

 
Changed:
<
<
--disable-wccp --enable-snmp --disable-ident-lookups --with-large-files

(If you have a 64-bit OS, you don't need --with-large-files)

>
>
One thing you can do is make a dummy installation of our tarball. It can be installed anywhere by any user. Then do a diff of your squid.conf with our squid.conf. For startup and shutdown procedures, you are on your own (unless of course you are using someone else's package).
 

Responsible: DaveDykstra \ No newline at end of file

Revision 112014-07-09 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and because of these problems it has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. Collapsed_forwarding was added to squid-3.4 released on 2 May 2014 but hasn't been tested at scale yet. If you use squid-3.4 and can guarantee that its only clients will be Frontier or possibly CVMFS (especially not another squid), it may work, but you will be a pioneer so you may encounter other bugs.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and because of these problems it has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. Collapsed_forwarding has been added to the development version of squid-3.5 but it has not been released yet. If you use squid-3.5 and can guarantee that its only clients will be Frontier or possibly CVMFS (especially not another squid), it may work, but you will be a pioneer so you may encounter other bugs.
  So for the Frontier application:

Revision 102014-06-25 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and because of these problems it has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. If you can guarantee that the squid will only be used by a very small number of jobs simultaneously and the only clients will be Frontier or possibly CVMFS, it may work, but you will be a pioneer so you may encounter other bugs.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and because of these problems it has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. Collapsed_forwarding was added to squid-3.4 released on 2 May 2014 but hasn't been tested at scale yet. If you use squid-3.4 and can guarantee that its only clients will be Frontier or possibly CVMFS (especially not another squid), it may work, but you will be a pioneer so you may encounter other bugs.
  So for the Frontier application:
Line: 86 to 86
 
Changed:
<
<
That's basically it, except for one important thing. For Frontier we monitor our squids remotely using SNMP. This is not turned on by default in squid, so to use it you have to turn it on at compilation time. If you want to use SNMP there are a few more settings needed in the squid.conf, especially the ACL access for whatever machines are allowed to read the SNMP information. Scientific Linux/Redhat Enterprise Linux squid rpms should already have SNMP enabled at compilation time. Therefore, it should be possible to enable monitoring by adding something like the following to your squid.conf:
>
>
That's basically it, except for one important thing. For Frontier we monitor our squids remotely using SNMP. This is not turned on by default in squid, so to use it you have to turn it on at compilation time. If you want to use SNMP there are a few more settings needed in the squid.conf, especially the ACL access for whatever machines are allowed to read the SNMP information. Scientific Linux/Redhat Enterprise Linux squid rpms should already have SNMP enabled at compilation time. Therefore, it should be possible to enable monitoring by adding something like the following to your squid.conf which allow requests from the main CERN network and the CERN Hungary data center:
 
Changed:
<
<
acl HOST_MONITOR src 127.0.0.1/32 128.142.0.0/16 188.185.0.0/17
>
>
acl HOST_MONITOR src 127.0.0.1/32 128.142.0.0/16 188.184.128.0/17 188.185.128.0/17
 acl snmppublic snmp_community public

snmp_access allow snmppublic HOST_MONITOR

Revision 92014-01-29 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and because of these problems it has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. If you can guarantee that the squid will only be used by a very small number of users simultaneously and the only clients will be Frontier or possibly CVMFS, it may work, but you will be a pioneer so you may encounter other bugs.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and because of these problems it has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. If you can guarantee that the squid will only be used by a very small number of jobs simultaneously and the only clients will be Frontier or possibly CVMFS, it may work, but you will be a pioneer so you may encounter other bugs.
  So for the Frontier application:

Revision 82014-01-28 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. UNDER NO CIRCUMSTANCES use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and because of these problems it has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. WE STRONGLY DISCOURAGE attempting to use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and because of these problems it has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use. If you can guarantee that the squid will only be used by a very small number of users simultaneously and the only clients will be Frontier or possibly CVMFS, it may work, but you will be a pioneer so you may encounter other bugs.
  So for the Frontier application:

Revision 72013-11-04 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. UNDER NO CIRCUMSTANCES use version squid-3.X as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and because of these problems it has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later (the default on EL5 is squid2.6.X). If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. UNDER NO CIRCUMSTANCES use version squid-3.X (which is the default on EL6) as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and because of these problems it has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use.
  So for the Frontier application:

Revision 62013-04-11 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. UNDER NO CIRCUMSTANCES use version squid-3.X as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. UNDER NO CIRCUMSTANCES use version squid-3.X as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and because of these problems it has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use.
  So for the Frontier application:

Revision 52013-01-08 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid might be feeding other squids then we highly recommend using our version as we have patches that have not yet (as of October 2012 at least) been incorporated by the squid project. UNDER NO CIRCUMSTANCES use version squid-3.X as it is known to be missing features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid might be feeding other squids then we highly recommend using our version as we have patches (for squid bug #2831 and squid bug #2833) that have not yet (as of January 2013 at least) been incorporated by the squid project. UNDER NO CIRCUMSTANCES use version squid-3.X as it is known to be missing crucial features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use.
  So for the Frontier application:

Revision 42012-12-26 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using squid version other than the frontier-squid distribution

Line: 88 to 88
  That's basically it, except for one important thing. For Frontier we monitor our squids remotely using SNMP. This is not turned on by default in squid, so to use it you have to turn it on at compilation time. If you want to use SNMP there are a few more settings needed in the squid.conf, especially the ACL access for whatever machines are allowed to read the SNMP information. Scientific Linux/Redhat Enterprise Linux squid rpms should already have SNMP enabled at compilation time. Therefore, it should be possible to enable monitoring by adding something like the following to your squid.conf:
Changed:
<
<
acl HOST_MONITOR src 127.0.0.1/32 frontier.cern.ch acl HOST_MONITOR_NAME srcdomain cmsdbsfrontier.cern.ch
>
>
acl HOST_MONITOR src 127.0.0.1/32 128.142.0.0/16 188.185.0.0/17
 acl snmppublic snmp_community public

snmp_access allow snmppublic HOST_MONITOR

Deleted:
<
<
snmp_access allow snmppublic HOST_MONITOR_NAME
 snmp_access deny all

Revision 32012-11-30 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"

Using squid version other than the frontier-squid distribution

Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below).

Changed:
<
<
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid might be feeding other squids then we highly recommend using our version as we have patches that have not yet (as of October 2012 at least) been incorporated by the squid project. UNDER NO CIRCUMSTANCES use version squid-3.X as it is known to be missing features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use.
>
>
NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid might be feeding other squids then we highly recommend using our version as we have patches that have not yet (as of October 2012 at least) been incorporated by the squid project. UNDER NO CIRCUMSTANCES use version squid-3.X as it is known to be missing features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use.
  So for the Frontier application:

Revision 22012-11-01 - DaveDykstra

Line: 1 to 1
 
META TOPICPARENT name="InstallSquid"
Added:
>
>

Using squid version other than the frontier-squid distribution

 Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below). NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid might be feeding other squids then we highly recommend using our version as we have patches that have not yet (as of October 2012 at least) been incorporated by the squid project. UNDER NO CIRCUMSTANCES use version squid-3.X as it is known to be missing features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use.
Line: 102 to 104
  The compilation options we currently use are (Squid-2.6STABLE18 or later):
Changed:
<
<
--disable-wccp --enable-snmp --disable-ident-lookups --with-large-files
>
>
--disable-wccp --enable-snmp --disable-ident-lookups --with-large-files
  (If you have a 64-bit OS, you don't need --with-large-files)

Revision 12012-10-30 - DaveDykstra

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="InstallSquid"
Squid has a lot of parameters that are set in a file called squid.conf. These should be tuned according to the application. What we are describing below are changes to the default parameters that are needed for the Frontier application. The normal application for squid is to cache internet web pages, which is a bit different than Frontier. For web pages there are lots and lots of rather small objects. In the Frontier application there are relatively few objects, but some of them are much bigger. You really should use a squid of at least version squid-2.6.X to get collapsed_forwarding (see below). NOTE: All versions of squid before September 2009 have a bug that affects Frontier performance, so you should use squid version squid-2.7.STABLE7 or later. If your squid might be feeding other squids then we highly recommend using our version as we have patches that have not yet (as of October 2012 at least) been incorporated by the squid project. UNDER NO CIRCUMSTANCES use version squid-3.X as it is known to be missing features (such as collapsed_forwarding) and bug fixes (such as squid bug #7) essential to Frontier and has not been tested to see if it might work under even some limited conditions. Squid-3 is a complete rewrite of squid in C++ instead of C so it doesn't have nearly as long of a history of production use.

So for the Frontier application:

1) If there's any chance that you will need to support Frontier clients that are older than the December 2010 release (version 2.8.0), essential changes for squid-2.7 or later are:

hierarchy stoplist cgi-bin
refresh_pattern -i /cgi-bin/    0       0%      0

These two changes allow queries with ? in the URL to be cached. By default squid does not cache the dynamic web pages usually expected from a question mark, and the older Frontier clients used the question mark in all URLs. Without these two changes, squid won't cache anything for Frontier clients older than December 2010. (The default values of these two lines are:

hierarchy stoplist cgi-bin ?
refresh_pattern -i (/cgi-bin/|\?) 0    0%      0

so it's not much of a change.)

For squid-2.6 or earlier, the "hierarchy stoplist" line still needs to be changed, but instead of the

refresh_pattern -i /cgi-bin/    0       0%      0
line there is a line:
acl QUERY urlpath_regex cgi-bin \?
that needs to be changed to:
acl QUERY urlpath_regex cgi-bin

2) Hardware dependent changes

cache_mem
cache_dir

You have to define a parameter cache_dir which tells the squid where to keep the information on disk and how large it should be. This should be at least 20000 megabytes but probably no more than 70% of the partition size (to allow room for log files and other uses). The other hardware parameter is cache_mem. The default cache_mem is only 8 MB and is probably a very old default and should be increased some, but we have found that squid performs better for large objects out of the disk cache than the memory cache. We recommend at most 1/8 of the physical RAM and no more than 128 MB, leaving a lot of memory for disk buffering.

3) Tuning changes

maximum_object_size 1048576 KB
maximum_object_size_in_memory 128 KB
The first item lets us cache objects up to 1 GB in size (which is a lot more than most web pages - the default is only 4 MB). To date, we have often cached tarballs up to 300 MB in size, and more might be needed. The second item lets us use the cache_mem we gave it above (the default is only 8 KB).

An important option for squid-2.6 and later is:

collapsed_forwarding on 
This option combines requests agressively so that a file is retrieved only once from the origin server. This is a very good idea for computer farms so make sure it is on.

Finally, if your squid might possibly feed other squids then set this:

ignore_ims_on_miss on
The default for that option prevents caching when an upstream squid sends an If-Modified-Since request and the object isn't already cached.

4) Recommended log file changes

strip_query_terms off
cache_store_log none
logformat squid %>a %ui %un [%{%d/%b/%Y:%H:%M:%S +0000}tl] "%rm %ru HTTP/%rv"  %Hs %<st %Ss:%Sh %tr "%{X-Frontier-Id}>h"

Since the log files can get very big, we run a cron job every night to rotate the log files, keeping 10 days worth. The cron job runs a script that looks like:

#!/bin/bash 

SQUID_DIR=/nthome/bjb/frontier/frontier-cache/squid
FNCRON_DIR=/nthome/bjb/frontier/frontier-cache/utils/cron

$SQUID_DIR/sbin/squid -k rotate 2>&1 >> $FNCRON_DIR/daily.log 

You could also use a logrotate.d script.

Daily might not be enough, however. On heavily used squids we run another cron hourly that checks to see if access.log is greater than a chosen size and if so does an extra rotation.


That's basically it, except for one important thing. For Frontier we monitor our squids remotely using SNMP. This is not turned on by default in squid, so to use it you have to turn it on at compilation time. If you want to use SNMP there are a few more settings needed in the squid.conf, especially the ACL access for whatever machines are allowed to read the SNMP information. Scientific Linux/Redhat Enterprise Linux squid rpms should already have SNMP enabled at compilation time. Therefore, it should be possible to enable monitoring by adding something like the following to your squid.conf:

acl HOST_MONITOR src 127.0.0.1/32 frontier.cern.ch
acl HOST_MONITOR_NAME srcdomain cmsdbsfrontier.cern.ch
acl snmppublic snmp_community public

snmp_access allow snmppublic HOST_MONITOR
snmp_access allow snmppublic HOST_MONITOR_NAME
snmp_access deny all

all in the appropriate places in squid.conf. You may also need to open firewall and/or iptables holes for the addresses on the HOST_MONITOR line above.

One thing you can do is make a dummy installation of our tarball. It can be installed anywhere by any user. Then do a diff of your squid.conf with our squid.conf. For startup and shutdown procedures, you are on your own (unless of course you are using someone else's package).

The compilation options we currently use are (Squid-2.6STABLE18 or later):

--disable-wccp --enable-snmp --disable-ident-lookups --with-large-files

(If you have a 64-bit OS, you don't need --with-large-files)

Responsible: DaveDykstra

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback