Howto Cache Youtube with SQUID


What is LUSCA/SQUID ?

LUSCA is an advance version or Fork of  SQUID 2. The Lusca project aims to fix the shortcomings in the Squid-2. It also supports a variety of clustering protocols. By Using it, you can cache some dynamic contents that you previously can’t do with the squid.
For example
#  Video Cachingi.e Youtube / tube etc . . .
#  Windows / Linux Updates / Anti-virus , Anti-Malware i.e. Avira/ Avast / MBAM etc . . .
#  Well known sites i.e. facebook / google / yahoo etch. etch.
#  Download caching mp3′s/mpeg/avi etc . . .

Advantages of Youtube Caching   !!!

In most part of the world, bandwidth is very expensive, therefore it is (in some scenarios) very useful to Cache Youtube videos or any other flash videos, so if one of user downloads video / flash file , why again the same user or other user can’t download the same file from the CACHE, why he sucking the internet pipe for same content again n again?
Peoples on same LAN ,sometimes watch similar videos. If I put some youtube video link on on FACEBOOK, TWITTER or likewise , and all my friend will  watch that video and that particular video gets viewed many times in few hours. Usually the videos are shared over facebook or other social networking sites so the chances are high for multiple hits per popular videos for my lan users / friends.
This is the reason why I wrote this article. I have implemented Ubuntu with LUSCA/ Squid on it and its working great, but to achieve some results you need to have some TB of storage drives in your proxy machine.

Disadvantages of Youtube Caching   !!!

The chances, that another user will watch the same video, is really slim. if I search for something specific on youtube, i get more then hundreds of search results for same video. What is the chance that another user will search for the same thing, and will click on the same link / result? Youtube hosts more than 10 million videos. Which is too much to cache anyway. You need lot of space to cache videos. Also accordingly you will be needing ultra modern fast hardware with tons of RAM to handle such kind of cache giant. anyhow Try it
We will divide this article in following Sections
1#  Installing SQUID / LUSCA in UBUNTU
2#  Setting up SQUID / LUSCA Configuration files
3#  Performing some Tests, testing your Cache HIT
4# Using ZPH TOS to deliver cached contents to clients vai mikrotik at full LAN speed, Bypassing the User Queue for cached contents.

1#  Installing SQUID / LUSCA in UBUNTU

I assume your ubuntu box have 2 interfaces configured, one for LAN and second for WAN. You have internet sharing already configured. Now moving on to LUSCA / SQUID installation.
Download LUSCA source and compile it using,
1mkdir /temp
2cd /temp
4tar xzvf LUSCA_HEAD-r14809.tar.gz
Update & Install some tools for ubuntu
1sudo apt-get update
2sudo apt-get install gcc build-essential sharutils ccze libzip-dev automake1.9
3cd LUSCA_HEAD-r14809
Now compile LUSCA with following options
1./configure '--prefix=/usr/local/squid' '--enable-removal-policies=heap,lru' '--disable-dependency-tracking' '--disable-arp-acl' '--disable-cache-digests' '--enable-cachemgr-hostname=localhost' '--disable-delay-pools' '--enable-epoll' '--enable-external-acl-helpers=ip_user' '--disable-ident-lookups' '--enable-linux-netfilter' '--disable-referer-log' '--enable-removal-policies=heap,lru' '--disable-snmp' '--disable-ssl' '--enable-storeio=aufs,null,coss' '--disable-useragent-log' '--disable-wccpv2' '--with-aio' '--with-maxfd=1048576' '--with-dl' '--with-pthreads' 'build_alias=i686-redhat-linux-gnu' 'host_alias=i686-redhat-linux-gnu' 'targe_alias=i686-redhat-linux-gnu''--enable-truncate' '--disable-unlinkd' '--with-large-files' '--disable-htcp'
2sudo make all
3sudo make install
All of Lusca/Squid configuration files can be found at
/usr/local/squid/etc/
and squid executable can be found at
/usr/local/squid/sbin/
Now We will edit squid.conf file to make it customize according to our requirements by . . .
1nano /usr/local/squid/etc/squid.conf
Delete all previously lines , and paste the following lines.
Remember following squid.conf is not very neat and clean , you will find many un necessary junk entries in it, but as I didn’t had time to clean them all, so you may clean them as per your targets and goals.
001# SQUID / LUSCS TEST CONFIG FILE
002# Email: aacable@hotmail.com
003# Web  : http://bolozer.com
004 
005# PORT and Transparent Option
006http_port 8080 transparent
007server_http11 on
008icp_port 0
009 
010# Cache Directory , modify it according to your system.
011# but first create directory in root by mkdir /cache1
012# and then issue this command  chown proxy:proxy /cache1
013# [for ubuntu user is proxy, in Fedora user is SQUID]
014# I have set 10 GB for caching reserved just for caching ,
015# adjust it according to your need.
016# My recommendation is to have one cache_dir per drive. zzz
017 
018#store_dir_select_algorithm round-robin
019cache_dir aufs /cache1 10000 16 256
020cache_replacement_policy heap LFUDA
021memory_replacement_policy heap GDSF
022 
023# If you want to enable DATE time n SQUID Logs,use following
024emulate_httpd_log on
025logformat squid %tl %6tr %>a %Ss/%03Hs %<st %rm %ru %un %Sh/%<A %mt
026log_fqdn off
027 
028# How much days to keep users access web logs
029# You need to rotate your log files with a cron job. For example:
030# 0 0 * * * /usr/local/squid/bin/squid -k rotate
031logfile_rotate 14
032debug_options ALL,1
033cache_access_log /var/log/squid/access.log
034cache_log /var/log/squid/cache.log
035cache_store_log /var/log/squid/store.log
036 
037#I used DNSAMSQ service for fast dns resolving
038#so install by using "apt-get install dnsmasq" first
039dns_nameservers 127.0.0.1 192.168.2.4 192.168.2.3
040ftp_user anonymous@
041ftp_list_width 32
042ftp_passive on
043ftp_sanitycheck on
044 
045#ACL Section
046acl all src 0.0.0.0/0.0.0.0
047acl manager proto cache_object
048acl localhost src 127.0.0.1/255.255.255.255
049acl to_localhost dst 127.0.0.0/8
050acl SSL_ports port 443 563 # https, snews
051acl SSL_ports port 873 # rsync
052acl Safe_ports port 80 # http
053acl Safe_ports port 21 # ftp
054acl Safe_ports port 443 563 # https, snews
055acl Safe_ports port 70 # gopher
056acl Safe_ports port 210 # wais
057acl Safe_ports port 1025-65535 # unregistered ports
058acl Safe_ports port 280 # http-mgmt
059acl Safe_ports port 488 # gss-http
060acl Safe_ports port 591 # filemaker
061acl Safe_ports port 777 # multiling http
062acl Safe_ports port 631 # cups
063acl Safe_ports port 873 # rsync
064acl Safe_ports port 901 # SWAT
065acl purge method PURGE
066acl CONNECT method CONNECT
067http_access allow manager localhost
068http_access deny manager
069http_access allow purge localhost
070http_access deny purge
071http_access deny !Safe_ports
072http_access deny CONNECT !SSL_ports
073http_access allow localhost
074http_access allow all
075http_reply_access allow all
076icp_access allow all
077 
078#==========================
079# Administrative Parameters
080#==========================
081 
082# I used UBUNTU so user is proxy, in FEDORA you may use use squid
083cache_effective_user proxy
084cache_effective_group proxy
085cache_mgr aacable@hotmail.com
086visible_hostname proxy.aacable.net
087unique_hostname aacable@hotmail.com
088 
089#=============
090# ACCELERATOR
091#=============
092half_closed_clients off
093quick_abort_min 0 KB
094quick_abort_max 0 KB
095quick_abort_pct 100
096vary_ignore_expire on
097reload_into_ims on
098pipeline_prefetch on
099range_offset_limit 512 KB
100read_timeout 30 minutes
101log_fqdn off
102memory_pools off
103 
104# If you want to hide your proxy machine from being detected at various site use following
105via off
106 
107#============================================
108# OPTIONS WHICH AFFECT THE CACHE SIZE / zaib
109#============================================
110# If you have 4GB memory in Squid box, we will use formula of 1/3
111# You can adjust it according to your need. IF squid is taking too much of RAM
112# Then decrease it to 128 MB or even less.
113cache_mem 128 MB
114minimum_object_size 0 bytes
115maximum_object_size 700 MB
116maximum_object_size_in_memory 128 KB
117 
118#============================================================$
119# SNMP , if you want to generate graphs for SQUID via MRTG
120#============================================================$
121#acl snmppublic snmp_community gl
122#snmp_port 3401
123#snmp_access allow snmppublic all
124#snmp_access allow all
125 
126#============================================================
127# ZPH , To enable cache content to be delivered at full lan speed,
128# To bypass the queue at MT.
129#============================================================
130tcp_outgoing_tos 0x30 all
131zph_mode tos
132zph_local 0x30
133zph_parent 0
134zph_option 136
135 
136# Caching Youtube
137acl videocache_allow_url url_regex -i \.youtube\.com\/get_video\?
138acl videocache_allow_url url_regex -i \.youtube\.com\/videoplayback \.youtube\.com\/videoplay \.youtube\.com\/get_video\?
139acl videocache_allow_url url_regex -i \.youtube\.[a-z][a-z]\/videoplayback \.youtube\.[a-z][a-z]\/videoplay \.youtube\.[a-z][a-z]\/get_video\?
140acl videocache_allow_url url_regex -i \.googlevideo\.com\/videoplayback \.googlevideo\.com\/videoplay \.googlevideo\.com\/get_video\?
141acl videocache_allow_url url_regex -i \.google\.com\/videoplayback \.google\.com\/videoplay \.google\.com\/get_video\?
142acl videocache_allow_url url_regex -i \.google\.[a-z][a-z]\/videoplayback \.google\.[a-z][a-z]\/videoplay \.google\.[a-z][a-z]\/get_video\?
143acl videocache_allow_url url_regex -i proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/
144acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/
145acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
146acl videocache_allow_url url_regex -i \.vimeo\.com\/(.*)\.(flv|mp4)
147acl videocache_allow_url url_regex -i va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
148acl videocache_allow_url url_regex -i \.youporn\.com\/(.*)\.flv
149acl videocache_allow_url url_regex -i \.msn\.com\.edgesuite\.net\/(.*)\.flv
150acl videocache_allow_url url_regex -i \.tube8\.com\/(.*)\.(flv|3gp)
151acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv
152acl videocache_allow_url url_regex -i \.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram|m4v)
153acl videocache_allow_url url_regex -i \.apniisp\.com\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram|m4v)
154acl videocache_allow_url url_regex -i \.break\.com\/(.*)\.(flv|mp4)
155acl videocache_allow_url url_regex -i redtube\.com\/(.*)\.flv
156acl videocache_allow_dom dstdomain .mccont.com .metacafe.com .cdn.dailymotion.com
157acl videocache_deny_dom  dstdomain .download.youporn.com .static.blip.tv
158acl dontrewrite url_regex redbot\.org \.php
159acl getmethod method GET
160 
161storeurl_access deny dontrewrite
162storeurl_access deny !getmethod
163storeurl_access deny videocache_deny_dom
164storeurl_access allow videocache_allow_url
165storeurl_access allow videocache_allow_dom
166storeurl_access deny all
167 
168storeurl_rewrite_program /etc/squid/storeurl.pl
169storeurl_rewrite_children 7
170storeurl_rewrite_concurrency 10
171 
172acl store_rewrite_list urlpath_regex -i \/(get_video\?|videodownload\?|videoplayback.*id)
173acl store_rewrite_list urlpath_regex -i \.flv$ \.mp3$ \.mp4$ \.swf$ \
174storeurl_access allow store_rewrite_list
175storeurl_access deny all
176 
177refresh_pattern -i \.flv$ 5259487 99999999% 5259487  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth store-stale negative-ttl=0
178refresh_pattern -i \.mp3$ 5259487 99999999% 5259487  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth store-stale negative-ttl=0
179refresh_pattern -i \.mp4$ 5259487 99999999% 5259487  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth store-stale negative-ttl=0
180refresh_pattern -i \.swf$ 5259487 99999999% 5259487  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth store-stale negative-ttl=0
181refresh_pattern -i \.gif$ 5259487 99999999% 5259487  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth store-stale negative-ttl=0
182refresh_pattern -i \.jpg$ 5259487 99999999% 5259487  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth store-stale negative-ttl=0
183refresh_pattern -i \.jpeg$ 5259487 99999999% 5259487  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private  ignore-auth store-stale negative-ttl=0
184refresh_pattern -i \.exe$ 5259487 99999999% 5259487  override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-no-store ignore-private  ignore-auth store-stale negative-ttl=0
185 
186# 1 year = 525600 mins, 1 month = 43200 mins, 1 day = 1440
187refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?)    43200 999999% 43200 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims store-stale
188refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.flv?)    43200 999999% 43200 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims store-stale
189refresh_pattern \.(ico|video-stats) 43200 999999% 43200 override-expire ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth override-lastmod  negative-ttl=10080 store-stale
190refresh_pattern \.etology\?                                     43200 999999% 43200 override-expire ignore-reload ignore-no-cache store-stale
191refresh_pattern galleries\.video(\?|sz)                         43200 999999% 43200 override-expire ignore-reload ignore-no-cache store-stale
192refresh_pattern brazzers\?                                      43200 999999% 43200 override-expire ignore-reload ignore-no-cache store-stale
193refresh_pattern \.adtology\?                                    43200 999999% 43200 override-expire ignore-reload ignore-no-cache store-stale
194refresh_pattern ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 43200 20% 43200 ignore-no-cache ignore-no-store ignore-private override-expire ignore-reload ignore-auth  store-stale negative-ttl=40320 max-stale=10
195refresh_pattern ^.*safebrowsing.*google  43200 999999% 43200 override-expire ignore-reload ignore-no-cache ignore-private ignore-auth  negative-ttl=10080 store-stale
196refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk)    43200 999999% 43200 override-expire ignore-reload ignore-private store-stale negative-ttl=10080
197refresh_pattern ytimg\.com.*\.jpg                                       43200 999999% 43200 override-expire ignore-reload store-stale
198refresh_pattern images\.friendster\.com.*\.(png|gif)                    43200 999999% 43200 override-expire ignore-reload store-stale
199refresh_pattern garena\.com                                             43200 999999% 43200 override-expire reload-into-ims store-stale
200refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png)          43200 999999% 43200 override-expire ignore-reload store-stale
201refresh_pattern vid\.akm\.dailymotion\.com.*\.on2\?                     43200 999999% 43200 ignore-no-cache override-expire override-lastmod store-stale
202refresh_pattern mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png)    43200 999999% 43200 reload-into-ims override-expire ignore-private    store-stale
203refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\.                     43200 999999% 43200 reload-into-ims ignore-no-cache ignore-no-store ignore-reload override-expire store-stale
204refresh_pattern ^http:\/\/www.onemanga.com.*\/                          43200 999999% 43200 reload-into-ims ignore-no-cache ignore-no-store ignore-reload override-expire store-stale
205refresh_pattern ^http://v\.okezone\.com/get_video\/([a-zA-Z0-9]) 43200 999999% 43200 override-expire ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth override-lastmod  negative-ttl=10080 store-stale
206 
207#images facebook
208refresh_pattern -i \.facebook.com.*\.(jpg|png|gif)                      129600 999999% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store store-stale
209refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3)                 129600 999999% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store store-stale
210refresh_pattern  static\.ak\.fbcdn\.net*\.(jpg|gif|png)                 129600 999999% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store store-stale
211refresh_pattern ^http:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png)        129600 999999% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store store-stale
212 
213#All File
214refresh_pattern -i \.(3gp|7z|ace|asx|bin|deb|divx|dvr-ms|ram|rpm|exe|inc|cab|qt)       43200 999999% 43200 ignore-no-cache ignore-no-store  override-expire override-lastmod reload-into-ims store-stale
215refresh_pattern -i \.(rar|jar|gz|tgz|bz2|iso|m1v|m2(v|p)|mo(d|v)|arj|lha|lzh|zip|tar)  43200 999999% 43200 ignore-no-cache ignore-no-store  override-expire override-lastmod reload-into-ims store-stale
216refresh_pattern -i \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|dat|ad|txt|dll)         43200 999999% 43200 ignore-no-cache ignore-no-store  override-expire override-lastmod reload-into-ims store-stale
217refresh_pattern -i \.(avi|ac4|mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p)|og(x|v|a|g)|rm|r(a|p)m|snd|vob) 43200 999999% 43200 ignore-no-cache ignore-no-store  override-expire override-lastmod reload-into-ims store-stale
218refresh_pattern -i \.(pp(t?x)|s|t)|pdf|rtf|wax|wm(a|v)|wmx|wpl|cb(r|z|t)|xl(s?x)|do(c?x)|flv|x-flv) 43200 999999% 43200 ignore-no-cache ignore-no-store  override-expire override-lastmod reload-into-ims store-stale
219 
220refresh_pattern -i (/cgi-bin/|\?)  0  0%  0
221refresh_pattern ^gopher:        1440    0%      1440
222refresh_pattern ^ftp:           10080   95% 43200 override-lastmod reload-into-ims store-stale
223refresh_pattern .               1440    95% 43200 override-lastmod reload-into-ims store-stale
Now create cache dir and logs files , and assign them necessary permissions.
1mkdir /cache1
2chown proxy:proxy /cache1
3mkdir /var/log/squid
4chmod 777 /var/log/squid
Now initialize cache dir by
1/usr/local/squid/sbin/squid -z
SOTEURL.PL
Now We have to create an important file name storeurl.pl , which is very important and actually it does the
main job to pull video from cache.
1mkdir /etc/squid
2touch /etc/squid/storeurl.pl
3chmod +x /etc/squid/storeurl.pl
4nano /etc/squid/storeurl.pl
Now paste the following lines, then Save and exit.
001#!/usr/bin/perl
002# $Rev$
003# by chudy_fernandez@yahoo.com
005$|=1;
006while (<>) {
007@X = split;
008$X[1] =~ s/&sig=.*//;
009$x = $X[0] . " ";
010$_ = $X[1];
011$u = $X[1];
012 
013# compatibility for old cached get_video?video_id
014if (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?(videoplayback\?id=.*?|video_id=.*?)\&(.*?)/) {
015$z = $2; $z =~ s/video_id=/get_video?video_id=/;
016print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $z . "\n";
017 
018# youtube HD itag=22
019} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?\&(itag=22).*?\&(id=[a-zA-Z0-9]*)/) {
020print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $2 . "&" . $3 . "\n";
021 
022# youtube Normal screen always HD itag 35, Normal screen never HD itag 34, itag=18 <--normal?
023} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?\&(itag=[0-9]*).*?\&(id=[a-zA-Z0-9]*)/) {
024print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $3 . "\n";
025 
026} elsif (m/^http:\/\/www\.google-analytics\.com\/__utm\.gif\?.*/) {
028 
029#Cache High Latency Ads
030} elsif (m/^http:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|\.googlesyndication\.com|yieldmanager|cpxinteractive)(.*)/) {
031$y = $3;$z = $2;
032for ($y) {
033s/pixel;.*/pixel/;
034s/activity;.*/activity/;
035s/(imgad[^&]*).*/\1/;
036s/;ord=[?0-9]*//;
037s/;&timestamp=[0-9]*//;
038s/[&?]correlator=[0-9]*//;
039s/&cookie=[^&]*//;
040s/&ga_hid=[^&]*//;
041s/&ga_vid=[^&]*//;
042s/&ga_sid=[^&]*//;
043# s/&prev_slotnames=[^&]*//
044# s/&u_his=[^&]*//;
045s/&dt=[^&]*//;
046s/&dtd=[^&]*//;
047s/&lmt=[^&]*//;
048s/(&alternate_ad_url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
049s/(&url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
050s/(&ref=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
051s/(&cookie=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
052s/[;&?]ord=[?0-9]*//;
053s/[;&]mpvid=[^&;]*//;
054s/&xpc=[^&]*//;
055# yieldmanager
056s/\?clickTag=[^&]*//;
057s/&u=[^&]*//;
058s/&slotname=[^&]*//;
059s/&page_slots=[^&]*//;
060}
061print $x . "http://" . $1 . $2 . $y . "\n";
062 
063#cache high latency ads
064} elsif (m/^http:\/\/(.*?)\/(ads)\?(.*?)/) {
065print $x . "http://" . $1 . "/" . $2  . "\n";
066 
067# spicific servers starts here....
068} elsif (m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) {
069print $x . "http://" . $1 . "\n";
070 
071#cdn, varialble 1st path
072} elsif (($u =~ /filehippo/) && (m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {
073@y = ($1,$2,$4,$5);
074$y[0] =~ s/[a-z0-9]{2,5}/cdn./;
075print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";
076 
077#rapidshare
078} elsif (($u =~ /rapidshare/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)([a-z]*\.[^\/]{3}\/[a-z]*\/[0-9]*)\/(.*?)\/([^\/\?\&]{4,})$/)) {
079print $x . "http://cdn." . $3 . "/SQUIDINTERNAL/" . $5 . "\n";
080 
081} elsif (($u =~ /maxporn/) && (m/^http:\/\/([^\/]*?)\/(.*?)\/([^\/]*?)(\?.*)?$/)) {
082print $x . "http://" . $1 . "/SQUIDINTERNAL/" . $3 . "\n";
083 
084#like porn hub variables url and center part of the path, filename etention 3 or 4 with or without ? at the end
085} elsif (($u =~ /tube8|pornhub|xvideos/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)\.([a-z]*[0-9]?\.[^\/]{3}\/[a-z]*)(.*?)((\/[a-z]*)?(\/[^\/]*){4}\.[^\/\?]{3,4})(\?.*)?$/)) {
086print $x . "http://cdn." . $3 . $5 . "\n";
087#...spicific servers end here.
088 
089#photos-X.ak.fbcdn.net where X a-z
090} elsif (m/^http:\/\/photos-[a-z].ak.fbcdn.net\/(.*)/) {
091print $x . "http://photos.ak.fbcdn.net/" . $1  . "\n";
092 
093#general purpose for cdn servers. add above your specific servers.
094} elsif (m/^http:\/\/([0-9.]*?)\/\/(.*?)\.(.*)\?(.*?)/) {
095print $x . "http://squid-cdn-url//" . $2  . "." . $3 . "\n";
096 
097#for yimg.com doubled
098} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/) {
099print $x . "http://cdn.yimg.com/"  . $3 . "\n";
100 
101#for yimg.com with &sig=
102} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*)/) {
103@y = ($1,$2);
104$y[0] =~ s/[a-z]+[0-9]+/cdn/;
105$y[1] =~ s/&sig=.*//;
106print $x . "http://" . $y[0] . ".yimg.com/"  . $y[1] . "\n";
107 
108#generic http://variable.domain.com/path/filename."ext" or "exte" with or withour "?"
109} elsif (m/^http:\/\/(.*)([^\.\-]*?\..*?)\/(.*)\.([^\/\?\&]{3,4})(\?.*)?$/) {
110@y = ($1,$2,$3,$4);
111$y[0] =~ s/(([a-zA-A-]+[0-9-]+)|(.*cdn.*)|(.*cache.*))/cdn/;
112print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";
113 
115} elsif (m/^http:\/\/(([A-Za-z]+[0-9-]+)*?|.*cdn.*|.*cache.*)\.(.*?)\.(.*?)\/(.*)$/) {
116print $x . "http://cdn." . $3 . "." . $4 . "/" . $5 .  "\n";
117 
118# spicific extention that ends with ?
119} elsif (m/^http:\/\/(.*?)\/(.*?)\.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|on2)\?(.*)/) {
120print $x . "http://" . $1 . "/" . $2  . "." . $3 . "\n";
121 
122# all that ends with ;
123} elsif (m/^http:\/\/(.*?)\/(.*?)\;(.*)/) {
124print $x . "http://" . $1 . "/" . $2  . "\n";
125 
126} else {
127print $x . $_ . "\n";
128}
129}
Now start SQUIDServer by
1/usr/local/squid/sbin/squid
TIP:
To start SQUID Server in Debug mode, to check any erros, use

1/usr/local/squid/sbin/squid -d1

3# TESTING YOUR CACHE HIT

It’s time to hit the ROAD and do performing some tests.

YOUTUBE TEST

Open Youtube and watch any Video. After complete download, Check the same video from another client. You will notice that it download very quickly , you can watch the bar moving fast.
As Shown in the example Below . . .

monitor the Squid access LOG. You will see cache hit TPC_HIT for this video.
As Shown in the example Below . . .

MUSIC DOWNLOAD TEST

Now test any music download. For example Go to
http://www.apniisp.com/songs/indian-movie-songs/ladies-vs-ricky-bahl/690/1.html
As Shown in the example Below . . .


and download any song , after its downloaded, goto 2nd client pc, and download the same song, and monitor the Squid access LOG. You will see cache hit TPC_HIT for this song.
As Shown in the example Below . . .

EXE / PROGRAM  DOWNLOAD TEST

Now test any .exe file download.
Goto http://www.rarlabs.com and download any package. After Download completes, goto 2nd client pc , and download the same file again. and monitor the Squid access LOG. You will see cache hit TPC_HIT for this file.
As Shown in the example Below . . .

SQUID LOGS

More to come , , ,
Title : Howto Cache Youtube with SQUID
Description : What is LUSCA/SQUID ? LUSCA is an advance version or Fork of  SQUID 2 . The Lusca project aims to fix the shortcomings in the Squid-2. I...

1 Response to "Howto Cache Youtube with SQUID"

  1. buat artikel yang kreatif mas jangan asal CoPas.

    http://aacable.wordpress.com/2012/01/11/howto-cache-youtube-with-squid-lusca-and-bypass-cached-videos-from-mikrotik-queue/

    BalasHapus