naschenweng.info

In a previous post I explained the significance of SNR and attenuation for ADSL lines.

I have spent now a significant amount of time to write scripts to retrieve the ADSL-statistics for my Netgear DG834v4 ADSL router. The script will equally work on any Broadcom based modem-router which allows access to telnet and supports the command "adslctl".

The solution is really simple and consists of one perl-script collating the information and a CGI-script which then produces the stats in real time from your webserver:

I have deployed the solution on my Synology DiskStation DS107+ (FW722) and had the solution running now for a day. If you want to run this on the Synology, you will need to install rrdtool via ipkg. If you want to run it on another Linux-based environment, ensure that you have access to the command-line version of rrdtool and rrdcgi (which comes bundled with rrdtool).

My current setup produces stats every 15 minutes and reports on the following stats:

• Ping statistics. Currently against my local ISP - saix.net, but scripts are easy to modify to include your own pings
• ADSL signal to noise ratio (up / down)
• ADSL attenuation (up / down)
• Power output (up / down)
• Maximum ADSL data rate (up / down)
• ADSL sync speed (up / down)

All stats are reported per hour, day, week and month. In the future I want to enhance the scripts to provide additional useful reports for the Netgear DG834.

By now you are probably asking "Where is the download-link?" --- well here is the thing, my hosting and the popularity of this site is starting to cost me a bundle. So I am introducing donation-ware. It is up to you to decide if you want to donate USD 1,00 or a million (yeah baby) using PayPal via the donation-button below. The donation is absolutely save and you will receive the download-link automatically via email (link is valid for 24 hours). Further, I will email you any updates to the scripts and will look at implementing any recommendations you have.

I think this sounds like a fair deal, especially since it took me in access of 60 hours just getting those scripts done (don't get a fright though - the scripts are simple - that's the beauty of the solution).

If you want to use the scripts, donate below (you require a running version of Perl with Net::Telnet and Net::Ping, rrdtool -- the scripts will run on the Synology out of the box).

In a previous post I described how to install Squid on the Synology. This small tutorial will now explain how to display your usage behaviour in Squid via Webalizer.

You need to have ipkg installed and need to have an understanding on how to include 3rd party apps (i.e. via phpsrc).

1. Install webalizer: ipkg install webalizer
2. Create a new config file: cp /opt/etc/webalizer.conf.sample /opt/etc/webalizer.conf
3. Adjust your log-file entry and point it to your Squid-access log: LogFile /opt/var/squid/logs/access.log
   
4. Adjust your output directory (this is the HTML-directory from which you want to display your stats): OutputDir /volume1/webapps/scripts/squid/usage
5. Create a cronjob entry (remember that the crontab needs to have tabs instead of spaces): 5 0 * * * root /opt/bin/webalizer -F squid -A 100 -C 100 -R 100 -S 100 -U 100 -e 100 -E 100 -Q

The above will create usage statistics every day (5 minutes past midnight) and publish the stats to the output-directory.

Some more cool stuff:

• Use HideUrl to hide sites from the statistics. I typically hide access to all my local servers, since I am only interested in the outbound usage.
• Use GroupUrl and HideUrl together to group certain websites and make it one line-item in the stats. I access the PS3ZA-forum frequently, so I don't want to have every URL show, since it fills up the stats. Using GroupUrl www.ps3za.co.za/forum and HideUrl www.ps3za.co.za/forum will create one line-item in the stats. You can do the same with your RSS-feeds and others.
• To avoid your logs filling up, you want to add a cronjob to rotate your squid access-logs (remember those TABS - this job will rotate the logs 30 minutes past midnight every month): 30 * 1 * * root /opt/sbin/squid -k rotate. Important: Webalizer don't rotate the logs during the month - I was not able to get Webalizer to analyse those logs and you will lose your monthly stats.

In a previous post I detailed how to install Squid on a Synology DS107+. This post explains how you are able to include Squid's cache-manager UI. This post is for advanced users and I am not going to re-explain how to configure a 3rd party application in the DS.

1. Adjust your cachemgr.conf and include your Squid's url. In my case this is an extra line with 172.16.0.97:8080
2. Ensure that you have configured the correct ACLs for the cache-manager. If you took my squid.conf then you are fine.

3. Test the cache-manager: squidclient -h 172.16.0.97 -p 8080 mgr:info

   Code:

   	Squid Object Cache: Version 2.6.STABLE21
   	Start Time:  Fri, 19 Sep 2008 13:57:53 GMT
   	Current Time:  Fri, 19 Sep 2008 14:08:44 GMT
   	Connection information for squid:
   	Number of clients accessing cache:  2
   	Number of HTTP requests received:  154
   	Number of ICP messages received:  0
   	Number of ICP messages sent:  0
   	Number of queued ICP replies:  0
   	Request failure ratio:   0.01
   	Average HTTP requests per minute since start:  14.2
   	Average ICP messages per minute since start:  0.0
   	Select loop called: 135277 times, 4.810 ms avg

4. The Squid installation comes with the cachemgr.cgi in /opt/libexec/cachemgr.cgi. Best way is to symlink (ln -s) the file into your 3rd party-directory

With the above information you will be able to tune Squid accordingly.

There you are, with that 1TB NAS and you surf mostly the same websites and in the process waste plenty of time waiting on downloads. So why not install your own Squid-proxy server on your NAS?

With the Synology and the pre-requisite of having ipkg installed - this takes no more than 10 minutes. In my example, my NAS IP is 172.16.0.97 and my IP range on my LAN is 172.16.0.0 - adjust this accordingly below:

1. Install squid: ipkg install squid

2. Adjust Squid's config-file located in /opt/etc/squid/squid.conf:

   Code:

   	## SQUID CONFIG
   	cache_mgr Gerd@Naschenweng.info
   	## Those are the ports the proxy is going to listen to
   	http_port 172.16.0.97:3128
   	http_port 172.16.0.97:8080
   	#  TAG: visible_hostname
   	#  The host-name of the proxy-server. Can really be anything
   	visible_hostname MuffinStationProxy
   	# DISK CACHE OPTIONS
   	# -----------------------------------------------------------------------------
   	# Disk-cache options. Just adjust the cache-siz (in my case 20GB)
   	cache_replacement_policy lru
   	cache_dir ufs /opt/var/squid/cache/ 20000 16 256
   	minimum_object_size 0 KB
   	maximum_object_size 2097152 KB
   	maximum_object_size_in_memory 1024 KB
   	# MEMORY CACHE OPTIONS
   	# -----------------------------------------------------------------------------
   	#  TAG: cache_mem  (bytes)
   	cache_mem 8 MB
   	memory_replacement_policy lru
   	# ACCESS CONTROLS
   	# -----------------------------------------------------------------------------
   	acl all src 0.0.0.0/0.0.0.0
   	acl manager proto cache_object
   	acl localhost src 127.0.0.1/255.255.255.255
   	acl our_networks src 172.16.0.0/24 172.16.1.0/24
   	acl to_localhost dst 127.0.0.0/8
   	http_access allow manager localhost
   	http_access allow manager our_networks
   	http_access deny manager
   	# Allow all clients from my network
   	http_access allow our_networks
   	# And finally deny all other access to this proxy
   	http_access deny all
   	#Allow ICP queries from everyone
   	icp_access allow all
   	# LOG-FILES
   	# -----------------------------------------------------------------------------
   	access_log /opt/var/squid/logs/access.log squid
   	#cache_log none
   	#cache_log /opt/var/squid/logs/cache.log
   	#cache_access_log none
   	#cache_access_log /opt/var/squid/logs/access.log
   	#cache_store_log none
   	#cache_store_log /opt/var/squid/logs/store.log
   	# OPTIONS FOR TUNING THE CACHE
   	# -----------------------------------------------------------------------------
   	#  TAG: cache
   	#  A list of ACL elements which, if matched, cause the request to
   	#  not be satisfied from the cache and the reply to not be cached.
   	#  In other words, use this to force certain objects to never be cached.
   	#
   	#  You must use the word 'DENY' to indicate the ACL names which should
   	#  NOT be cached.
   	#
   	#  Default is to allow all to be cached
   	#We recommend you to use the following two lines.
   	acl QUERY urlpath_regex cgi-bin \?
   	cache deny QUERY
   	refresh_pattern ^ftp:    1440  20%  10080
   	refresh_pattern ^gopher:  1440  0%  1440
   	refresh_pattern .           0  20%  4320
   	refresh_pattern \.gif 1440 50% 40320 reload-into-ims
   	refresh_pattern \.jpg 1440 50% 40320 reload-into-ims
   	refresh_pattern \.tif 4320 50% 43200
   	refresh_pattern \.png 1440 50% 40320 reload-into-ims
   	refresh_pattern \.jpeg 1440 50% 40320 reload-into-ims
   	refresh_pattern ^http://*.google.*/.* 720 100% 4320
   	# refresh patterns to enable caching of MS windows update
   	refresh_pattern windowsupdate\.microsoft\.com/.*\.(cab|exe|psf) 4320 100% 120960 reload-into-ims
   	refresh_pattern update\.microsoft\.com/.*\.(cab|exe|psf) 4320 100% 120960 reload-into-ims
   	refresh_pattern office\.microsoft\.com/.*\.(cab|exe|psf) 4320 100% 120960
   	refresh_pattern windowsupdate\.com/.*\.(cab|exe|psf) 4320 100% 120960 reload-into-ims
   	refresh_pattern download\.microsoft\.com/.*\.(cab|exe|psf) 4320 100% 120960 reload-into-ims
   	refresh_pattern microsoft\.com 4320 100% 10080
   	pipeline_prefetch on
   	# Apache mod_gzip and mod_deflate known to be broken so don't trust
   	# Apache to signal ETag correctly on such responses
   	acl apache rep_header Server ^Apache
   	broken_vary_encoding allow apache
   	# Leave coredumps in the first cache dir
   	coredump_dir /opt/var/squid/cache
   	# Disable cachemgr password
   	cachemgr_passwd none all

3. Take note from my above config, that I chose a cache-size of 20(!) GB (cache_dir).
4. Validate your Squid configuration with squid -k parse
5. Create the Squid cache-directories with squid -z
6. Start Squid manually to check for errors: squid -NCd1
7. Create a symbolic link so that Squid starts automatically: ln -s /opt/etc/init.d/S80squid /usr/syno/etc/rc.d/
8. Once you restart the NAS, Squid should be started automatically (log files are in /opt/var/squid/logs)

Dummy error: Happened to me - if Squid starts and you don't notice any improvements in browsing speed, make sure that you have your browser's proxy settings adjusted

IMPORTANT: As I have the caching server within a DMZ/Firewall, security-concerns are secondary. All users having access to the LAN and fall within the IP-range will automatically have access to the caching-proxy. The implementation of Squid was for improving the browsing/web-experience (speed has improved by almost 200% and average bandwidth consumption dropped by 30%).

Read more »

In a previous article I showed how to make the Synology reboot a router via a cronjob.

There are many more options - such as automatically rebooting your Synology NAS thereafter - you would just extend the script by another two lines:

Code: