Advert-Blocker 0.5

Browsers, email, chat, etc.
Message
Author
User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

Advert-Blocker 0.5

#1 Post by sc0ttman »

Pup-Advert-Blocker
block many online ads in any browser


This tool will download a list of known advertising servers and add them to your "/etc/hosts" file - blocking them from you.

Then you can browse the web without seeing or downloading a large number of popular and very annoying online ads :D

This tool is very simple, but does the trick nicely, I think.

Using all services currently blocks around 30,300 ad servers (Aug 2010)

Suggestions and improvements still welcome.

Changelog:

version 0.5
- added a fix I accidentally un-added! :roll:
- changed detection of empty list
- cleaned the code a tiny little bit
- added a timeout to wget (avoids endless pausing)

version 0.4
- added (only some of) Barry Ks bugfixes (whoops! fixed in v0.5)

version 0.3
- fixed duplicate removal (using dos2unix to convert carriage returns)
- updated help GUI

version 0.2
- added more servers
- servers can be combined or chosen separately
- ads can be disabled by not selecting any servers
- (duplicates are still found in the final built list!) :(

(Thanks to this thread) (thx 2 lguleder etc)
Attachments
pup-ad-blocker.png
(40.7 KiB) Downloaded 7896 times
pup-advert-blocker.pet
version 0.5
(3.13 KiB) Downloaded 4029 times
pup-advert-blocker.pet
version 0.4
(3.15 KiB) Downloaded 1973 times
pup-advert-blocker.pet
version 0.3
(2.99 KiB) Downloaded 1920 times
pup-advert-blocker.pet
version 0.2
(3.03 KiB) Downloaded 1844 times
pup-advert-blocker.pet
version 0.1
(2.74 KiB) Downloaded 1876 times
Last edited by sc0ttman on Thu 02 Sep 2010, 05:34, edited 16 times in total.
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

User avatar
Iguleder
Posts: 2026
Joined: Tue 11 Aug 2009, 09:36
Location: Israel, somewhere in the beautiful desert
Contact:

#2 Post by Iguleder »

Once of those moments when I wonder why the forums don't have a "like" button :lol:

Goo'job!
[url=http://dimakrasner.com/]My homepage[/url]
[url=https://github.com/dimkr]My GitHub profile[/url]

User avatar
chrismt
Posts: 255
Joined: Wed 21 Apr 2010, 04:37

#3 Post by chrismt »

@ Scottman

Thanks!

This is what I was asking someone to make

You can learn a lot of things from Hostsman, a windows Hosts Manager that blocks ads and malware

Here

http://www.abelhadigital.com/hostsman

Image

This PET. is my favorite!

You can also add more adblock hosts providers like MVPS etc

You not only block ads but also malware sites in Windows with Hostsman because it has got many hosts providers as default.

You can even update all the hosts providers automatically and delete duplicates


Check it out!

Regards

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#4 Post by technosaurus »

I made a simple command line script that will integrate both lists and sort the duplicates out (the commented out line also removes some know adult content)

Code: Select all

#!/bin/sh
wget -c -O hosts1 http://sysctl.org/cameleon/hosts
wget -c -O hosts2 http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext
#wget -c -O hosts3 http://www.mvps.org/winhelp2002/hosts.txt #removes adult content
cat /etc/hosts hosts1 hosts2 hosts3 |grep ^[1-9] |sed "s/\t//g"|sort |uniq >/etc/hosts
#rm -f hosts1 hosts2
other possible host lists:
http://my.opera.com/Tamil/blog/ad-block
http://code.google.com/p/fanboy-adblock ... ce/browse/
http://www.technobeta.com/download/urlfilter.ini
http://my.opera.com/Ghostshaw/blog/
http://www.technobeta.com/download/urlfilter.ini

here is the link for adblock plus, but it will require some manipulation to make it work
https://easylist-downloads.adblockplus.org/easylist.txt
https://easylist-downloads.adblockplus. ... rivacy.txt
more here: http://adblockplus.org/en/subscriptions and here:
http://easylist.adblockplus.org/
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

#5 Post by sc0ttman »

version 0.2 released - see main post.
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#6 Post by technosaurus »

Nice work!
Bugfix#1

Code: Select all

cat /tmp/adlist1 /tmp/adlist2 /tmp/adlist3 /tmp/adlist4 |grep ^[1-9] |sed "s/\t//g"|sort |uniq -u 
leave off the -u from uniq otherwise you will miss anything that is on more than one list

Bugreport
in order for lines to be seen as true duplicates, the formatting has to be exactly the same (uniq compares the entire line as a string), so you may need to pass it through sed a couple of times to replace combinations of tabs/spaces/etc... with a single space in all entries and no trailing tabs/spaces possibly leading tabs/spaces as well

sed 's/[ \t]*/ /' #replace combinations of tabs/spaces with a space
sed 's/[ \t]*$//' #remove trailing spaces/tabs
sed 's/[^ \t]*//' #remove leading spaces/tabs
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

#7 Post by sc0ttman »

updated to 0.3

all duplicates now removed from hosts file automatically :D

I personally think this (or similar) should be included in Puppy, not because I made it of course, but because it makes browsing soooooo much faster and less annoying!

thanks for the help guys
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

User avatar
BarryK
Puppy Master
Posts: 9392
Joined: Mon 09 May 2005, 09:23
Location: Perth, Western Australia
Contact:

#8 Post by BarryK »

Hi, great app, I posted a bug fix for 0.3:

http://bkhome.org/blog/?viewDetailed=01777
[url]https://bkhome.org/news/[/url]

User avatar
jemimah
Posts: 4307
Joined: Wed 26 Aug 2009, 19:56
Location: Tampa, FL
Contact:

#9 Post by jemimah »

Nice. I've added it to Puppeee and Fluppy.

User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

#10 Post by sc0ttman »

version 0.4 is out..

Added Barry Ks bugfixes.. but removed hard-coding of app title.

Thanks for the help! :)
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#11 Post by technosaurus »

I adapted it for standard dialog for the minimalist puppies using PupnGo. This will work with dialog on the command line or Xdialog (gtk1 or gtk2) in X, but not as pretty either way (I removed any non-essentials to work with dialog)

Code: Select all

#!/bin/sh
[ -z $DISPLAY ] && DIALOG=dialog || DIALOG=Xdialog
for x in `$DIALOG --stdout --checklist "Choose your ad blocking service(s)" 0 0 4 1 "mvps.org" ON 2 "systcl.org" ON 3 "technobeta.com" ON 4 "yoyo.org" ON |tr "/" " " |tr '\"' ' '`; do
	case $x in
	1)wget -c -4 -t 0 -O /tmp/adlist1 'http://www.mvps.org/winhelp2002/hosts.txt';;
	2)wget -c -4 -t 0 -O /tmp/adlist2 'http://sysctl.org/cameleon/hosts';;
	3)wget -c -4 -t 0 -O /tmp/adlist3 'http://www.technobeta.com/download/urlfilter.ini';;
	4)wget -c -4 -t 0 -O /tmp/adlist4 'http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext';;
	*)echo $x;;
	esac
done
touch /tmp/adlist{1,2,3,4}
cat /etc/hosts /tmp/adlist{1,2,3,4} |grep ^[1-9] |sed '/^$/d' |sed "s/\t/ /g"|sed '/^#/d' |sed 's/  / /g' |dos2unix -u |sort |uniq > /etc/hosts
rm -f /tmp/adlist{1,2,3,4}
Edit: this one should work better

Code: Select all

#!/bin/sh
[ -z $DISPLAY ] && DIALOG=dialog || DIALOG=Xdialog
[ ! -f /etc/hosts.usr ] && touch /etc/hosts && cp -f /etc/hosts /etc/hosts.usr #echo 127.0.0.1 localhost puppypc >/etc/hosts #r1 maybe user added others
for x in `$DIALOG --stdout --checklist "Choose your ad blocking service(s)" 0 0 5 1 "mvps.org" ON 2 "systcl.org" ON 3 "technobeta.com" ON 4 "yoyo.org" ON 5 "turn off adblocking" off |tr "/" " " |tr '\"' ' '`; do
   case $x in
   1)wget -c -4 -t 0 -O /tmp/adlist1 'http://www.mvps.org/winhelp2002/hosts.txt';;
   2)wget -c -4 -t 0 -O /tmp/adlist2 'http://sysctl.org/cameleon/hosts';;
   3)wget -c -4 -t 0 -O /tmp/adlist3 'http://www.technobeta.com/download/urlfilter.ini';;
   4)wget -c -4 -t 0 -O /tmp/adlist4 'http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext';;
   5)ln -sf /etc/hosts.usr /etc/hosts && exit;;
   *)echo $x;;
   esac
done
touch /tmp/adlist{1,2,3,4}
cat /etc/hosts.usr /tmp/adlist{1,2,3,4} |sed 's/^[ \t]*//' |sed 's/\t/ /g' |sed 's/  / /g' |grep ^[1-9] |dos2unix -u |sort |uniq > /etc/hosts.adblock
ln -sf /etc/hosts.adblock /etc/hosts
Last edited by technosaurus on Tue 31 Aug 2010, 02:03, edited 4 times in total.
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

User avatar
abushcrafter
Posts: 1418
Joined: Fri 30 Oct 2009, 16:57
Location: England
Contact:

#12 Post by abushcrafter »

Could you add the option to disable ad blocking please. So when one is on sites like lastfm there services like there player works.
[url=http://www.adobe.com/flashplatform/]adobe flash is rubbish![/url]
My Quote:"Humans are stupid, though some are clever but stupid." http://www.dependent.de/media/audio/mp3/System_Syn_Heres_to_You.zip http://www.systemsyn.com/

User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

#13 Post by sc0ttman »

abushcrafter wrote:Could you add the option to disable ad blocking please. So when one is on sites like lastfm there services like there player works.
My GUI already has that feature. Just don't select any lists and click Start'.
You just need to restart your browser, I think..

The code above, by techonosaurus, probably just needs the '/etc/hosts' part removed from the 'cat' command, in order to enable the removal of all ad lists (I think - correct me if I am wrong!)
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

User avatar
abushcrafter
Posts: 1418
Joined: Fri 30 Oct 2009, 16:57
Location: England
Contact:

#14 Post by abushcrafter »

sc0ttman wrote:
abushcrafter wrote:Could you add the option to disable ad blocking please. So when one is on sites like lastfm there services like there player works.
My GUI already has that feature. Just don't select any lists and click Start'.
You just need to restart your browser, I think..

The code above, by techonosaurus, probably just needs the '/etc/hosts' part removed from the 'cat' command, in order to enable the removal of all ad lists (I think - correct me if I am wrong!)
Great!
[url=http://www.adobe.com/flashplatform/]adobe flash is rubbish![/url]
My Quote:"Humans are stupid, though some are clever but stupid." http://www.dependent.de/media/audio/mp3/System_Syn_Heres_to_You.zip http://www.systemsyn.com/

User avatar
sc0ttman
Posts: 2812
Joined: Wed 16 Sep 2009, 05:44
Location: UK

#15 Post by sc0ttman »

Not so great.. I'm dumb.. Removing the /etc/hosts from technosaurus' code will just break it and do weird things..
But you can do it in mine, that was correct at least..
[b][url=https://bit.ly/2KjtxoD]Pkg[/url], [url=https://bit.ly/2U6dzxV]mdsh[/url], [url=https://bit.ly/2G49OE8]Woofy[/url], [url=http://goo.gl/bzBU1]Akita[/url], [url=http://goo.gl/SO5ug]VLC-GTK[/url], [url=https://tiny.cc/c2hnfz]Search[/url][/b]

User avatar
chrismt
Posts: 255
Joined: Wed 21 Apr 2010, 04:37

#16 Post by chrismt »

version 0.4 is giving me an error

I selected all the list and I tried to update

After some time, a dialog box popped up showing that none has been selected

And, after some seconds when I closed the first box, another dialog popped up saying, all are updated

But when I checked the Hosts, I find that nothing is updated

fyujj
Posts: 102
Joined: Sat 04 Apr 2009, 17:40

#17 Post by fyujj »

technosaurus, sc0ttman or whoever, could you adapt the script to use yad?
I'm using Debian and wouldn't like to install gtkdialog or xdialog since they're considered deprecated.

User avatar
sinc
Posts: 545
Joined: Wed 22 Jul 2009, 20:35
Location: Tampa, FL USA

#18 Post by sinc »

chrismt wrote:version 0.4 is giving me an error
same error here. v.4 not working right.

User avatar
technosaurus
Posts: 4853
Joined: Mon 19 May 2008, 01:24
Location: Blue Springs, MO
Contact:

#19 Post by technosaurus »

Yad pet is here:
http://www.murga-linux.com/puppy/viewtopic.php?t=58306
for my version all you need to do is have yad output to stdout any combination of ...
1 2 3 4 5

and place it into
`Xdialog ... ...`

zenity/yad break compatibility with the standard dialog formats that dialog, cdialog, xdialog, whiptail and kdialog all basically share in common ... so it would need to be recoded ... but you can always use the dialog version from a terminal (just get rid of the checking for DISPLAY and set DIALOG to dialog)
Check out my [url=https://github.com/technosaurus]github repositories[/url]. I may eventually get around to updating my [url=http://bashismal.blogspot.com]blogspot[/url].

fyujj
Posts: 102
Joined: Sat 04 Apr 2009, 17:40

#20 Post by fyujj »

Thank you technosaurus, that's even better (using just the terminal). I don't know bash (or whatever scripting language). I left it like this:

Code: Select all

#!/bin/sh
DIALOG=dialog || DIALOG=dialog
for x in `$DIALOG --stdout --checklist "Choose your ad blocking service(s)" 0 0 4 1 "mvps.org" ON 2 "systcl.org" ON 3 "technobeta.com" ON 4 "yoyo.org" ON |tr "/" " " |tr '\"' ' '`; do
   case $x in
   1)wget -c -4 -t 0 -O /tmp/adlist1 'http://www.mvps.org/winhelp2002/hosts.txt';;
   2)wget -c -4 -t 0 -O /tmp/adlist2 'http://sysctl.org/cameleon/hosts';;
   3)wget -c -4 -t 0 -O /tmp/adlist3 'http://www.technobeta.com/download/urlfilter.ini';;
   4)wget -c -4 -t 0 -O /tmp/adlist4 'http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext';;
   *)echo $x;;
   esac
done
touch /tmp/adlist{1,2,3,4}
cat /etc/hosts /tmp/adlist{1,2,3,4} |grep ^[1-9] |sed '/^$/d' |sed "s/\t/ /g"|sed '/^#/d' |sed 's/  / /g' |dos2unix -u |sort |uniq > /etc/hosts
rm -f /tmp/adlist{1,2,3,4}
and it showed the dialog fine in the terminal, downloaded and created fine /tmp/adlist1 to /tmp/adlist4 but then just left a blank /etc/hosts and also didn't delete those temporary files.
I have safely backed up my original hosts file so no worry regarding this. I have dos2unix installed too.

Post Reply