Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Sat 17 Nov 2018, 17:18
All times are UTC - 4
 Forum index » Advanced Topics » Puppy Projects
Gethosts script
Moderators: Flash, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 1 of 1 [8 Posts]  
Author Message
jockjunior

Joined: 29 Jan 2008
Posts: 150
Location: Lancashire,U.K.

PostPosted: Wed 13 Apr 2016, 14:58    Post subject:  Gethosts script
Subject description: Script to download and merge hosts files from internet
 

Hi all, downloaded Debiandog and am very impressed with it Saintless, some good work done there.
this script I found on the internet but have amended it to work in DebianDog. Sounds impressive but all it took was to alter the directories, and remove the sudo commands to suit DebianDog. It should work with any Puppy as well . Place the script in /bin and make it executable.

#!/bin/bash

# If this is our first run, save a copy of the system's original hosts file and set to read-only for safety
if [ ! -f /root/hosts-system ]
then
echo "Saving copy of system's original hosts file..."
cp /etc/hosts /root/hosts-system
chmod 444 /root/hosts-system
fi

# Perform work in temporary files
temphosts1=$(mktemp)
temphosts2=$(mktemp)

# Obtain various hosts files and merge into one
echo "Downloading ad-blocking hosts files..."
wget -nv -O - http://winhelp2002.mvps.org/hosts.txt >> $temphosts1
wget -nv -O - http://hosts-file.net/ad_servers.asp >> $temphosts1
wget -nv -O - http://someonewhocares.org/hosts/hosts >> $temphosts1
wget -nv -O - "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext" >> $temphosts1

# Do some work on the file:
# 1. Remove MS-DOS carriage returns
# 2. Delete all lines that don't begin with 127.0.0.1
# 3. Delete any lines containing the word localhost because we'll obtain that from the original hosts file
# 4. Replace 127.0.0.1 with 0.0.0.0 because then we don't have to wait for the resolver to fail
# 5. Scrunch extraneous spaces separating address from name into a single tab
# 6. Delete any comments on lines
# 7. Clean up leftover trailing blanks
# Pass all this through sort with the unique flag to remove duplicates and save the result
echo "Parsing, cleaning, de-duplicating, sorting..."
sed -e 's/\r//' -e '/^127.0.0.1/!d' -e '/localhost/d' -e 's/127.0.0.1/0.0.0.0/' -e 's/ \+/\t/' -e 's/#.*$//' -e 's/[ \t]*$//' < $temphosts1 | sort -u > $temphosts2

# Combine system hosts with adblocks
echo Merging with original system hosts...
echo -e "\n# Ad blocking hosts generated "$(date) | cat /root/hosts-system - $temphosts2 > /root/hosts-block

# Clean up temp files and remind user to copy new file
echo "Cleaning up..."
rm $temphosts1 $temphosts2
echo "Done."
echo
echo "Copy ad-blocking hosts file with this command:"
echo " cp /root/hosts-block /etc/hosts"
echo
echo "You can always restore your original hosts file with this command:"
echo " cp /root/hosts-system /etc/hosts"
echo "so don't delete that file! (It's saved read-only for your protection.)"
echo

hope it's of interest to someone

Phil (old git silver surfer)

edit it puts the files hosts-block and hosts-system in folder /root
Back to top
View user's profile Send private message Visit poster's website 
LazY Puppy


Joined: 21 Nov 2014
Posts: 2007
Location: Germany

PostPosted: Wed 13 Apr 2016, 15:39    Post subject:  

Shouldn't this more related being in the Programming section?
_________________
RSH

"you only wanted to work your Puppies in German", "you are a separatist in that you want Germany to secede from Europe" (musher0) Laughing

No, but I gave my old drum kit away for free to a music store collecting instruments for refugees! Wink
Back to top
View user's profile Send private message 
drunkjedi


Joined: 24 May 2015
Posts: 897

PostPosted: Wed 13 Apr 2016, 22:05    Post subject:  

I believe the built in gui app, pup-advert-blocker use similar script.
I thought Debian dog had it too.
Back to top
View user's profile Send private message 
saintless


Joined: 11 Jun 2011
Posts: 3882
Location: Bulgaria

PostPosted: Thu 14 Apr 2016, 04:34    Post subject: Re: Gethosts script
Subject description: Script to download and merge hosts files from internet
 

Hi jockjunior.

The script attached as debdog-advert-blocker in zip archive with modifications for multiuser (it will save in $HOME/ instead in /root and message how to run the script as user).
I can make deb package in a few days with menu entries for debdog repository but I prefer to have the source mentioned in the script. Can you give link to it?

BTW pup-advert-blocker seems to work from quick testing in DebianDog without modifications after installing dos2unix package.

Edit: The archive removed. Here is download link for the deb package:
http://kazzascorner.com.au/saintless/DebianDog/DebianDog-Jessie/No-repo-packages/debdog-advert-blocker_1.0.0-1_i386.deb

Toni

_________________
Farewell, Nooby, you will be missed...

Last edited by saintless on Fri 15 Apr 2016, 04:22; edited 2 times in total
Back to top
View user's profile Send private message MSN Messenger 
mcewanw

Joined: 16 Aug 2007
Posts: 3200

PostPosted: Thu 14 Apr 2016, 06:08    Post subject:  

Maybe from here, Toni:

https://www.kubuntuforums.net/showthread.php?56419-Script-to-automate-building-an-adblocking-hosts-file

EDIT: Someone's mods to the script, which he used on a router, allowed the creation of a whitelist of sites not wanting blocked (for some people still wanting google-analytics to work I think):

https://www.kubuntuforums.net/showthread.php?64018-Block-adverts-with-an-automatically-generated-hosts-file-on-a-Linux-router

William
Back to top
View user's profile Send private message Visit poster's website 
jockjunior

Joined: 29 Jan 2008
Posts: 150
Location: Lancashire,U.K.

PostPosted: Thu 14 Apr 2016, 16:12    Post subject:  

Hi Toni, here's where i got the original script from

https://www.kubuntuforums.net/showthread.php?56419-Script-to-automate-building-an-adblocking-hosts-file

sorry for delay in replying been away from computer

Phil
Back to top
View user's profile Send private message Visit poster's website 
saintless


Joined: 11 Jun 2011
Posts: 3882
Location: Bulgaria

PostPosted: Fri 15 Apr 2016, 01:40    Post subject:  

jockjunior wrote:
Hi Toni, here's where i got the original script from

https://www.kubuntuforums.net/showthread.php?56419-Script-to-automate-building-an-adblocking-hosts-file

Thanks Phil, I will make deb package in the next few days.
I will post the information in DD thread when I'm ready.

mcewanw wrote:
EDIT: Someone's mods to the script, which he used on a router, allowed the creation of a whitelist of sites not wanting blocked (for some people still wanting google-analytics to work I think):

https://www.kubuntuforums.net/showthread.php?64018-Block-adverts-with-an-automatically-generated-hosts-file-on-a-Linux-router

Thanks William. I think to include only the attached zip archive script in the package. It is simple and doesn't change /etc/hosts automatically + option to restore easy the original file later. No problem for me if someone likes to improve it or add extra scripts in the package later.

Edit: Download link for debdog-advert-blocker_1.0.0-1_i386.deb
The package contains two scripts in /usr/local/bin with menu entry Settings -> DebDog-advert-blocker
One of the scripts is unchanged original version gethosts.
The second script debdog-advert-blocker executes gethosts opening terminal window. Starts gethosts with "sudo -E" for user account or without sudo for root.

Toni
debdog-advert-blocker.png
 Description   
 Filesize   106.99 KB
 Viewed   233 Time(s)

debdog-advert-blocker.png


_________________
Farewell, Nooby, you will be missed...
Back to top
View user's profile Send private message MSN Messenger 
LesStrater

Joined: 15 Jun 2016
Posts: 2

PostPosted: Sat 18 Jun 2016, 09:34    Post subject: Re: Gethosts script
Subject description: Script to download and merge hosts files from internet
 

This script is exactly what I've been looking for, with minor modification. I've used the MVPS hosts file for years to block ad content, and this script will automate the process of downloading and installing updated versions of it every month. The MVPS hosts file (IMHO) is the best one out there.

I did find one major bug in the script: The script states that it will "Delete all lines that don't begin with 127.0.0.1 -- and it correctly does that. The problem is that all the server addresses in the MVPS hosts file begin with with local host "0.0.0.0", not "127.0.0.1". This means that the script downloads the MVPS list and then promptly deletes all the entries in it.
I tried for hours to modify that convoluted SED command to also include "0.0.0.0" from its exclusion parameter, but I'm not a strong enough programmer, and none of the examples I found online worked properly. In the end, I wound up using two SED commands, one to delete each local machine address, then I appended the two lists into one.

Additionally, I deleted the downloading of the HUGE hosts file from "hosts-file.net". Whoever maintains this list has some overzealous political/moral motivation behind their efforts. I found several video-streaming servers which I regularly use being blocked by this list. I'm not talking porn, but regular movie and TV show sites like Primewire. (I did also notice MANY porn servers on the list--for those of you who care.) I'm interested in ad-blocking, not content-blocking...

Thanks to jockjunior (Phil) for making this script available!
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 1 of 1 [8 Posts]  
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Puppy Projects
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0670s ][ Queries: 12 (0.0137s) ][ GZIP on ]