Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Fri 15 Dec 2017, 10:24
All times are UTC - 4
 Forum index » Advanced Topics » Puppy Derivatives
Banksy 3 - aka "The Personator"
Moderators: Flash, JohnMurga
Post new topic   Reply to topic View previous topic :: View next topic
Page 4 of 7 [94 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6, 7 Next
Author Message
greengeek


Joined: 20 Jul 2010
Posts: 4745
Location: Republic of Novo Zelande

PostPosted: Wed 06 Apr 2016, 14:32    Post subject:  

Sylvander wrote:

It is somewhat slow, but I figured that was because it must load the files from a slow optical disk/drive.
<Yes, optical media loading is slow. If it was loading over a gig of data you would really notice it even more. I was thinking that a cdrw may be more like a DVD in terms of larger data capacity but you have confirmed that the actual content on the cdrw is well less than normal CD capacity anyway.

This suggests that the missing 1.5GB reported by partview is either erroneous or caused by something like the file expansion process after loading CD data, or maybe caused by some artifact of the unusual way banksy is layered. Maybe both. Either way it looks like having a swap partition is necessary during a personalisation process that involves significant addition of data/programs. Less critical once running from the completed personal CD - but I still need to look closer at personal storage calculation and usage.
.
Quote:
366MB is much smaller than the 8.5GB - 7.1 GB = 1.4 GB of free storage that disappears during the making of the ISO.
There are good reasons why a lot of storage is consumed during that process - the /tmp/banksydistillor directory holds all of the added data ready to build into the new personal sfs, then it is formed into the new sfs which is also in /tmp, then that sfs is copied into an "isobuild" directory along with other necessary files (all of this also in /tmp) then that isobuild directory is compiled into the new iso (yet another large file sitting in /tmp). So you can see a heap of data accumulates in /tmp during the process.

I think I can smarten the process somewhat to make it more thrifty on RAM. More testing to follow. Till then I suggest the USB swap partition trick may be the key.

Also - installing the banksy files to HDD definitely speeds booting (although I know you are not comfortable with that structure...)
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 4299
Location: West Lothian, Scotland, UK

PostPosted: Wed 06 Apr 2016, 15:58    Post subject:  

greengeek wrote:
installing the banksy files to HDD definitely speeds booting (although I know you are not comfortable with that structure...)

I like:
a. Banksy running totally in RAM with no use of any writable drive.
b. The fact that the Banksy CD-RW is read-only.

I don't like:
c. Any writable storage media being used during normal use of Banksy.
e.g. Swap partition. [Might a swap partition be infected, and so infect a subsequent Banksy session?]
Might a full install of Banksy [to an internal HDD partition] be infected?
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 4745
Location: Republic of Novo Zelande

PostPosted: Wed 06 Apr 2016, 18:48    Post subject:  

Sylvander wrote:
I like:
a. Banksy running totally in RAM with no use of any writable drive.
b. The fact that the Banksy CD-RW is read-only.
I agree.

Quote:
I don't like:
c. Any writable storage media being used during normal use of Banksy.
e.g. Swap partition.
Also agree

Quote:
[Might a swap partition be infected, and so infect a subsequent Banksy session?]
I have read that this is impossible because any code in the swap area becomes useless after shutting down a session. However, swap is writable so I'd have to say that my guess is that there is more risk with it than without it. (Although we have to bear in mind that RAM itself is writable so in that case maybe there is not much difference - unless there is some attack vector that can "collect" tainted data from the previous swap session. If that is possible then I would venture to suggest such complex hacker skills would most likely reside in the hands of nation states such as was revealed by the stuxnet virus - in which case I have no show of protecting my system against that level of penetration)

Having said that I would be happy to take the risk of using the usb swap just for the time it takes to get the personalisation/burn done if the amount of data being added required the extra space during the processing. (I hope to reduce the required space in the next impersonator script).

Quote:
Might a full install of Banksy [to an internal HDD partition] be infected?
I consider it highly unlikely because the two sfs in use are both readonly (in contrast with normal puppies where the personal storage file or folder is writable). In order to corrupt/infect the banksy code it would be necessary to unsquash, modify, resquash, then overwrite one of the sfs files. I can't imagine that being very likely except by local hands-on operation.

I would rate the use of Wine as a greater risk than use of swap or HDD install - especially if permitting the mono and gecko downloads. I don't have any grounds for saying that - it's just a gut feeling. Do the mono and gecko downloads ask you any questions or explain what they are for? Not as far as I could tell. That in itself would concern me. I prefer to carry the most minimalistic payload possible to get the job done - less code, less exposure. That's my mantra anyway.

Your testing has brought to my attention that my system always has 10GB of swap partition available - I never thought about that being a security risk till now. I may not sleep well tonight Smile
.
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 4299
Location: West Lothian, Scotland, UK

PostPosted: Wed 06 Apr 2016, 21:29    Post subject:  

1. "I have read that this is impossible because any code in the swap area becomes useless after shutting down a session."
I'd be happy if I knew that any swap partition in use was zero-filled as part of Banksy shut down, but I guess that's impractical.
My lack of knowledge leads me to imagine stuff remaining in the swap partition [left there by (Banksy or) some other Puppy] being brought to life during the latest Banksy session.

2. "we have to bear in mind that RAM itself is writable so in that case maybe there is not much difference"
But surely there IS a difference.
I believe the contents of SWAP is persistent, whereas the content of RAM is completely lost after a relatively short time [not persistent].

3. "I would be happy to take the risk of using the usb swap just for the time it takes to get the personalisation/burn done"
same here.

4. "I hope to reduce the required space in the next impersonator script"
I like it.

5. " In order to corrupt/infect the banksy code it would be necessary to unsquash, modify, resquash, then overwrite one of the sfs files. I can't imagine that being very likely except by local hands-on operation."
Nice to hear.

6. "I would rate the use of Wine as a greater risk than use of swap or HDD install"
Yes, I'm not happy that I use WINE, but I hate the prospect of attempting to [transfer the vault contents, and] switch to using a Puppy alternative that is also portable [and works really well] like "Acerose Password Vault" [APV].

7. "Do the mono and gecko downloads ask you any questions or explain what they are for? Not as far as I could tell."
Same here.
MONO
GECKO

8. "Your testing has brought to my attention that my system always has 10GB of swap partition available - I never thought about that being a security risk till now."
I hope I'm not raising false fears; it's just something that troubles me.
Hence, I'll avoid using a swap partitition until I come to know it poses no risk.
Back to top
View user's profile Send private message 
version2013

Joined: 08 Sep 2013
Posts: 310
Location: Florida, USA

PostPosted: Wed 06 Apr 2016, 23:21    Post subject:  

Sylvander wrote:

7. "Do the mono and gecko downloads ask you any questions or explain what they are for? Not as far as I could tell."
Same here.
MONO
GECKO

More links:
Mono
Gecko
Back to top
View user's profile Send private message Visit poster's website 
greengeek


Joined: 20 Jul 2010
Posts: 4745
Location: Republic of Novo Zelande

PostPosted: Thu 07 Apr 2016, 03:37    Post subject:  

Hi Sylvander and version2013 - thanks for the mono and gecko links. I'm uncomfortable about the mono being an interface for .NET although I can see that some Wine programs will need it. Gecko worries me less, but I think I would only load this if i knew it was definitely required for a specific program.

Sylvander wrote:
My lack of knowledge leads me to imagine stuff remaining in the swap partition [left there by (Banksy or) some other Puppy] being brought to life during the latest Banksy session.
I can't rule that out unfortunately. Who knows what hackers are capable of. All I can say is I am unaware of any hijack problems (except for one browser hijack on a dodgy site) during the time I have been using banksy (but hey, who would know if any of my data had been spied on??)
[The browser hijack told me my HDD had been encrypted and that I needed to pay a ransom. The browser and system was fully locked. However after a forced powerdown all was perfect]

There are three issues I am looking at to improve the performance of banksy during the personator process:

1) Add it to my longterm list to graft the PAE kernel version of Slacko 5.6 into banksy to allow addressing of more memory

2) Try variations on the "swappiness" setting as mentioned by partsman here. This won't be of any help to those who absolutely don't want to use swap, but may improve memory usage significantly and will probably allow significantly greater amounts of data to be added in the banksy personalisation process before swap becomes a requirement. I just gave it a quick try and found my RAM usage went up from 192MB (out of 2GB) to 1.5GB. It's a bit scary that my system was only using 192MB of my available RAM. (Mind you I don't know if I can trust my conky figures yet - I need to find good quality metrics to evaluate this stuff).
@Sylvander - I am interested in the bargraph figures shown by htop during any future personalisation attempts that you make. I have always wondered why my htop RAM usage figures were so low - I convinced myself that the code must be very efficient but now I realise I had huge swap all the time and RAM usage was affected by a low "swappiness" figure that seems to be inherent in Slacko. More testing required.

3) Make ongoing improvements to the impersonator script and the way it accumulates and handles data in /tmp. I seem to recall that sfs can be squashed on the fly somehow - maybe that would permit more efficient handling of personalisation data.
.

Last edited by greengeek on Sun 17 Apr 2016, 13:42; edited 2 times in total
Back to top
View user's profile Send private message 
ASD

Joined: 22 Mar 2016
Posts: 99

PostPosted: Thu 07 Apr 2016, 05:01    Post subject:  

In banksy 3 (as with many Pups) the first directory listed is "archive" which has a unique property.

Although the note inside the archive directory does not seem directly relevant to Banksy 3, it might be worth a try:

e.g. moving a large Wine pet (and/or other pet/sfs test files) to the archive directory and then clicking the "impersonator" button. Then watching for interesting effects (if any) during impersonation and upon a reboot.

A few days ago I tried Banksy 3 and all went well until I clicked the impersonator button and the process failed. Whilst Banksy 3 seems to have some nice features, it's not for me.
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 4745
Location: Republic of Novo Zelande

PostPosted: Thu 07 Apr 2016, 21:22    Post subject:  

ASD wrote:
In banksy 3 (as with many Pups) the first directory listed is "archive" which has a unique property.

Although the note inside the archive directory does not seem directly relevant to Banksy 3, it might be worth a try:
Thanks ASD - I had not actually read the text file in /archive until you mentioned it but it seems relevant to a multisession CD/DVD whch i have hardly ever used, and which I think I have fully disabled in banksy.

Quote:
A few days ago I tried Banksy 3 and all went well until I clicked the impersonator button and the process failed.
I probably should have made some cautionary notes on the relevant posts to the effect that banksy "impersonator" version is an experimental effort mostly for Sylvanders testing. The normal banksy 3 "personator" available in the first post of the thread requires significantly less resources than "impersonator" as it doesn't try to grab as much data.

Having said that I do still need to improve that original version to make it more efficient and I realise it is not to everyones tastes. Thanks for trying it and for your comments about /archive.
cheers!
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 4299
Location: West Lothian, Scotland, UK

PostPosted: Sat 09 Apr 2016, 06:27    Post subject:  

greengeek wrote:
@Sylvander - I am interested in the bargraph figures shown by htop during any future personalisation attempts that you make. I have always wondered why my htop RAM usage figures were so low

Erm...
I'm in the middle of a 3rd attempt at personalizing and using "impersonator", and I've just realized I don't really know exactly what you mean by the above. [What are the "bargraph figures"?]
Right now [some personalizations completed, but "impersonator" not yet running] htop shows mem = 118/3019MB.
2 CPU cores: jumping around between 4% & 14%
swp: 0/0MB [connected a Flash Drive, and made a swap partition during the session, is it not being used?]

Guess I'll abort this session, and wait 'till you explain in detail what you need.
Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 4745
Location: Republic of Novo Zelande

PostPosted: Sun 10 Apr 2016, 18:46    Post subject:  

Sylvander wrote:
I've just realized I don't really know exactly what you mean by the above. [What are the "bargraph figures"?]
There are a number of ways to identify how much of the available RAM is being used and I was referring to the top left hand corner of the Htop screen (Menu, System, Htop system process viewer) as I thought that might be a good way to view how close the system was to consuming all of it's available RAM (and therefore at risk of needing "swap" or else running out of memory). Pic attached.

Quote:
htop shows mem = 118/3019MB
Yes, you're looking at the graphs I was meaning. I am surprised that it only shows 118MB in use - given that the base banksy sfs is over 200MB and your personal sfs will add extra MB over that..

Quote:
swp: 0/0MB [connected a Flash Drive, and made a swap partition during the session, is it not being used?]
Swap is only available if it is discovered during booting (or if you manually control it with the swapon or swapoff command)

Just for comparison purposes I will attach below an Htop bargraph which suggests I am using 276MB out of 1984MB available RAM. This is weird as this particular session is using a HUGE personalised banksy (I added HEAPS of extra browsers, word processors etc etc) and I have opened multiple windows to run many programmes at once. I really expected it to show a much higher figure. Maybe the Htop bargraphs are not to be trusted.

By contrast I have also included a screenshot of the "free" command which offers what I would consider a more accurate view of my RAM usage. That pic suggests I am using about 1.6GB of RAM which is more in line with my expectations given the amount of stuff I have loaded and running.

You mention 3 attempts at using the impersonator - is it doing the business so far or is it giving you trouble?

I think I would make two caveats with the current impersonator version:

1) If you don't have a swap partition you will need to avoid adding excessive personalisations as the total RAM required by the impersonator is roughly 5 times the final size of the iso (eg if banksy is 220MB and you add 200MB personalisations you will need at least 420 x 5 = 2.1GB RAM for the process). I am currently working on improving this.

2) If you are able to add a swap partition for the duration of the impersonator process it will give you much greater ability to add personalisations. I run a 10GB HDD swap partition. I reckon a 4GB or 8GB usb stick would be a suitable alternative although it pays to remember that usb port speed and usb stick write speeds are significantly less than HDD drives.
.
htop_bargraphs.jpg
 Description   
 Filesize   12.15 KB
 Viewed   550 Time(s)

htop_bargraphs.jpg

free_command.jpg
 Description   
 Filesize   11.38 KB
 Viewed   545 Time(s)

free_command.jpg

Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 4299
Location: West Lothian, Scotland, UK

PostPosted: Mon 11 Apr 2016, 11:45    Post subject:  

1. Repeat of attempt-3.
Completed all of my customizations, then took the screenshots displayed below.
I'll [close all open windows, then] complete using "impersonator" to make the ISO and burn the CD-RW, then reboot using the new CD and report back.

2. The new Banksy3 CD-RW disk booted just fine, and works as it aught it seems.
00.png
 Description   
 Filesize   3.62 KB
 Viewed   521 Time(s)

00.png

01.png
 Description   
 Filesize   6.53 KB
 Viewed   525 Time(s)

01.png

Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 4745
Location: Republic of Novo Zelande

PostPosted: Tue 12 Apr 2016, 03:47    Post subject:  

Sylvander wrote:
The new Banksy3 CD-RW disk booted just fine, and works as it aught it seems.
Hey thanks for all your testing and feedback - much appreciated. I'm glad you've been getting a useable outcome - that mirrors my own trials in the sense that impersonator works really well as long as there is adequate working memory.

If you are keen for more testing I have made a newer version of b3impersonator which is more efficient on RAM usage. I think it might now be possible for your original trial (which included the mono and gecko downloads but did not have a swap partition) to complete successfully within the 3019MB RAM that banksy addresses with its nonPAE kernel.
(If you are sick of testing thats fine Smile )

Downloads here:

EDIT 2 : please don't use this version. Better version available two posts further down
EDIT : This version works perfectly burning to CD but I made a booboo with the DVD burning script. New version to be uploaded soon. *****
http://www.mediafire.com/download/bf8in11j1y8n4m4/b3impgeneric_RC2.iso
http://www.mediafire.com/download/1275u23sx6dhaey/b3impgeneric_RC2.iso-md5.txt
EDIT : This version works perfectly burning to CD but I made a booboo with the DVD burning script. New version to be uploaded soon. *****
EDIT 2 : please don't use this version. Better version available two posts further down
.

Last edited by greengeek on Sun 17 Apr 2016, 13:48; edited 3 times in total
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 4299
Location: West Lothian, Scotland, UK

PostPosted: Tue 12 Apr 2016, 05:22    Post subject:  

1. Attempt number-4 in progress.
a. I've burned the CD-RW of the latest rc2.
No swap partition was used.
Personal storage = 1.5G
Free = 842M [lowest value]
That completed OK.
About to reboot, and will report back.

b. Booted OK, and working as it aught.
Easy and simple to do.
"Just works". Very Happy
SUPERB!

c. Won't be able go give further replies until Sat 16th April 2016.
00.png
 Description   
 Filesize   3.59 KB
 Viewed   491 Time(s)

00.png

01.png
 Description   
 Filesize   4.62 KB
 Viewed   495 Time(s)

01.png

Back to top
View user's profile Send private message 
greengeek


Joined: 20 Jul 2010
Posts: 4745
Location: Republic of Novo Zelande

PostPosted: Sat 16 Apr 2016, 05:55    Post subject:  

I have uploaded a new version of the banksy3 impersonator which has the DVD burn script corrected and has a couple of conveniences added:

- a taskbar shortcut for the /root folder (makes it quick and easy to get access to the filesystem)

- a taskbar shortcut for xvkbd virtual keyboard which can be a more secure way to enter data rather than using keyboard keystrokes.

- ability to use shift+ctrl+c to copy data from the terminal
- ability to use shift+ctrl+v to paste data into the terminal

Download links:

http://www.mediafire.com/download/mfd68pmvjst5wh7/b3impgeneric_RC5.iso
http://www.mediafire.com/download/4uuo8k98jbs4l5h/b3impgeneric_RC5.iso-md5.txt

Note for new users :
- Impersonator is still in testing. The normal functionality all works well but the actual "impersonator" script that collects ALL of your personalisations requires quite a lot of RAM to manipulate the data into the new iso. You may also (temporarily) need to have a swap partition available (2GB - 10GB depending how much data you want to add). My laptop has 2GB of RAM and a 10GB swap partition on HDD. The swap partition only starts being used when I add quite a lot of extra data or sizeable programs into the running system prior to running the impersonator script.

- The "impersonator" script has been designed to do a "one time" grab of the personalisations - so it is necessary to boot the generic iso then add all desired changes and customisations in one session, then run the impersonator script. (It has not been tested for multiple "progressive" changes such as a savefile is intended for)

- Many PCs that are now in the "older" bracket would have 2GB or more RAM so that is really the target market for banksy. I also run it on a 1GB netbook but that required me to add a usb swap partition temporarily and it slowed the personalisation process dramatically. Still worked though.

Last edited by greengeek on Sun 17 Apr 2016, 14:35; edited 3 times in total
Back to top
View user's profile Send private message 
Sylvander

Joined: 15 Dec 2008
Posts: 4299
Location: West Lothian, Scotland, UK

PostPosted: Sat 16 Apr 2016, 11:59    Post subject:  

Tried the usual with RC5, and everything worked just fine.
No swap partition was used.
Personal storage = 1.5G
Free = 841M [lowest value]

After burning the CD-RW...
All the windows closed automatically when I clicked the button to acknowledge completion. NICE! Very Happy

Booted OK, and working as it aught.
Back to top
View user's profile Send private message 
Display posts from previous:   Sort by:   
Page 4 of 7 [94 Posts]   Goto page: Previous 1, 2, 3, 4, 5, 6, 7 Next
Post new topic   Reply to topic View previous topic :: View next topic
 Forum index » Advanced Topics » Puppy Derivatives
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.0768s ][ Queries: 14 (0.0067s) ][ GZIP on ]