Puppy Linux Discussion Forum Forum Index Puppy Linux Discussion Forum
Puppy HOME page : puppylinux.com
"THE" alternative forum : puppylinux.info
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

The time now is Sun 21 Dec 2014, 16:19
All times are UTC - 4
 Forum index » House Training » HOWTO ( Solutions )
How to do RAID0 with USB thumb drives using lvm
Moderators: Flash, Ian, JohnMurga
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
Page 1 of 1 Posts_count  
Author Message
arrgk

Joined: 11 Sep 2008
Posts: 1

PostPosted: Thu 11 Sep 2008, 15:57    Post_subject:  How to do RAID0 with USB thumb drives using lvm
Sub_title: Slackware 12.0 packages work on eee
 

I have a eee running Pupeee off an 8g SDHC. I also have a pair of 16G USB thumb drives. So we want to tie the USB drives together and make them go fast. I chose raid0. Reliability will not be an issue. I do not need raid1 or any other mirror. I chose to use the raid built in to Linux lvm, because you can mix raid0, raid1, and ordinary lvm partitions all on the same virtual drive. You can also resize them very easily. In other words, administration is easy with lvm.

first what did not work ::
dmsetup :: almost useless, --notable does not work
very difficult to master, forget it

ndadm :: not worth the effort

Here is a breakdown of what I did.

/ install gslapt ( great stuff )
/ install Slackware 12.0 lvm and devmapper
/ modprobe dm-mod
/ pvcreate /dev/sdb1
/ pvcreate /dev/sdc1
/ vgcreate Egrp00 /dev/sdb1 /dev/sdc1
/ lvcreate -L 4567m -i 2 -I 64 Egrp00 -n Eraid00
/ mkreiserfs /dev/Egrp00/Eraid00

And you are ready to rock.

Speed read test ::
dd if=/dev/sdb of=/dev/null bs=512 count=40960
gave 10.5 MBs
dd if=/dev/Egrp00/Eraid00 bs=512 count=40960
gave 19.6 MBs

93.333% of double the speed of one drive is what I count as success.

I use a journaled fs on flash !! ? yep. With normal heavy usage, this will last for years, and replacing any failing stick is easy, as is adding to the array. The BIG advantage of lvm raid is that you can add drives which have different sizes. So, for me lvm is a clear winner.

The various commands like vgs, vgscan, ... lvs, ... are at your fingertips.
Read the lvm howto, and make this turn on at boot. You get one very fat pupeee.

Thanks for the gslapt package and to the pupeee team
Back to top
View user's profile Send_private_message 
tester321

Joined: 26 Apr 2008
Posts: 10

PostPosted: Sun 14 Dec 2008, 19:56    Post_subject: Re: lvm and RAID with USB thumb drives
Sub_title: Slackware 12.0 packages work on eee
 

arrgk wrote:
...
Here is a breakdown of what I did.

/ install gslapt ( great stuff )
/ install Slackware 12.0 lvm and devmapper


arrgk: Can you explain how to install the Slackware lvm onto Puppy? I have searched but haven't found it and I am not sure how to do it.

Thanks in advance for any info.
Back to top
View user's profile Send_private_message 
rrolsbe

Joined: 15 Nov 2006
Posts: 183

PostPosted: Sat 04 Apr 2009, 10:40    Post_subject: Re: lvm and RAID with USB thumb drives [SOLVED]
Sub_title: Slackware 12.0 packages work on eee
 

tester321 wrote:
arrgk wrote:
...
Here is a breakdown of what I did.

/ install gslapt ( great stuff )
/ install Slackware 12.0 lvm and devmapper


arrgk: Can you explain how to install the Slackware lvm onto Puppy? I have searched but haven't found it and I am not sure how to do it.

Thanks in advance for any info.


Since no one replied two your request, hopefully the following will help.

My plan is to purchase four fairly fast USB flash sticks and install them into a four port Targus USB hub. Then create a poor mans SSD device for improved speed. The purpose of the remainder of this post is to verify that it SHOULD work before I purchase the USB flash devices.


I used Puppy 4.11 for the following on my ASUS EEE 904HA

Code:
# uname -a
Linux puppypc 2.6.25.16 #1 Tue Aug 26 10:45:53 GMT-8 2008 i686 GNU/Linux

Get these two slackware packages from the following web site http://packages.slackware.it/

device-mapper-1.02.19-i486-2.tgz
lvm2-2.02.09-i486-1.tgz


I choose the slackware version 12.0 maybe you could use 12.1 or 12.2.
I think you use 12.0 for Puppy Linux 4.11 and
use 12.2 if you are using Puppy Linux 4.2. Comments Welcome

Convert them to pet packages with the commands below:
Code:
tgz2pet device-mapper-1.02.19-i486-2.tgz
tgz2pet lvm2-2.02.09-i486-1.tgz

Install the two newly created pet packages via the file browser. Install the device-mapper... 1st then the lvm2... pet.

Load the device mapper module
Code:
modprobe dm-mod


Verify it loaded
Code:
# lsmod | grep mod
dm_mod                 56160  3


Add modprobe to your /etc/rc.d/rc.local file so the device mapper is loaded at boot time.

Code:
# cat /etc/rc.d/rc.local
#this file called from rc.sysinit
#you can edit this file
#When firewall is installed, will append lines to this file...
modprobe pciehp pciehp_force=1
modprobe asus_acpi
modprobe dm-mod
acpid

if [ -x /etc/rc.d/rc.firewall ]; then
  /etc/rc.d/rc.firewall start
 
# diff rc.local rc.local_orig
6d5
< modprobe dm-mod


Do the LVM stuff as below:

Code:
# pwd
/etc/lvm
# diff lvm.conf lvm.conf_orig
76,78c76
<     # sysfs_scan = 1  orig
<     # Line below changed by rro so the pvcreate commands below would not fail
<     sysfs_scan = 0
---
>     sysfs_scan = 1   


pvcreate /dev/sdc1      #8Gig Kingston Flash
pvcreate /dev/sdd1      #2Gig Kingston Flash
vgcreate flash_lvm /dev/sdc1 /dev/sdd1
lvcreate -L 2000m -i 2 -I 64 flash_lvm -n flash_raid00

I have not totally figured out exactly what the options do in the lvcreate command above, comments welcome.

Format the newly created stripped device, I choose ext2:
Code:
 mke2fs /dev/flash_lvm/flash_raid00


Mount the LVM RAID0 Volume with the two USB flash devices stripped for improved speed (Hopefully anyway).

Code:
mount /dev/flash_lvm/flash_raid00 /mnt/test

# mount
/dev/flash_lvm/flash_raid00 on /mnt/test type ext2 (rw,errors=continue)

# df  -kh
/dev/flash_lvm/flash_raid00
                      2.0G   29M  1.8G   2% /mnt/test




I have not rebooted or added the above mount command into my fstab yet.

I probably missed a step or two, but hopefully this will will save fellow forum members some time. PLEASE feel free to make comments in this tread.

Regards
Ron

UPDATE:

When I rebooted the BIOS reassigned my LVM devices from /dev/sdc1& /dev/sdd1 to /dev/sdb1 & /dev/sdd1. After going through the entire procedure, it worked again; however, after rebooting I got an error when trying to mount the LVM volumes. See below

I found an article on the web that stated-- if you made any changes to the /etc/lvm/lvm.conf file you needed to make a change to the initrd.gz file. Maybe this has something to do with not being able to mount the LVM devices after a reboot. Anyone have any idea what might need to be changed in the initrd.gz file? I inflated the initrd.gz file and it is a cpio archive.

This web article http://www.bigbruin.com/reviews05/thumbraid_6 indicated a speed up using four devices of 2X on reads and 6X on writes; however, He was using a different USB hub and different USB flash devices. I have read other articles where the read speeds using four devices was better that 2x. See dd commands below

The USB hub I used for these tests is a four port Targus model PA055 and the USB flash sticks were both Kingston DataTraveler one an 8G and the other a 2G.

Code:
# mount /dev/flash_b_d_lvm/flash_b_d__raid00 /mnt/test
mount: /dev/flash_b_d_lvm/flash_b_d__raid00 is [b]not a valid block device
[/b]
# ls -al /dev/flash_b_d_lvm/flash_b_d__raid00         
lrwxrwxrwx 1 root root 43 2009-04-04 12:15 /dev/flash_b_d_lvm/flash_b_d__raid00 -> /dev/mapper/flash_b_d_lvm-flash_b_d__raid00

# ls -al /dev/mapper/flash_b_d_lvm-flash_b_d__raid00
brw-rw---- 1 root root 253, 0 2009-04-04 11:57 /dev/mapper/flash_b_d_lvm-flash_b_d__raid00

# lsmod | grep mod
dm_mod                 56160  0

When I did the following dd speed tests the LVM raid was slower on reads??? Did not test write speeds.

Code:
dd if=/dev/sdb1 of=/dev/null bs=512 count=4000
 dd if=/dev/sdd1 of=/dev/null bs=512 count=4000

 dd if=/dev/flash_b_d_lvm/flash_b_d__raid00 of=/dev/null bs=512 count=4000

# vgdisplay
  --- Volume group ---
  VG Name               flash_b_d_lvm
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               9.37 GB
  PE Size               4.00 MB
  Total PE              2398
  Alloc PE / Size       500 / 1.95 GB
  Free  PE / Size       1898 / 7.41 GB
  VG UUID               wgYgfX-q1CT-zReN-fExh-xyS3-Zfw1-Pn2KhZ
   
#
# lvdisplay
  --- Logical volume ---
  LV Name                /dev/flash_b_d_lvm/flash_b_d__raid00
  VG Name                flash_b_d_lvm
  LV UUID                bgc4pv-IieA-FHs0-wsRk-1K8D-YDMV-UGUp2e
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                1.95 GB
  Current LE             500
  Segments               1
  Allocation             inherit
  Read ahead sectors     0


I was feeling good until I rebooted and tested the read speed..

Edited_times_total
Back to top
View user's profile Send_private_message 
Dougal


Joined: 19 Oct 2005
Posts: 2505
Location: Hell more grotesque than any medieval woodcut

PostPosted: Sun 05 Apr 2009, 06:35    Post_subject: Re: lvm and RAID with USB thumb drives [SOLVED]
Sub_title: Slackware 12.0 packages work on eee
 

rrolsbe wrote:
UPDATE:

When I rebooted the BIOS reassigned my LVM devices from /dev/sdc1& /dev/sdd1 to /dev/sdb1 & /dev/sdd1.

To make sure you have the same device names every boot, you might want to use the uuid of the drives, rather than /dev/sd[a-z].

Disktype should give you the info. For example:
Code:
bash-3.00# disktype /dev/sdb

--- /dev/sdb
Block device, size 1.906 GiB (2046557696 bytes)
DOS/MBR partition map
Partition 1: 1.003 GiB (1077479424 bytes, 2104452 sectors from 63, bootable)
  Type 0x83 (Linux)
  Ext3 file system
    UUID BDBE2A9B-F9A6-4EB1-A3D4-9FF0C99FB709 (DCE, v4)
    Volume size 1.003 GiB (1077477376 bytes, 263056 blocks of 4 KiB)
Partition 2: 917.8 MiB (962357760 bytes, 1879605 sectors from 2104515)
  Type 0x83 (Linux)
  Ext3 file system
    UUID 25037EF3-A8C8-4689-AF9B-6C20508FA860 (DCE, v4)
    Volume size 917.8 MiB (962355200 bytes, 234950 blocks of 4 KiB)


Now, you might be able to use the lvm commands directly with the uuid (you can put the uuid -- or even the disk label -- instead of the device in fstab), but if not, you can always use udev to give you the uuid device nodes.
I'm including the udev rules file that handles all persistent storage node creation (as Puppy doesn't use this...). You just need to put it in /etc/udev/rules.d and hopefully it won't cause any trouble.

Anyway, here's the kind of thing you will get:
Code:
bash-3.00# ls -1 /dev/disk/by-uuid/                                             07e2703e-0118-47fb-84f9-98570b9f41bf
25037ef3-a8c8-4689-af9b-6c20508fa860
bdbe2a9b-f9a6-4eb1-a3d4-9ff0c99fb709

Those are all links to the device nodes, but you can nonetheless mount them (and so I assume use them with lvm):
Code:
bash-3.00# mount "/dev/disk/by-uuid/25037ef3-a8c8-4689-af9b-6c20508fa860" /mnt/data/
bash-3.00# ls /mnt/data/
archive  dev           etc  lost+found  proc          root  tmp  var
bin      devx_214.sfs  lib  mnt         pup_214R.sfs  sbin  usr  zdrv_214R.sfs


Oh, and you should use the "[code]" formatting to make your post more readable.
60-persistent-storage.rules.gz
Description 
gz

 Download 
Filename  60-persistent-storage.rules.gz 
Filesize  1.36 KB 
Downloaded  542 Time(s) 

_________________
What's the ugliest part of your body?
Some say your nose
Some say your toes
But I think it's your mind
Back to top
View user's profile Send_private_message 
rrolsbe

Joined: 15 Nov 2006
Posts: 183

PostPosted: Sun 05 Apr 2009, 13:50    Post_subject: Re: lvm and RAID with USB thumb drives [SOLVED] Finally
Sub_title: Slackware 12.0 packages work on eee
 

Dougal wrote:
rrolsbe wrote:
UPDATE:

When I rebooted the BIOS reassigned my LVM devices from /dev/sdc1& /dev/sdd1 to /dev/sdb1 & /dev/sdd1.

To make sure you have the same device names every boot, you might want to use the uuid of the drives, rather than /dev/sd[a-z].

Disktype should give you the info. For example:
Code:
bash-3.00# disktype /dev/sdb

--- /dev/sdb
Block device, size 1.906 GiB (2046557696 bytes)
DOS/MBR partition map
Partition 1: 1.003 GiB (1077479424 bytes, 2104452 sectors from 63, bootable)
  Type 0x83 (Linux)
  Ext3 file system
    UUID BDBE2A9B-F9A6-4EB1-A3D4-9FF0C99FB709 (DCE, v4)
    Volume size 1.003 GiB (1077477376 bytes, 263056 blocks of 4 KiB)
Partition 2: 917.8 MiB (962357760 bytes, 1879605 sectors from 2104515)
  Type 0x83 (Linux)
  Ext3 file system
    UUID 25037EF3-A8C8-4689-AF9B-6C20508FA860 (DCE, v4)
    Volume size 917.8 MiB (962355200 bytes, 234950 blocks of 4 KiB)


Now, you might be able to use the lvm commands directly with the uuid (you can put the uuid -- or even the disk label -- instead of the device in fstab), but if not, you can always use udev to give you the uuid device nodes.
I'm including the udev rules file that handles all persistent storage node creation (as Puppy doesn't use this...). You just need to put it in /etc/udev/rules.d and hopefully it won't cause any trouble.

Anyway, here's the kind of thing you will get:
Code:
bash-3.00# ls -1 /dev/disk/by-uuid/                                             07e2703e-0118-47fb-84f9-98570b9f41bf
25037ef3-a8c8-4689-af9b-6c20508fa860
bdbe2a9b-f9a6-4eb1-a3d4-9ff0c99fb709

Those are all links to the device nodes, but you can nonetheless mount them (and so I assume use them with lvm):
Code:
bash-3.00# mount "/dev/disk/by-uuid/25037ef3-a8c8-4689-af9b-6c20508fa860" /mnt/data/
bash-3.00# ls /mnt/data/
archive  dev           etc  lost+found  proc          root  tmp  var
bin      devx_214.sfs  lib  mnt         pup_214R.sfs  sbin  usr  zdrv_214R.sfs


Oh, and you should use the "[code]" formatting to make your post more readable.


Dougal

Thanks for the post

The pvcreate commands below appear to write a new random UUID ID's to both flash devices. I think writing this UUID should (is intended to) solve the problem of being scanned in different later. I believe the mount command is NOT referencing by the physical device names? If true, I still do not know why the LVM devices can't be mounted using the same mount command after a reboot.

This go round, I used the same two flash devices plugged directly into two of my EEE USB ports. The speed WAS faster using the RAIDO, see dd results later in this post. So I guess my TARGUS USB 2.0 hub is the bottleneck?

[code]pvcreate -Zn /dev/sdb1 //The -Zn does not rewrite the 1st 2048 bytes
pvcreate -Zn /dev/sdd1 //The 1st 2048 bytes contain the partition name
//Linux LVM below in the fdisk -l output[/code]

[code]# fdisk -l
Disk /dev/sdb: 2046 MB, 2046033920 bytes
63 heads, 62 sectors/track, 1023 cylinders
Units = cylinders of 3906 * 512 = 1999872 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 1023 1997888 8e Linux LVM

Disk /dev/sdd: 8023 MB, 8023703552 bytes
247 heads, 62 sectors/track, 1023 cylinders
Units = cylinders of 15314 * 512 = 7840768 bytes

Device Boot Start End Blocks Id System
/dev/sdd1 1 1023 7833080 8e Linux LVM
[/code]
[code]vgcreate flash_b_d_lvm /dev/sdb1 /dev/sdd1
lvcreate -L 2000m -i 2 -I 64 flash_b_d_lvm -n flash_b_d_raid00

mke2fs /dev/flash_b_d_lvm/flash_b_d_raid00
mount /dev/flash_b_d_lvm/flash_b_d_raid00 /mnt/data
[/code]
[code]# dd if=/dev/flash_b_d_lvm/flash_b_d_raid00 of=/dev/null bs=512 count=
4000
4000+0 records in
4000+0 records out
2048000 bytes (2.0 MB) copied, 0.0917553 s, 22.3 MB/s
# dd if=/dev/sdb1 of=/dev/null bs=512 count=4000
4000+0 records in
4000+0 records out
2048000 bytes (2.0 MB) copied, 0.161036 s, 12.7 MB/s
# dd if=/dev/sdd1 of=/dev/null bs=512 count=4000
4000+0 records in
4000+0 records out
2048000 bytes (2.0 MB) copied, 0.151505 s, 13.5 MB/s[/code]

Now for a reboot.

Again, Thank You Very Much for your reply.
Regards, Ron

PS. I placed the file you attached in your post in /etc/udev/rules.d then rebooted. Did not have the directory /dev/disk after the reboot? I then hide the original file in the rule.d directory and Puppy would not boot into X. I tried to use the CODE quoting did not work, not sure why? I code marked a LARGE section then several small code sections, neither worked?

FINALLY Working!!!!

[code]vgchange -a y flash_b_d_lvm
mount -t ext2 /dev/flash_b_d_lvm/flash_b_d_raid00 /mnt/data

#mount
/dev/flash_b_d_lvm/flash_b_d_raid00 on /mnt/data type ext2 (rw,errors=continue)[/code]
disktype.txt.gz
Description  LVM debug before and after reboot.
gz

 Download 
Filename  disktype.txt.gz 
Filesize  1.23 KB 
Downloaded  528 Time(s) 
Back to top
View user's profile Send_private_message 
SilverPuppy


Joined: 28 May 2009
Posts: 143

PostPosted: Wed 14 Oct 2009, 00:08    Post_subject: Thanks a bunch.....  

This is really great, I was wondering how to do this.

I'm sorta obsessive-compulsive, you see, and hate to toss anything that still works. I'm going to use this to create RAID 0 swapping with old HDDs less than 1GB. Those old drives were terribly slow, so RAID 0 will double it up for me and help that little problem.

I hope it works. The Slackware file repo browser is down right now, so I had to get creative with the text directory files....but I got the files, finally. We'll get it done. Very Happy
Back to top
View user's profile Send_private_message 
SilverPuppy


Joined: 28 May 2009
Posts: 143

PostPosted: Mon 19 Oct 2009, 10:11    Post_subject: OOPS......  

Well, after arguing with it for awhile, and somewhat getting it working, I discovered that what I was attempting wasn't necessary at all.........all I had to do was create 2 swap partitions on 2 separate devices, replace the swapon command with the "real" one (it's in another forum) so that I could make them the same priority, and viola! Striped swap. So that's that.

I did get it so that the things would sorta work if I formatted it EXT2, but it was very slow, much slower than the physical devices' native speed, and I don't gather quite why. It seemed to take breaks, and have lots of I/O errors. I'm not sure that wasn't due to a mismatch between the partition type flag (which I never changed from swap) and the FS.....however it may be, I now have a very good idea of how to make this all work for future reference, and it's all wunnerful! I'm sure I can make it work in the future. I'm now that much smarter thanks to you all!
Back to top
View user's profile Send_private_message 
Display_posts:   Sort by:   
Page 1 of 1 Posts_count  
Post_new_topic   Reply_to_topic View_previous_topic :: View_next_topic
 Forum index » House Training » HOWTO ( Solutions )
Jump to:  

Rules_post_cannot
Rules_reply_cannot
Rules_edit_cannot
Rules_delete_cannot
Rules_vote_cannot
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group
[ Time: 0.1182s ][ Queries: 13 (0.0042s) ][ GZIP on ]