How to reduce the size of a HDD image?

Puppy related raves and general interest that doesn't fit anywhere else
Post Reply
Message
Author
John Alien
Posts: 17
Joined: Sat 27 May 2017, 00:53

How to reduce the size of a HDD image?

#1 Post by John Alien »

It's a bit of a general linux question.. How can I reduce an image of a 250gb hard drive to write the image to a 128gb ssd, provided that there is enough free space on the 250gb hard disk?

If I write the hard drive as a qcow2 image it will reduce the size? Or does it have to also be written to a device which has TRIM enabled?

Can I use dd to write the qcow2 image to an ssd?

User avatar
Flash
Official Dog Handler
Posts: 13071
Joined: Wed 04 May 2005, 16:04
Location: Arizona USA

#2 Post by Flash »

Have you considered defragmenting the 250 GB drive? Done properly, defragmenting should cram all the data on the drive to the beginning of the drive, with no wasted space. Assuming there is less data on that drive than will fill the drive you want to copy it to, that should do the trick if the trick can indeed be done.

I think dd copies the data, bit for bit, including unused areas in a badly fragmented drive. So you'd need to defragment a drive before you used dd to copy everything from it to a smaller drive.

s243a
Posts: 2580
Joined: Tue 02 Sep 2014, 04:48
Contact:

Re: Reducing the size of an image

#3 Post by s243a »

John Alien wrote:It's a bit of a general linux question.. How can I reduce an image of a 250gb hard drive to write the image to a 128gb ssd, provided that there is enough free space on the 250gb hard disk?

If I write the hard drive as a qcow2 image it will reduce the size? Or does it have to also be written to a device which has TRIM enabled?

Can I use dd to write the qcow2 image to an ssd?
I'm not sure. You could try it and see what happens. Here are some instructions.

https://blog.sviluppoweb.eu/2010/07/16/ ... x-qemukvm/

I think though if you defragment a disk that you shouldn't need to image the whole thing so maybe either "dd" or "safecopy" would be fine.


Once you've created the image you can mount it as a loop device.

Here are three example mount commands:

Code: Select all

losetup /dev/loop3 /mnt/sdg6/part3.dat
mount -o loop -t ntfs-3g /mnt/sdg6/part3.dat /mnt/sdg4/
mount -o loop /mnt/sdg6/part3.dat /mnt/sdg4/
Some cloning tools can create spare images. For example with ntfsclone you can create a sparse image of an ntfs partition.

Some further research tells me some ways that you can create a sparse file:

Code: Select all

dd if=/dev/sdb1 | cp --sparse=always /proc/self/fd/0 new-sparse-file
https://unix.stackexchange.com/question ... rom-a-raid

Code: Select all

$ ddrescue -S -b8M /dev/sda1 /mount/external/backup/sda1.raw
https://unix.stackexchange.com/question ... parse-file

In the last option the capital "S" is for sparce copy in ddresecule. See the manpage:
https://linux.die.net/man/1/ddrescue

s243a
Posts: 2580
Joined: Tue 02 Sep 2014, 04:48
Contact:

#4 Post by s243a »

Flash wrote:Have you considered defragmenting the 250 GB drive? Done properly, defragmenting should cram all the data on the drive to the beginning of the drive, with no wasted space. Assuming there is less data on that drive than will fill the drive you want to copy it to, that should do the trick if the trick can indeed be done.

I think dd copies the data, bit for bit, including unused areas in a badly fragmented drive. So you'd need to defragment a drive before you used dd to copy everything from it to a smaller drive.
You beat me to it by 6 seconds! Anyway, I suspect that whatever approach the original post takes that better results will be obtained by defragmenting the disk first.

p310don
Posts: 1492
Joined: Tue 19 May 2009, 23:11
Location: Brisbane, Australia

#5 Post by p310don »

What is the image of? What is your end goal?

My experience with copying an old XP hard drive to a VM has been to defrag first and then resize the partition to leave almost no free space. Then create the image. Then create a VM from the resultant image.

You can also reduce the size of the image for a VM using tools for the VM program (I use VirtualBox).

John Alien
Posts: 17
Joined: Sat 27 May 2017, 00:53

#6 Post by John Alien »

In case it's unclear, the goal is to copy the bootloader and the operating system from a 250gb harddrive to a 128gb ssd

I just found a possible solution combining these two ideas:

https://serverfault.com/questions/43912 ... ty-portion
https://raspberrypi.stackexchange.com/q ... ting-possi

use gparted to shrink the disk to minimum possible size, leaving rest of the space unallocated
Use dd to create an .img of the disk
Use fdisk to check where the last partition on the disk ends
Use truncate on the .img
dd .img to ssd


I'd say it's very possible resizing a partition defragments the disk... Does anyone know?

I see p310don does something similar. And there is a defrag involved in it's own right.

Not that I have anything against defragging, but is it necessary to do before reducing the partition size?

Also if I'm pretty sure I'll need to truncate after making the disk image. AFAIK dd will always output the whole disk irregardless of where partitions are.

I wonder if qcow2 can be used to make an image only the size the disk's partitions...

p310don
Posts: 1492
Joined: Tue 19 May 2009, 23:11
Location: Brisbane, Australia

#7 Post by p310don »

What is the original system? Is it linux or windows?

Linux is much more tolerant of being copied. I have had success with simply copying everything over to a new hard drive / SSD. Windows is not tolerant at all of such a thing.

If it is windows, a defrag is necessary (if the drive is fragmented) to compress the partition down.

If linux source, boot from live medium (CD or USB) Puppy is great for this! Compress entire hard drive into single .gz file, uncompress onto bootable SSD.

If windows source, defrag (which essentially compresses and decompresses into one area of the disk), then boot Puppy with live medium, use gparted to resize partition to have no free space. Use PUDD to copy the partition from the old drive to the new SSD (PUDD being a basic front end for DD). Then use gparted again to grow the partition to whatever size you like (probably to fill the disk).

You can use the windows method for Linux of course. YMMV with how tolerant your OS of choice is.

User avatar
Flash
Official Dog Handler
Posts: 13071
Joined: Wed 04 May 2005, 16:04
Location: Arizona USA

#8 Post by Flash »

I don't know if resizing a disk defragments it, but I think copying (not using dd) automatically defragments the data in the process of copying it. In other words, I think that copying first assembles entire files before writing them to the destination.

User avatar
Burn_IT
Posts: 3650
Joined: Sat 12 Aug 2006, 19:25
Location: Tamworth UK

#9 Post by Burn_IT »

The key here is to stop using the word IMAGE.
What you are trying to do is copy the CONTENTS of the disk to the SSD.
An IMAGE can really only be copied to a file or to a disk with the same architecture (or larger).
In your case you are not only wanting just the actual data, but you need it repacking for the new format of the SSD.
The safest way of doing it is with a disk handling tool that is designed to copy old disks to new ones such as GPARTED or any other stand alone Disk/Prtition tool.
It is also best to use a bootable tool so that both the intended source and target disks are not in use.
"Just think of it as leaving early to avoid the rush" - T Pratchett

John Alien
Posts: 17
Joined: Sat 27 May 2017, 00:53

#10 Post by John Alien »

p310don wrote:What is the original system? Is it linux or windows?
The source is puppy.
p310don wrote:If linux source, boot from live medium (CD or USB) Puppy is great for this! Compress entire hard drive into single .gz file, uncompress onto bootable SSD.
So I can use dd to store the contents of a disk in a gz file and I can point dd towards the entire device ie. /dev/sdb as the input file? And is it compressing with gz the operation which removes the empty space in this case?
p310don wrote:Use PUDD to copy the partition from the old drive to the new SSD (PUDD being a basic front end for DD).
Puppy is amazingly versatile, good work everyone involved.

John Alien
Posts: 17
Joined: Sat 27 May 2017, 00:53

#11 Post by John Alien »

Burn_IT wrote:The key here is to stop using the word IMAGE.
Ok.

p310don
Posts: 1492
Joined: Tue 19 May 2009, 23:11
Location: Brisbane, Australia

#12 Post by p310don »

The source is puppy.
Well that opens up a whole other can of worms. Most operating systems aren't made to be portable. Puppy is.

If you used a frugal install, you can simply make the SSD bootable, copy grub or grub4dos or whichever bootloader you have, copy the puppy files from the ISO and then copy over your save file & VOILA puppy is installed on the new drive. No need to much about with DD, PUDD, qcow2 or any other convoluted process. That is one of the million things that makes puppy awesome.
So I can use dd to store the contents of a disk in a gz file and I can point dd towards the entire device ie. /dev/sdb as the input file? And is it compressing with gz the operation which removes the empty space in this case?
DD can create an image of the drive. PUDD creates that image, but compresses it into a GZ file, making it about half the size. DD (and PUDD) give you an exact duplication of the drive, including the empty bits.
You can simply compress the files on your drive into an archive of your choice, and then uncompress them to get rid of the empty spaces between files. This is what defrag does. If your source is Puppy, I might assume you have an ext3 or 4 file system, so you don't really need to worry about defrag. You can of course create the archive file and then unpack it onto the SSD, but that is an extra step beyond simply copying the files.


For what you're doing, my method would be:

Install SSD into PC.

Fire it up to boot Puppy. Either from USB, or from old HDD if you have more than one drive installed.

Open Gparted, format the SSD to be bootable. Right Click, select manage flags, tick make bootable.

Click Install on desktop

Go through the install process

The copy your old hard drive's files to the SSD, overwriting everything.

Cross your fingers

Turn off the PC. Take out the old drive. Turn on the PC.

Cross your other fingers.

Marvel at the 40x speed increase from SSD :)

John Alien
Posts: 17
Joined: Sat 27 May 2017, 00:53

#13 Post by John Alien »

Sounds good. Will I have to also install grub on the ssd as part of the installation process?

I'm a little unsure of how grub works.

Format the SSD to be bootable.
Install puppy, let the installer partition.
Install Grub (or should grub be installed before puppy? Does it matter?)
Copy, essentially something like 'cp -rf /mnt/sda/ /mnt/sdb/'

And then grub fits in the process like: Hardware should see the bootable drive, start grub, grub reads the partition table and then it will know where to find the kernel?

Which, hopefully, boots into the system I'm try to copy?

When grub reads the partition table, does it then read the grub configuration files from the partition with /boot?
Last edited by John Alien on Thu 29 Jun 2017, 23:44, edited 1 time in total.

cthisbear
Posts: 4422
Joined: Sun 29 Jan 2006, 22:07
Location: Sydney Australia

#14 Post by cthisbear »

My Chinese computer shop techie had a Toshiba laptop that had issues.

I gave him Puppy 4.21 at the time and newer Puppies later as rescue
disks. Plus many others.

He always copied the data with Puppy to a spare drive,
and used the hidden partition to do the Factory re-install.

Then recopy the data back.
Anyway this particular Toshiba refused to boot properly for some reason.
He redid it twice...no go.

I got him to give me a spare hard drive.
Copied his new but failing Windows install...

Program Files
Users
Windows
Documents and Settings
ProgramData

Booted a Windows disc....and installed Windows 7 just one percent,

Then I rebooted in Puppy, deleted any Win files and just copied
all the Win files from the spare hard drive back to the main drive.

Checked GParted that it could boot, and rebooted.

Voila! booted cleanly into Windows.
He could then copy the data back.

He knew Puppy was good...but this was awesome to him.

When I install any Windows system, I always boot Puppy straight after, and clean the new install out, and make a backup.

I then only ever install the drive1%, clean it out as above with Puppy, and recopy the files. Never had a fail yet.

Chris.

John Alien
Posts: 17
Joined: Sat 27 May 2017, 00:53

#15 Post by John Alien »

Very good to know that about Windows 7.

User avatar
Flash
Official Dog Handler
Posts: 13071
Joined: Wed 04 May 2005, 16:04
Location: Arizona USA

#16 Post by Flash »

cthisbear, please explain what you mean by installing Windows 7 "1%" :?

cthisbear
Posts: 4422
Joined: Sun 29 Jan 2006, 22:07
Location: Sydney Australia

#17 Post by cthisbear »

" explain what you mean by installing Windows 7 "1%" "

"""""""

Whether from dvd, usb etc; you just start the Windows install.

As soon as this action starts and files install..kill it.

"""'

Obviously copy all your data to a backup drive.

I also might run Driver Magician Lite to back up any extra system drivers
if Windows is still alive..

http://www.drivermagician.com/Lite.htm

If I am satisfied that there are no bad virus issues I just move the complete Windows install and all other folders to a folder I call

1 Win old....etc.

except for
System Volume Information
$RECYCLE.BIN or whatever name it comes up with.
pagefile.sys
hiberfile.sys
which I delete.

In saying this, it is amazing to find docs and pics in the Recycle Bin
that people really didn't want to delete.
Also in the root of the drive, because people are thick and nothing surprises me anymore.
Because you can bet if you delete someone will suddenly want it.
Why I don my Mandrake the Magician hat in case.

::::::

Always best to scan any saved data. Trust to nothing.

When I need a driver I burgle it by going into the old win setup folder

1 Win oldWindows\Windows\inf
and let the newer setup search for the driver I require.

I may then have to find the the files in the
1 Win old\Windows\System32 folder

and finish it by finding the last files again in the
1 Win oldWindows\Windows folder.

This has saved me time and grief over countless re-installs because
hardly anyone keeps the installation drivers and so many companies
never keep them on their sites because they are glad to sell you a new pc again. Who would guess that?
Packard Bell is a s..t company like that.

Again you can often use older Vista drivers in Windows 7 when there are no updates.
I have used sound drivers in Windows 10 from Windows 7...bloody Dell.

But you have to pick them and I usually use my above method.

Sometimes I try and unzip the driver pack into a new folder
and use the unzipped inf
from that folder to install the driver because Windows won't let the
driver installer run.

If there is plenty of hard drive space I always start with 2 partitions
so if ugliness happens later and there is no spare drive I can move files
quickly and start a new install again.

Personally I never reformat a drive unless I suspect the absolute
worst has occurred and I prefer DBan etc.

::::

Anyway just to give a quick visual.
I generally do a non destructive install.

Windows 7 Formatting and Clean Installation

https://www.youtube.com/watch?v=lG0k7Z-HDuc

View the video from say the time 7.37 until 17.59.
Then stop it....don't be nice if it won't.
Hold the Power Button down.

Reboot in Puppy and remove all the Win files.
Copy the backed up clean first install with Puppy, reboot....job done.

Here is my backed up clean install under a folder I call

c:\1 Win7 2015 Backup

Program Files
ProgramData
Users
Windows
autoexec.bat
config.sys
Documents and Settings
IO.SYS

Done this for years with Win95, 98, WinMe, XP, Win7, Win 8 and I am sure that Win 10 would give no issues.
The only problems to consider is the newer GPT partition issue.

Chris.

Post Reply