Defragment Linux file systems

Booting, installing, newbie
Post Reply
Message
Author
artsown
Posts: 403
Joined: Wed 12 Sep 2012, 18:35

Defragment Linux file systems

#1 Post by artsown »

On a hard drive, I had occasion to move three frugally installed puppies
from one ext2 partition to another (empty) partition. I was surprised to
experience a significant performance improvement when using the
moved pups. I thought "hmm, just like defragging Windoze".

It turns out that I had indeed defragged, and stumbled on the
recommended method of doing so (in a way). Googling "linux
defragment" turned up some good articles and explanations. Simply
moving the files out of a partition and back again will do the trick.

To check that everything is cool on sda2 (for example), in a terminal type:
fsck /dev/sda2 -f
(sda2 cannot be mounted for this check)

This will provide you with info on "non-contiguous inodes" which is linux-
speak for "fragmentation" :) I don't know how much fragmentation there
was before I moved pups to a different partition. I don't know how much
will likely lead to significant performance increase after defrag.

Art

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#2 Post by musher0 »

I'm glad you brought the subject back, artsown!

Fragmentation is something that occurs in magnetized hard drives, whatever
the OS. It is a physical thing, as in "physics".

On Linux, 10 % hard disk fragmentation is acceptable, although ever seen,
because the Linux file systems ("ext" or others) have incorporated ways NOT
to let fragmentation happen.

You will find my script "automverif" for hard drives, here. ("Automverif"
stands for automatic verification.) Please use the latest version if you do.

If I may... When you say:
Simply moving the files out of a partition and back again will do the trick.
I beg to differ! If you only do that, the files will go back where they
were, and the fragmentation rate will not change. You have to copy a rather
BIG FILE to the original disk before you recopy your big files for some
defragmentation to happen. This BIG FILE can be full of zeroes and you may
even erase it later, but the disk table used by the file system has to change.

The absolute defragger remains, as you guessed in your 1st paragraph:
-- copy all files from partition A to partition B.
-- reformat partition A
-- recopy all files from partition B to partition A.

BFN.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

artsown
Posts: 403
Joined: Wed 12 Sep 2012, 18:35

#3 Post by artsown »

Hi musher0,

As I understand from the articles I've read, the Linux ext2, ext3 and ext4
(at least) distribute files over the entire partition, keeping them as
separated as possible, so when they grow they can grow without
fragmenting them. If this is true, then it makes sense to me that using
very large partitions having only a few small pups in them should
minimize fragmentation.

Now, it so happens that I have no use for 250 gig hard drive space. So
I created three "small" partitions for various purposes, plus one partition
that's 179 gig. My three little pups drown in that huge space.

So my current experiment with this old pentium D class machine is to
see if any fragmentation of significance of the pup files eventually
occurs. Nutty, huh? :)

Do you have reason to think it will? If so, why?

Art

musher0
Posts: 14629
Joined: Mon 05 Jan 2009, 00:54
Location: Gatineau (Qc), Canada

#4 Post by musher0 »

Hello, artsown.

I'm not saying you need my script! ;) Use it or not -- as you wish.

What I'm saying is --
As hard as we try to limit defragmentation, it is a law of nature concerning
magnetized stuff: any large file stored as ones (1) and zeroes (0) on a hard
drive AND used often will eventually become fragmented.

I think you have a logical approach to it, though.

For the record, Linux does have a FILE defrag utility, written in Python:
https://github.com/ThomasCX/defragfs
One advantage is that the partition the file is on can stay mounted.

But be warned, you will have to repeat the process many times before your
big file finally congeals into one single fragment.

Finally, my preferred one-liner for doing an fsck of partitions is

Code: Select all

fsck."$2" -p -f -v /dev/"$1"
where $2 is the ext type and $1 is the partition.
The "-p" parameter sets automatic correction on.

Best of luck with your approach.

BFN.
musher0
~~~~~~~~~~
"You want it darker? We kill the flame." (L. Cohen)

User avatar
bigpup
Posts: 13886
Joined: Sun 11 Oct 2009, 18:15
Location: S.C. USA

#5 Post by bigpup »

This is only about hard drives.

I tried to do something about fragmentation in Linux formats and never found any program that worked that good.
As you say. Seems you had to run the program many times to really see any improvement.

To eliminate the possibility of fragmenting. (ext3, ext4 formats)
Make sure you always have some free space in a Linux formatted partition.
That gives it room to self adjust the location of files, as needed, to stop fragmentation.

I just checked a partition formatted ext 4 that I have used for a long time and has a lot of stuff on it that has been deleted, added, moved around, etc.........

The check results:
Total/best extents 37004/35921
Average size per extent 990 KB
Fragmentation score 0
[0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag]
This device (/dev/sdc7) does not need defragmentation.

This is already in most Puppies, but it only works on ext4 formats.
The partition has to be mounted to check.

Code: Select all

e4defrag -c /dev/partition ID.
Example:

Code: Select all

e4defrag -c /dev/sdc3
to check partition sdc3.

e4defrag (-v) option will run the DE-fraging process.

1
The things they do not tell you, are usually the clue to solving the problem.
When I was a kid I wanted to be older.... This is not what I expected :shock:
YaPI(any iso installer)

artsown
Posts: 403
Joined: Wed 12 Sep 2012, 18:35

#6 Post by artsown »

@bigpup
There's a issue with ext4 when using Fatdog's humungus initrd to get
fast booting. Ext4 does the opposite, causing bootup to be much
slower than when using ext2 or ext3. Jamesbond and Kirk have
talked about this.Kirk prefers ext3.

To me, this is unfortunate since e4defrag looks attractive, and
fatdog 721 is one of my favorite pups. Right now, I've split my little
kennel so I can use ext4 for xenialpup64 and tahrpup64, and ext3 (now)
for fatdog. Just a bit awkward, but no big deal :)

Art

User avatar
bigpup
Posts: 13886
Joined: Sun 11 Oct 2009, 18:15
Location: S.C. USA

#7 Post by bigpup »

Wonder what

Code: Select all

e4defrag -c
would show for that ext4 partition, if Fatdog was installed to it. :idea:
The things they do not tell you, are usually the clue to solving the problem.
When I was a kid I wanted to be older.... This is not what I expected :shock:
YaPI(any iso installer)

p310don
Posts: 1492
Joined: Tue 19 May 2009, 23:11
Location: Brisbane, Australia

#8 Post by p310don »

I once read a method of defragging linux that apparently works. I haven't used it.

Basically compress the entire hard drive into a tar.gz, wipe (format the partition) and then uncompress the tar.gz file onto the fresh partition.

User avatar
tallboy
Posts: 1760
Joined: Tue 21 Sep 2010, 21:56
Location: Drøbak, Norway

#9 Post by tallboy »

fsck was default in my old Debian distros, I think 30 bootups was the limit before you got the message at bootup that the HDD would be checked first.
True freedom is a live Puppy on a multisession CD/DVD.

s243a
Posts: 2580
Joined: Tue 02 Sep 2014, 04:48
Contact:

#10 Post by s243a »

artsown wrote:@bigpup
There's a issue with ext4 when using Fatdog's humungus initrd to get
fast booting. Ext4 does the opposite, causing bootup to be much
slower than when using ext2 or ext3. Jamesbond and Kirk have
talked about this.Kirk prefers ext3.

To me, this is unfortunate since e4defrag looks attractive, and
fatdog 721 is one of my favorite pups. Right now, I've split my little
kennel so I can use ext4 for xenialpup64 and tahrpup64, and ext3 (now)
for fatdog. Just a bit awkward, but no big deal :)

Art
Sorry to sidetrack things but what was the reason for using the large initrd rather than using a "base sfs" file like a typically puppy?

P.S. Why not put the fatdog64 save file on the ext4 partition and put the initrd file on a different partition? Heck, you could put the initrd on a fat partition if you wanted (I think).

artsown
Posts: 403
Joined: Wed 12 Sep 2012, 18:35

#11 Post by artsown »

@ s43a
The humungous initrd can speed up bootup times, as I had said.
In my experiences, the speedups have been quite impressive.

Your idea of placing the fatdog save file only on a ext4 partition
makes sense. I implemented this and so far so good :) I'm considering
the idea of placing only save files/folders in a partition that's set aside
for fragmentation checks and possible defrags.

Thanks for the idea!

Art

Post Reply