TestDisk: Flash Drive="CHS and LBA don't match" [SOLVED]

Using applications, configuring, problems
Message
Author
linuxcbon
Posts: 1312
Joined: Thu 09 Aug 2007, 22:54

#141 Post by linuxcbon »

Karl Godt, no,no, linux doesnt work like windows, it just needs a few lines in some module. I wonder is that doesn't exist already...

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#142 Post by Sylvander »

jrb:
1. In Slacko I downloaded the rar file and unpacked it...
Rebooted to FalconFour's_UBCD->MiniXP and got:
a. The ChipGenius exe file. [ran this and it worked]

b. The FlashGenius.exe file. [what does this do?]
c. The MyDiskTest.exe file. [what does this do?]
I'd run them here in MiniXP if I knew all would be well.

User avatar
jrb
Posts: 1536
Joined: Tue 11 Dec 2007, 19:56
Location: Smithers, BC, Canada

#143 Post by jrb »

Sylvander wrote:jrb:
1. In Slacko I downloaded the rar file and unpacked it...
Rebooted to FalconFour's_UBCD->MiniXP and got:
a. The ChipGenius exe file. [ran this and it worked]

b. The FlashGenius.exe file. [what does this do?]
c. The MyDiskTest.exe file. [what does this do?]
I'd run them here in MiniXP if I knew all would be well.
Through the miracle of Google Translate with which I found this tool in the first place :D :
FlashGenius http://www.mydigit.cn/flashgenius.htm
FlashGenius a FLASH memory parameter query tools, you can quickly find out the FLASH chip manufacturer, product category, operating voltage, storage capacity, chip version, feature type.
Green compact, easy-to-use operating software, electronics enthusiasts and digital service personnel around good assistant.
MyDiskTest http://www.mydigit.cn/mydisktest.htm:
U disk expansion detection tool first truly

Set several major functions in one: expansion detection, bad blocks scanning speed test, bad block shield
MyDiskTest expansion Recognition Tool is a U disk / SD card / CF card, mobile storage products. Can easily detect whether through the expansion capacity storage products, shoddy.
FLASH memory can also detect whether there is a bad block, whether black piece, does not destroy the original data of the disk.
U disk read and write speeds, and can test aging test storage products. Is the you pick U disk and memory card essential tool.
Sadly I couldn't seem to get anything out of these two. As a matter of fact MyDiskTest just crashed as soon as I hit what I hoped was the test button. :cry:

I think we may need someone who reads Chinese to figure out whats going on with these. Hurts to be technologically disadvantaged. :lol:

Just so you know I have my Win7 partition imaged 2 different ways and all my data on a separate partition.

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#144 Post by Sylvander »

Just ran:
A. FlashGenius.exe
The program ran OK.
Displayed a window with lots of boxes to input various values for the Flash Drive, so as to be given further information about the drive.

B. MyDiskTest.exe
The program ran OK.
Most of the text is in Chinese.
Other stuff is just strange characters.
The Verbatim Store-n-Go is detected/identified in the drive box.
There are 4 buttons; the last one is Stop.
1st has a magnifying glass [study the storage regions in detail?]
2nd seems to be moving/copying files. [there are none on the drive]
3rd eliminating faulty regions?
I'll try them [named as B1, B2, B3].

B1. Drive scan completed; I could see the LED blinking during scan.
Results are cryptic.
Time taken to complete various steps.
Lots of ???????'s after the times.
Drive identified: Verbatim etc.
Info in order = 692.63MB, 696.00MB, 694.00MB, 99.86%, 1024MB.

B2. Failed: big red X.

B3. Failed: big red X.

User avatar
jrb
Posts: 1536
Joined: Tue 11 Dec 2007, 19:56
Location: Smithers, BC, Canada

#145 Post by jrb »

Sylvander wrote:Info in order = 692.63MB, 696.00MB, 694.00MB, 99.86%, 1024MB.

B2. Failed: big red X.

B3. Failed: big red X.
It would certainly be interesting to know what those failures mean, couldn't complete test or drive unacceptable.

Did you check your drive properties with another program (partview?) to see if reported size had changed after running it through this?

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#146 Post by Sylvander »

jrb wrote:Did you check your drive properties with another program (partview?) to see if reported size had changed after running it through this?
Partview and GParted show all's well with all drives/partitions. :D

linuxcbon
Posts: 1312
Joined: Thu 09 Aug 2007, 22:54

#147 Post by linuxcbon »

Sylvander wrote:Partview and GParted show all's well with all drives/partitions. :D
Can you please unplug and then plug in the disk ? Then paste here the last lines of dmesg ?
Another thing, can you paste here the result of "hdparm -i /dev/sdb1" ?

It's weird no linux tool to reset a flash disk , except a russian software for windows ? I will have to ask kernel developers...

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#148 Post by Sylvander »

1. Final lines of the results of dmesg:

Code: Select all

[ 1021.468241] usb 1-1.1: Product: Store 'n' Go
[ 1021.468243] usb 1-1.1: Manufacturer: Verbatim
[ 1021.468244] usb 1-1.1: SerialNumber: de656f60c080aa
[ 1021.468834] scsi5 : usb-storage 1-1.1:1.0
[ 1022.471018] scsi 5:0:0:0: Direct-Access     Verbatim Store 'n' Go     1.00 PQ: 0 ANSI: 2
[ 1022.472256] sd 5:0:0:0: [sdb] 1425407 512-byte logical blocks: (729 MB/695 MiB)
[ 1022.472739] sd 5:0:0:0: [sdb] Write Protect is off
[ 1022.472743] sd 5:0:0:0: [sdb] Mode Sense: 00 00 00 00
[ 1022.473247] sd 5:0:0:0: [sdb] Asking for cache data failed
[ 1022.473249] sd 5:0:0:0: [sdb] Assuming drive cache: write through
[ 1022.475966] sd 5:0:0:0: [sdb] Asking for cache data failed
[ 1022.475969] sd 5:0:0:0: [sdb] Assuming drive cache: write through
[ 1022.477727]  sdb: sdb1
[ 1022.480105] sd 5:0:0:0: [sdb] Asking for cache data failed
[ 1022.480109] sd 5:0:0:0: [sdb] Assuming drive cache: write through
[ 1022.480112] sd 5:0:0:0: [sdb] Attached SCSI removable disk
# 
2.

Code: Select all

# hdparm -i /dev/sdb1

/dev/sdb1:
hdparm: ioctl 0x304 failed: Invalid argument
hdparm: HDIO_GET_IDENTITY: Invalid argument
# 

linuxcbon
Posts: 1312
Joined: Thu 09 Aug 2007, 22:54

#149 Post by linuxcbon »

Try "hdparm -i /dev/sdb" instead ?

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#150 Post by Sylvander »

Code: Select all

# hdparm -i /dev/sdb

/dev/sdb:
hdparm: ioctl 0x304 failed: Invalid argument
hdparm: HDIO_GET_IDENTITY: Invalid argument
# 

amigo
Posts: 2629
Joined: Mon 02 Apr 2007, 06:52

#151 Post by amigo »

hdparm doesn't work on flash drives.

User avatar
Karl Godt
Posts: 4199
Joined: Sun 20 Jun 2010, 13:52
Location: Kiel,Germany

#152 Post by Karl Godt »

hdparm does not run good to aquire information from usb flash drives.

On newer sda* kernels it should be invoked like

Code: Select all

hdparm -iI /dev/sd*
Note the BIG EGG "I" insted the little one. The little one is "just in case" .

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#153 Post by Sylvander »

Code: Select all

# hdparm -iI /dev/sdb

/dev/sdb:
hdparm: ioctl 0x304 failed: Invalid argument
hdparm: HDIO_GET_IDENTITY: Invalid argument
hdparm: HDIO_DRIVE_CMD: Invalid argument
# 

npierce
Posts: 858
Joined: Tue 29 Dec 2009, 01:40

#154 Post by npierce »

Sylvander wrote:Any idea what is different about this "Recovery Tool" that makes it able to do what all those other things failed to do?
Short answer - - -

This utility knew how to talk to the specific controller chip that is in your flash drive. And it knew what registers on the chip it needed to write to and what it had to write to them in order to properly do a low-level format. In short, it spoke the controller's language.

There are a lot of different models of flash drives in this world, and a lot of different controller chips for flash drives. There are a bunch of utilities that can do a low-level format on a flash drive, some designed for specific controller chips, some designed to support many controller chips, but apparently none that support every controller chip ever invented.

So the key was to find a utility that actually knew about the controller chip in your flash drive. (That's where we needed the 0a16:2004 (the vendor code and product code that you found with the lsusb command), and the database at flashboot.ru that listed the controller chips used in drives with the 0a16 vendor code and the utilities successfully used by others to talk to those chips.)

If your flash drive had been healthy, there would not have been the need to use a utility that knew how to do a low-level format with your specific controller chip. For a healthy drive, the Linux utilities you tried, such as mkdosfs and mke2fs would normally have worked, but they just do a filesystem format and you needed a low-level format because your drive had problems.


Long answer - - -

Along with the area used for filesystems by the operating system (be it Linux, Windows, or whatever), there is other data contained in your drive such as information about the drive itself and tables used to map logical disk sectors to physical blocks of flash memory.

The Linux kernel, like UNIX, allows programs to look at devices as if they are files. Of course, a piece of hardware is not just a file. For instance, accessing a hard drive requires heads to be positioned over the desired data, the correct head to be used to read the desired track, and formatting information to be interpreted to determine the desired sector. Or, for a flash drive, the flash data block that corresponds to the desired sector needs to be looked up. But the kernel driver and the firmware in the drive itself take care of those details, so that a program can just directly access any byte in the area of the drive used for filesystems. This doesn't include the area with the other data mentioned above.

Of course, most programs don't want to look at disk drives as files, they want to look at actual files on a mounted filesystem, and so most programs would rather talk to a partition's mount point (e.g., /mnt/sdb1) rather than its device node (e.g., /dev/sdb1) or the entire drive's device node (e.g., /dev/sdb). But some utility programs, such as mkdosfs do want to talk to the partition's device node. And other utilities, such as fdisk want talk to the device node for the entire drive.

When you format a partition in a flash drive with mkdosfs, you are initializing certain sectors of the partition with information used by the file system, such as the first sector (a.k.a. boot sector) (that contains a description of the filesystem (bytes per sector, sectors per cluster, location and number of allocation tables, etc.), the allocation tables themselves, and the partition's root directory (i.e, the top directory, not to be confused with the directory named "root").

Similar information is initialized when you use mke2fs, although of course the exact format is different for Linux filesystems.

Anyway, this filesystem format only writes to the partition, which is, obviously, part of the area of your drive that is used for filesystems. It doesn't write to the other areas containing information about the drive itself, or the tables used to map logical sectors to physical blocks. And so it is not a low-level format.

Usually, if the drive is healthy, this is enough, and a low-level format is not necessary. Your drive was not healthy.


If you are unfamiliar with the term "low-level format", I'll try to explain.

Exactly what happens during a low-level format is different for a flash drive than it was for magnetic media, like hard drives and floppy drives. But in both cases it involves writing data to the drive that is necessary for the drive to properly store and retrieve your data. This low-level data is not something that the operating system normally needs to concern itself with. The operating system is normally only interested in the data used for its filesystem format, the data in the directories, and the data in the files. On modern drives, the low-level data is normally only seen by the drive itself, not by the operating system. But without the low-level data being correct, the filesystem format data, the directory data, and the file data will not be stored correctly.

When talking of traditional hard drives or floppy drives, based on magnetic media, low-level formatting is what writes the addressing information to the unformatted magnetic particles on the surface of the disk, because, as you probably know, the disk contains not only the data that you write to it, but also many bits of data needed to identify the sectors so that your data can be found again after you write it.

Modern operating systems usually don't need to know the details of low-level formatting. They just need to know how to ask a disk drive to give it the data in sector 12345 (or whatever), or to say here is a sector's worth of data, please write it to sector 54321 (or whatever).

Hard drives have been given the low-level format at the factory. Floppy disks also come preformatted (assuming that such things are still manufactured :) ), but in the early days they did not, so operating systems have the ability to do a low-level format on them. But my guess is that the operating system doesn't need to know the details of low-level formatting even to do that. I suspect they just talk to the PC's BIOS or the firmware in the floppy drive and say, "please do a low-level format on this floppy."

(One exception was the first disk operating system for a personal computer (although, as you may remember, they were usually known as "home computers" or "micro computers" at the time) which did need to know about low-level formatting. If Steve Wozniak could figure a way to reduce the number of chips in Apple II hardware by even a single chip, he would do so. So rather than designing a complex controller for the floppy disk drive, he decided to just create a very simple controller, and do the complex stuff in software, in the operating system itself. So instead of just telling the drive to format the disk, Apple DOS had to tell the disk to start spinning and to move the read/write head to the first track, and then it sent a stream of data bits (containing the formatting data) directly to the head. Then it would command the stepping motor to move the head to the next track and write another stream of bits, and so on, and so forth.)

(I don't know if the original IBM PC DOS had to know any of the details of low-level formatting, but I suspect not. That knowledge was probably contained in the BIOS or the firmware for the floppy drive.)

But you don't want to know about magnetic media, you want to know about flash drives.

Since the main component of a flash drive is a big memory chip, one might think there would be no need for a low-level format. Shouldn't it be possible to directly address the chip? There shouldn't be any need to write addressing bits to the sectors in order to find them again. True. But things aren't that simple. Low-level formatting is very different for a flash drive than for a spinning magnetic disk. Although writing address bits is not necessary, other stuff is.

For instance, the type of flash memory used in most flash drives ("NAND flash") is written in blocks larger than the typical 512-byte sector size of a PC disk drive. And flash drives do "wear leveling" to prevent blocks from wearing out quickly because of being written much more often than others -- which means that when a sector is re-written, it is probably not going to be physically located in the same place in the flash chip as it was before.

For those reasons (and probably others), the logical sectors must be mapped to the physical blocks on the flash chip (i.e, sector 12345 (or whatever) is currently in block 9876 (or whatever)). It would be my guess that low-level formatting does stuff like set up the tables used for this mapping.

Also, there is configuration data contained in the flash drive, such as its size. Some of that probably gets rewritten as well. (This utility should automatically write the correct values there, but there are people in this world who use other utilities to write false values for the drive size. If you see an unbelievably good price on ebay for flash drives, chances are good that they have been counterfeited by doing just that. The drive may be advertised as, say, 32 GB, but contain maybe 8GB. You can plug it into your PC and it will claim to have 32 GB, and will work fine at first. But, needless to say, when a block of data beyond its real memory is written the result won't be pretty.)

Another important part of low-level formatting is probably the detection of bad blocks, and ensuring that the tables don't allow the bad blocks to be used. (It is not unusual for even a new flash drive to have some bad blocks -- but if the low-level format is good, you will never see them. It is more economical for a chip manufacturer to cram as many memory cells as possible onto a chip, knowing that some will likely be bad when manufactured, then it is to reduce the density enough to get perfect chips with all good cells, but fewer of them.)


Anyway, Sylvander, I've probably told you more here than you wanted. So I'll not ramble on further for now. Take all the above with a grain of salt, because I am certainly no expert on flash drives. I see the words "probably" and "guess" sprinkled liberally through my above text. Let those words remind you not to treat all the above as fact, since it's just the view from someone who doesn't yet have a clear understanding of the subject. I wish I could be more definite about all this, but I've only just begun learning about flash drives -- mostly what I've learned has been a result of this thread.

Sylvander
Posts: 4416
Joined: Mon 15 Dec 2008, 11:06
Location: West Lothian, Scotland, UK

#155 Post by Sylvander »

WOW! Thanks for the explanation npierce. :D

I've read it, and some of it is beyond my level of understanding, but I get the general idea, and will re-read again and again, to try to understand.

1. Understood all of the short answer.
Understand:
Low-level-format.

2. Don't understand:
Mount point; device node.

npierce
Posts: 858
Joined: Tue 29 Dec 2009, 01:40

#156 Post by npierce »

Hi Sylvander,

You're welcome.
Sylvander wrote:2. Don't understand:
Mount point; device node.
As you've probably noticed, Puppy (or any Linux) likes to organize everything into a tree of files -- even things that are not really files. Hardware devices like your flash drive and the individual partitions on your flash drive can be accessed via the /dev/ directory. By now you are well aware that, in your case, Puppy detects your flash drive at /dev/sdb and its first partition at /dev/sdb1. Each of those is known as a "device node" since they are not really files.

But even though they are not actual files, they can be read and written to as if they were actual files. For instance, when reading the partition table fdisk opens /dev/sdb just as if it were a file, then reads 64 bytes beginning with the byte which is 446 bytes from the beginning of the "file". Similarly, mkdosfs opens /dev/sdb1 just as if it were a file, and writes formatting information to the appropriate locations in partition 1.

The fdisk and mkdosfs utilities need to access places on the drive that most applications never need to go, and which do not appear as entries in the tree of directories and files on the flash drive partition. So it is appropriate for them to use /dev/sdb and /dev/sdb1.

But most applications are interested in accessing files. Applications certainly could access a file through /dev/sdb1 or even /dev/sdb, but it would involve a lot of work for them to do so. The application would have to know a lot about the format of the filesystem used (FAT32, ext2, or whatever) including its directory structure, and do a lot of math to track down all of the bytes in the desired file.

To avoid all of that math, we mount drives and let the kernel do the math. Then applications can just open a file and read or write any byte in the file without knowing anything about where it physically lives on a disk.

And to mount a file, you need a mount point.

Say I have a flash drive with two directories in the root directory of partition 1. (Again I am talking about the top directory of partition one, not a directory named "root". That can be confusing.) The two directories are named "Music" and "Photos". So the directory tree looks like this:

Code: Select all

/
|-- Music
`-- Photos
Then say that the tree of my Puppy directories looks something like this (with all the sub-directories omitted for clarity):

Code: Select all

/
|-- archive
|-- bin
|-- dev
|-- etc
|-- lib
|-- mnt
|-- opt
|-- proc
|-- root
|-- sbin
|-- sys
|-- usr
`-- var
Then I plug the drive in. Puppy detects it at /dev/sdb1, creates a directory at /mnt/sdb1, and mounts the drive on that directory. Now the tree of my Puppy directories looks like this:

Code: Select all

/
|-- archive
|-- bin
|-- dev
|-- etc
|-- initrd
|-- lib
|-- mnt
|   `-- sdb1
|       |-- Music
|       `-- Photos
|-- opt
|-- proc
|-- root
|-- sbin
|-- sys
|-- usr
`-- var
The /mnt/sdb1 directory is called a "mount point" because it is the place in the tree that partition 1 of the flash drive was mounted at.


In summary: The device node allows the raw data on the disk drive or partition to be read as one gigantic file. The mount point allows the files on the drive to be accessed through the filesystem's tree of directories and files.


I hope this makes things more clear, not less. :)

Post Reply