Recently I was handed an old Dell laptop to find out why it was so slow. There was also a problem with it not recognizing that it had an official Dell battery or power adapter.
Problems persisted after swapping the hard drive. I also checked for a memory configuration error that could cause a drastic slowdown. Naturally, it also had a dead battery for the real-time clock. That was easy to replace.
I eventually found that the UEFI code in serial flash memory used at bootup was corrupted. The natural solution is to reflash this memory with the latest version from the manufacturer, or at least get it back to original factory code.
The program to reflash UEFI memory won't start the process unless you have the machine plugged in, due to the risk of bricking the machine. The apparently unrelated problems with batteries and power now prevent you from restoring to factory code.
I'm sure this machine has a rootkit, and I strongly suspect I know which one.
Here's what I now think is going on:
https://arstechnica.com/information-tec ... ns-active/
https://asert.arbornetworks.com/lojax-fancy-since-2016/
It seems there were multiple versions of the rootkit using different C&C servers. It has been active since 2016. It is so hard to remove that one website suggests binning the motherboard.
All I need to do to get around this is to modify the reflashing program to ignore what the machine says about power. I can keep it plugged into my UPS, so I'm not worried about bricking it. Suggestions?
In any case, it is a hazard to users at present, so bricking it wouldn't be a great loss.
This is the nastiest malware I've ever dealt with.
LoJax rootkit
Throwing out ideas
Dell laptops have always been a bit of a nuisance with regard to making sure only their batteries and power adaptors are used so that when something is not quite right the laptop throws a fit and stops you doing what you want. Worth trying another power adaptor if you have one.
What model is the laptop? It may be possible to reprogramme the ROM with an external programmer
What model is the laptop? It may be possible to reprogramme the ROM with an external programmer
-
prehistoric
- Posts: 1744
- Joined: Tue 23 Oct 2007, 17:34
The laptop is a 2016 model with service tag 3Q3KKC2, but I have no interest in taking it apart to access the SPI flash memory. If I take it apart I'll use it for parts.
Using legacy boot isn't exactly a solution. Both UEFI and Legacy boot are performed by the same firmware, which we know is compromised. In fact UEFI is a separate OS which can access the Internet, change system settings, and basically do whatever it wants prior to booting into Windows or whatever. One of the clues to the nature of the problem was a lot of activity prior to booting the OS or after shutting down, but before power off.
The original LoJack modification to firmware was a security feature to find stolen laptops. Wiping the disk or changing it out would not stop the machine from reporting its new location, which might bring the police. Naturally, everyone assumed the bad actors would never know how to use this to install malware.
Now the problem may be solved. The first time I checked with Dell support I was told "your machine is out of warranty." Just minutes ago, when I turned it on again, it notified me of new updates from Dell. This one also warned me not to unplug it during update, but did not depend on the corrupted system to tell it about battery and power adapter.
I'll suspect Dell was dealing with some very negative publicity.
It has now installed the update, and rebooted. We'll see what happens after I run some new tests.
Using legacy boot isn't exactly a solution. Both UEFI and Legacy boot are performed by the same firmware, which we know is compromised. In fact UEFI is a separate OS which can access the Internet, change system settings, and basically do whatever it wants prior to booting into Windows or whatever. One of the clues to the nature of the problem was a lot of activity prior to booting the OS or after shutting down, but before power off.
The original LoJack modification to firmware was a security feature to find stolen laptops. Wiping the disk or changing it out would not stop the machine from reporting its new location, which might bring the police. Naturally, everyone assumed the bad actors would never know how to use this to install malware.
Now the problem may be solved. The first time I checked with Dell support I was told "your machine is out of warranty." Just minutes ago, when I turned it on again, it notified me of new updates from Dell. This one also warned me not to unplug it during update, but did not depend on the corrupted system to tell it about battery and power adapter.
I'll suspect Dell was dealing with some very negative publicity.
It has now installed the update, and rebooted. We'll see what happens after I run some new tests.
This is interesting, I have a 2016 Dell laptop, it's running fine at present, mainly running various puppies, occasionally booting into Windows 10. I did a check for a new BIOS version last weekend. There is a new version (A14) which i downloaded but it wouldn't install. I always run at home plugged in, so it should have run. It starts to run but just reboots and doesn't update the flash. The last time I flashed to the version it has at the moment (A13), it flashed without issue.
I'll have to check those articles you posted.
Edit: So I just tried again, this time I copied the update file 3350A14.exe to the root of the USB flash drive I use with my puppy installations.
I pressed F12 and selected the BIOS update, I selected the update file. I get the normal warnings that it is going to flash. I click the accept and then it just goes to a black screen, then reboots without updating.
I'll have to check those articles you posted.
Edit: So I just tried again, this time I copied the update file 3350A14.exe to the root of the USB flash drive I use with my puppy installations.
I pressed F12 and selected the BIOS update, I selected the update file. I get the normal warnings that it is going to flash. I click the accept and then it just goes to a black screen, then reboots without updating.
-
prehistoric
- Posts: 1744
- Joined: Tue 23 Oct 2007, 17:34
@Terry H,
See if you can install the free trial version of ESET security under Windows, they have been detecting LoJax for several months. I think the failed update is caused by a false indication that the computer has no charged battery or official power adapter. The safe option to avoid bricking the machine is to avoid flashing the memory if you don't have reliable power. It is also possible there are multiple levels of compromise to prevent you from removing the malware.
You can also check in Windows under system information to tell if it recognizes the battery and power adapter.
Like I said this one is really nasty.
See if you can install the free trial version of ESET security under Windows, they have been detecting LoJax for several months. I think the failed update is caused by a false indication that the computer has no charged battery or official power adapter. The safe option to avoid bricking the machine is to avoid flashing the memory if you don't have reliable power. It is also possible there are multiple levels of compromise to prevent you from removing the malware.
You can also check in Windows under system information to tell if it recognizes the battery and power adapter.
Like I said this one is really nasty.
Unfortunately too late for me . It appears I've bricked the laptop. It won't turn on at all now, just the power light comes on, screen is black no backlight. Doesn't appear to be running POST. I'm not sure what I'm going to do at present.prehistoric wrote:@Terry H,
The safe option to avoid bricking the machine is to avoid flashing the memory if you don't have reliable power. It is also possible there are multiple levels of compromise to prevent you from removing the malware.
-
prehistoric
- Posts: 1744
- Joined: Tue 23 Oct 2007, 17:34
@Terry H,
Ouch! You may need to replace the chip that holds the firmware. You might check on swapping the entire motherboard, if you can't find anyone who will do that repair. Bad flashes are known problems, and the main thing discouraging repairs is the limited value of the resulting repaired machine.
You could get the chip from a broken machine which has not had the UEFI code corrupted. Unfortunately it is really hard to get at the part you need to replace.
There may be some real hardware gurus on this forum who can tell you more about doing this on your specific machine.
Here's what I think started the problems.
This assumes you have an Intel CPU. This is the first time I've run into a complete failure to power on at all. There are some simple options. On desktops it is standard to use the jumper on the motherboard to clear non-volatile RAM. On laptops you can remove the coin-cell battery to force the settings to clear. I'm afraid most of your easy options like that don't apply, as in this article.
I should have warned you to use a standalone BIOS/UEFI flashing program instead of trying to flash from a corrupted windows system. It is entirely possible for an attacker to deliberately brick the machine if you are about to remove their code from the UEFI. I don't know that this happened, because there are plenty of things that can go wrong without malicious intent.
-----
Incidentally, if you haven't had any experience with SPI and BMS you may dismiss the chips involved as different components without enough leads to hold that memory. There was a (strenuously-disputed) Bloomberg article on a claimed hardware hack to SuperMicro servers using a chip smaller than a grain of rice with two pins. Such active components do exist, and it would be possible to disguise them as passive components like resistors or capacitors, the question is whether or not such a hack was attempted, and, if so, by whom.
-----
The machine that alerted me to that problem still runs -- slowly. I was able to install ESET software and run a scan. So far it has found 15 "threats" of which I would count a handful as serious problems.
The owner of this machine bought a new laptop when the old one was becoming slow, then transferred the system she already had to avoid having to configure a new system from scratch. She brought the previous compromises with it. She never ran a full malware scan.
I've seen this scenario before.
Ouch! You may need to replace the chip that holds the firmware. You might check on swapping the entire motherboard, if you can't find anyone who will do that repair. Bad flashes are known problems, and the main thing discouraging repairs is the limited value of the resulting repaired machine.
You could get the chip from a broken machine which has not had the UEFI code corrupted. Unfortunately it is really hard to get at the part you need to replace.
There may be some real hardware gurus on this forum who can tell you more about doing this on your specific machine.
Here's what I think started the problems.
This assumes you have an Intel CPU. This is the first time I've run into a complete failure to power on at all. There are some simple options. On desktops it is standard to use the jumper on the motherboard to clear non-volatile RAM. On laptops you can remove the coin-cell battery to force the settings to clear. I'm afraid most of your easy options like that don't apply, as in this article.
I should have warned you to use a standalone BIOS/UEFI flashing program instead of trying to flash from a corrupted windows system. It is entirely possible for an attacker to deliberately brick the machine if you are about to remove their code from the UEFI. I don't know that this happened, because there are plenty of things that can go wrong without malicious intent.
-----
Incidentally, if you haven't had any experience with SPI and BMS you may dismiss the chips involved as different components without enough leads to hold that memory. There was a (strenuously-disputed) Bloomberg article on a claimed hardware hack to SuperMicro servers using a chip smaller than a grain of rice with two pins. Such active components do exist, and it would be possible to disguise them as passive components like resistors or capacitors, the question is whether or not such a hack was attempted, and, if so, by whom.
-----
The machine that alerted me to that problem still runs -- slowly. I was able to install ESET software and run a scan. So far it has found 15 "threats" of which I would count a handful as serious problems.
The owner of this machine bought a new laptop when the old one was becoming slow, then transferred the system she already had to avoid having to configure a new system from scratch. She brought the previous compromises with it. She never ran a full malware scan.
I've seen this scenario before.
Thanks prehistoric, I tried disconnecting the RTC Battery already, didn't do anything. Beyond that, it is a bit too daunting for me. If it was just disassembly, and replace I could do it, but I'm not going to tackle the chip. The laptop is just 3 years old (April 2016), it has core i3-5005U CPU, so depending on cost for repair, I think it's worth attempting to get it repaired.
I am waiting on a return call from a local computer repair store to check what they can do.
I am waiting on a return call from a local computer repair store to check what they can do.
-
prehistoric
- Posts: 1744
- Joined: Tue 23 Oct 2007, 17:34
That sounds like the exact same laptop I was handed to fix.
You really should check for people on-line who might know the problem and how to fix it quickly. I've sent a machine with a well-identified problem all the way to New York City for repair because that was cheaper, even with shipping, than the local repairman. That situation may have changed, now that we have more choices locally.
The group in NYC had a CNC machine programmed to replace the chip we knew was bad. Instead of removing the motherboard, which would have given an even lower price, I opted to remove all the normal parts like battery, hard drive, DVD and memory, and have them remove the motherboard from the case. Many local shops actually use some similar facility for tricky motherboard repairs, rather than do it themselves and risk trashing the laptop.
Now, back to LoJax. I'm still not 100% certain the problem I found was the official LoJax exploit. What I have verified is that the BIOS/UEFI checksum was not right, and that the standard UEFI flash utility that ran under Windows would not flash it. I also confirmed that the system had multiple pieces of malware in the Windows system. I know there is something wrong with UEFI because Linux Mint tells me there is a problem in firmware when I boot Linux Mint into live mode from a DVD. This is completely independent of Windows.
This tells me the firmware is corrupted in some way, which I was already pretty sure of since the machine stopped recognizing both Dell batteries and Dell power adapters at the same time, i.e. it was not a simple problem with a battery contact.
What would have been safer for your situation would have been downloading a stand-alone flash program and the new firmware to a USB flash drive and booting that via the F12 key. Unfortunately, I don't yet know for sure if Dell has a version that will ignore the apparently missing battery. It is possible to flash the BIOS if the power adapter is missing or not recognized, by running off an adequately charged battery. We are back to a Catch 22.
(There are ways around the problem for other laptops, so there is probably one for Dell.)
If that was a deliberate action by whatever program corrupted the firmware it was a fiendish way to prevent users from removing it.
Here's a way to flash a Dell BIOS/UEFI when the battery is not recognized that I'm going to try now that I've cleaned the main system. Wish me luck!
(There are also ways to do this from a USB drive, if you don't trust the sytem.)
You really should check for people on-line who might know the problem and how to fix it quickly. I've sent a machine with a well-identified problem all the way to New York City for repair because that was cheaper, even with shipping, than the local repairman. That situation may have changed, now that we have more choices locally.
The group in NYC had a CNC machine programmed to replace the chip we knew was bad. Instead of removing the motherboard, which would have given an even lower price, I opted to remove all the normal parts like battery, hard drive, DVD and memory, and have them remove the motherboard from the case. Many local shops actually use some similar facility for tricky motherboard repairs, rather than do it themselves and risk trashing the laptop.
Now, back to LoJax. I'm still not 100% certain the problem I found was the official LoJax exploit. What I have verified is that the BIOS/UEFI checksum was not right, and that the standard UEFI flash utility that ran under Windows would not flash it. I also confirmed that the system had multiple pieces of malware in the Windows system. I know there is something wrong with UEFI because Linux Mint tells me there is a problem in firmware when I boot Linux Mint into live mode from a DVD. This is completely independent of Windows.
This tells me the firmware is corrupted in some way, which I was already pretty sure of since the machine stopped recognizing both Dell batteries and Dell power adapters at the same time, i.e. it was not a simple problem with a battery contact.
What would have been safer for your situation would have been downloading a stand-alone flash program and the new firmware to a USB flash drive and booting that via the F12 key. Unfortunately, I don't yet know for sure if Dell has a version that will ignore the apparently missing battery. It is possible to flash the BIOS if the power adapter is missing or not recognized, by running off an adequately charged battery. We are back to a Catch 22.
(There are ways around the problem for other laptops, so there is probably one for Dell.)
If that was a deliberate action by whatever program corrupted the firmware it was a fiendish way to prevent users from removing it.
Here's a way to flash a Dell BIOS/UEFI when the battery is not recognized that I'm going to try now that I've cleaned the main system. Wish me luck!
(There are also ways to do this from a USB drive, if you don't trust the sytem.)
-
prehistoric
- Posts: 1744
- Joined: Tue 23 Oct 2007, 17:34
Some success. I've succeeded in flashing the UEFI/BIOS and now pass the scan of the UEFI. Still have the problem of not recognizing a Dell battery.
This machine has been running Windows 10, which naturally means you can't access the command prompt you need to carry out the instructions in that article I linked. Here's how to reach this on W10. There has also been a change to the Powershell so that this does not accept commands to execute programs in the current directory unless explicitly told. Linux/Unix users will understand the problem. Here is the command I used to execute a forced UEFI update on a Dell Inspiron5459:
I ran this from inside the Downloads directory for the account I was using (there is more than one such directory for multiple users) using a "run as administrator" powershell.
Now, I just have to figure out why it still doesn't recognize a Dell battery.
A new system scan is underway.
Added: while poking around in the file system I got a clue as to how far back the system ported to this laptop was compromised. One folder is labeled "old floppy disk data".
This machine has been running Windows 10, which naturally means you can't access the command prompt you need to carry out the instructions in that article I linked. Here's how to reach this on W10. There has also been a change to the Powershell so that this does not accept commands to execute programs in the current directory unless explicitly told. Linux/Unix users will understand the problem. Here is the command I used to execute a forced UEFI update on a Dell Inspiron5459:
Code: Select all
.\Inspiron5459_130 /forceitNow, I just have to figure out why it still doesn't recognize a Dell battery.
A new system scan is underway.
Added: while poking around in the file system I got a clue as to how far back the system ported to this laptop was compromised. One folder is labeled "old floppy disk data".
-
prehistoric
- Posts: 1744
- Joined: Tue 23 Oct 2007, 17:34
Also a hardware fault
Debugging tip: Don't always assume you are dealing with a single fault.
The assumption that a broken system has only a single fault is only made for convenience, and is most likely to be true in systems that are very carefully maintained, so that you notice quickly when onset of a problem occurs. If you don't notice the initial fault you are likely to get a cascade of failures.
This machine not only had multiple malware problems, it also had a hardware problem. My personal bias is toward finding malware, because I almost always find it when people hand me machines that will still boot, but have problems.
When I questioned the owner about the origin of the problem, she described repeatedly trying to get the battery to charge, and swapping batteries. It now looks like repeatedly putting batteries in damaged the connection between the battery connector and the circuit board. This should be easy to touch up with a soldering iron -- once you get at it.
I'm guessing, based on file dates, that the software problems led to batteries not being recognized or charging, and trying different batteries repeatedly damaged the connection to the motherboard. In every other respect she insists the machine has been barely used at all.
Anyone who wants to question if a 3-year old Dell laptop is worth this effort can do so. If I were charging for my time, we would already be over budget.
The assumption that a broken system has only a single fault is only made for convenience, and is most likely to be true in systems that are very carefully maintained, so that you notice quickly when onset of a problem occurs. If you don't notice the initial fault you are likely to get a cascade of failures.
This machine not only had multiple malware problems, it also had a hardware problem. My personal bias is toward finding malware, because I almost always find it when people hand me machines that will still boot, but have problems.
When I questioned the owner about the origin of the problem, she described repeatedly trying to get the battery to charge, and swapping batteries. It now looks like repeatedly putting batteries in damaged the connection between the battery connector and the circuit board. This should be easy to touch up with a soldering iron -- once you get at it.
I'm guessing, based on file dates, that the software problems led to batteries not being recognized or charging, and trying different batteries repeatedly damaged the connection to the motherboard. In every other respect she insists the machine has been barely used at all.
Anyone who wants to question if a 3-year old Dell laptop is worth this effort can do so. If I were charging for my time, we would already be over budget.
-
prehistoric
- Posts: 1744
- Joined: Tue 23 Oct 2007, 17:34
@TerryH,
While repairmen are inside your laptop replacing the chip make sure they check the solder connections on the battery connector and the one for the power adapter. These are easy to touch up with a soldering iron, once you get to them.
Rant: I keep finding laptops with soldered connections being used to handle mechanical forces between either the battery or power plug and the circuit board. This is just plain bad design.
The only good aspect is that someone may hand you a dead laptop for free that is actually pretty easy to repair.
While repairmen are inside your laptop replacing the chip make sure they check the solder connections on the battery connector and the one for the power adapter. These are easy to touch up with a soldering iron, once you get to them.
Rant: I keep finding laptops with soldered connections being used to handle mechanical forces between either the battery or power plug and the circuit board. This is just plain bad design.
The only good aspect is that someone may hand you a dead laptop for free that is actually pretty easy to repair.
Both my daily use laptops were dirt cheap because they didn't work properly or overheated.
Both now have faster CPUs and maximum memory after having cleaned a few contacts and sorted out bad BIOS settings.
Both had the CPU set to run at 100% ALL the time rather than on demand.
Now it is badly designed WEB sites that cause overheating.
Both now have faster CPUs and maximum memory after having cleaned a few contacts and sorted out bad BIOS settings.
Both had the CPU set to run at 100% ALL the time rather than on demand.
Now it is badly designed WEB sites that cause overheating.
"Just think of it as leaving early to avoid the rush" - T Pratchett
" I tried disconnecting the RTC Battery alread "
Look probably won't work but...
Disconnect the RTC Battery
Take out laptop battery, memory, hard drives.
Push power on button for over 30 seconds....minimum.
I beat a Dell password on my wife'sDell laptop that way.
No other method worked to get it going.
Hard drive stayed decrypted but at least everything else worked
after that.
Worth a go.
Chris.
Look probably won't work but...
Disconnect the RTC Battery
Take out laptop battery, memory, hard drives.
Push power on button for over 30 seconds....minimum.
I beat a Dell password on my wife'sDell laptop that way.
No other method worked to get it going.
Hard drive stayed decrypted but at least everything else worked
after that.
Worth a go.
Chris.
-
prehistoric
- Posts: 1744
- Joined: Tue 23 Oct 2007, 17:34
It would be nice if cthisbear's idea worked, but I'm afraid you are dealing with a bad flash of the SPI chip. There are clip-on programmers that might be used, though I don't have experience with them, particularly for laptops. I am not too bothered by buying a small cheap device to reprogram some system-on-a-chip devices as used in stuff like 3D printers that lack a bootloader. Knowledgeable repair people probably have better devices to install Dell firmware.
The company in the U.S. with the best knowledge of repairing Dell laptops is probably PartsPeople in Austin, TX, where Dell began.
I ran into a problem when I tried to order a new board for the battery connector for the machine I've been repairing. Their automated system insists that the service code for this machine says it is a 15" laptop with a different battery and connector board. Since I have measured the screen at over 16.5" I'm pretty sure this is a 17" laptop. I believe this is one of many examples of Dell creating an interim design with parts from two different models. I'm going to have to take it almost completely apart to get a picture and part number for the battery connector board I need. While I'm in there, I'll check on the power connector board, which I suspect just went out. Both are relatively cheap, the problem is the labor required to replace them.
I didn't want to do a complete teardown until I am ready to replace parts. I expect most of you have experience with laptops that never went back together after being left apart too long, because you forgot some detail of the process. I put the parts I take off in separate compartments of a plastic case used for fishing tackle so that I will replace them in the right order when I put it back together. I've found some examples of laptops where there are tiny differences in screws that mean you have to be careful about which go in which holes. Other people place the screws on pieces of tape to keep them in order.
The company in the U.S. with the best knowledge of repairing Dell laptops is probably PartsPeople in Austin, TX, where Dell began.
I ran into a problem when I tried to order a new board for the battery connector for the machine I've been repairing. Their automated system insists that the service code for this machine says it is a 15" laptop with a different battery and connector board. Since I have measured the screen at over 16.5" I'm pretty sure this is a 17" laptop. I believe this is one of many examples of Dell creating an interim design with parts from two different models. I'm going to have to take it almost completely apart to get a picture and part number for the battery connector board I need. While I'm in there, I'll check on the power connector board, which I suspect just went out. Both are relatively cheap, the problem is the labor required to replace them.
I didn't want to do a complete teardown until I am ready to replace parts. I expect most of you have experience with laptops that never went back together after being left apart too long, because you forgot some detail of the process. I put the parts I take off in separate compartments of a plastic case used for fishing tackle so that I will replace them in the right order when I put it back together. I've found some examples of laptops where there are tiny differences in screws that mean you have to be careful about which go in which holes. Other people place the screws on pieces of tape to keep them in order.