Posted 11 September 2008 - 10:24 AM
Currently one of our customer sites seems to be experiencing random file corruptions.
System is running XPSP3 and all critical updates, with AVG 8.0 (commercial) antivirus, Skype on a US robotics USB phone, and a bunch of our own software, some of which uses Python (version 2.4.0). The system is using a RAID-1 array, Adaptec 1200A controller, and has not shown up any fault events, so I doubt it's simple hardware corruption. System is remote from our office (in the US), and we control it using GoToMyPC. The system is largely identical to 30 other sites worldwide which have not had the problem (or at least not in a visible way).
We have had two principal occurrences on this site, both happening at or shortly after an automatic Windows Update. Not sure whether that's relevant, though, as one of the file corruptions happened yesterday afternoon, after having fixed the Python installation corrupted the previous night.
Comparing the corrupted files with the originals, the symptoms are the affected files are corrupted by the 4 bytes at 0x....0-3 and 0x....8-B are replicated to 0x....4-7 and 0x....C-F respectively. Corruption starts at file offset 0xYY0070 (where YY has been 00, 01, 0B, 1E) and continues to 0xYYFFFF, every alternate 4bytes showing the corruption. I've seen this now in various files:
(from python run-time) win32api.pyd, win32gui.pyd, python-2.4.msi, msvcr71.dll (suspected - didn't analyse the corrupted file)
(from system32) PortableDeviceApi.dll, msjet40.dll, ipsecsvc.dll
We did have a single event on another site, where the 4bytes at the end of a block were replicated to the subsequent 4 bytes (i.e. bytes 0xYYYY00-03 were corrputed). Unlike the more recent events, this corruption was only sparse within the affected files, with no obvious pattern. I originally thought this may have been on 256byte boundaries, but then found some on 32byte boundaries. This affected about 30 files in System32 and a dozen or so of our own executable files. As it completely prevented system startup, we had to fix that one by complete re-install of XP.
I don't think the files themselves are significant. I did think for a while that they may have been files running at the time of the failure (particularly from the other site evidence), but that wouldn't have applied to the corrupted msi file.
Anyone heard of anything like this? AVG shows no infections, and about 25 cookie warnings. If at all possible we need to be able to diagnose this using the GoToMyPC connection, as the customer is not particularly PC savvy, and otherwise it's a plane ride for one of our US engineers.