In the past Microsoft's business plan seemed to be based around the idea of "bigger and better". As each OS was released it was generally more stable than the previous version and had more capabilities - I understand that ME may have been an exception. For those that treat their PCs as work tools this is a good reason to upgrade.
Unfortunately, as far as Microsoft was concerned, XP was so successful in those aims that for a lot of people there was no longer a reason to upgrade. Not only was XP unlikely to crash but, by the time that Vista came out, there were so many free utilities to allow you to customise XP that it did everything that was needed and all that Vista seemed to offer was some graphical tweaks, and not everybody lives for eye candy, and DX 10 for gaming.
There were also two customer issues that Microsoft suffered from with XP, namely mass beta testing and specification problems. Anyone who got hold of XP early and had to wait for patches and Service Packs to resolve problems and complete the operating system, as it should have been on release, were probably going to be reluctant to make the same mistake twice. Also, those people who believed the official line and bought systems with 256 Mb RAM and found that there was no way that the minimum RAM was sufficient to use their machines as they expected were probably not going to invest in a Vista system that met minimum specs without user feedback - as I recall Vista was similar to XP in that doubling the minimum memory was necessary to get a workable system.
Then there is the life span of a Microsoft OS, both in terms of the support that is provided and also it no longer being the latest one. Vista was released in 2006 and Windows 7 in 2009. Based on the ease and cost of obtaining replacement parts for a computer, it's life is considered by some to be five years. Anyone who bought a XP machine after 2004 could be reasonably expected to avoid Vista if XP did what was required. The moment that Microsoft announced that there was new OS in production people would try to keep XP going long enough to avoid Vista as there would be years less updates for it.
You can also factor in with the above the increase in processor power and RAM needed with each new version of Windows - the price of bigger and better. Upgrading just the OS was not something that could be done by everyone. For those that needed a major upgrade it was going to involve an investment in a new machine rather than just an installation disk, and if it ain't broke, why replace it.