Jump to content


 


Register a free account to unlock additional features at BleepingComputer.com
Welcome to BleepingComputer, a free community where people like yourself come together to discuss and learn how to use their computers. Using the site is easy and fun. As a guest, you can browse and view the various discussions in the forums, but can not create a new topic or reply to an existing one unless you are logged in. Other benefits of registering an account are subscribing to topics and forums, creating a blog, and having no ads shown anywhere on the site.


Click here to Register a free account now! or read our Welcome Guide to learn how to use this site.

Photo

Progressive Server 2003 Infection with Network/Workstation Involvement


  • Please log in to reply
24 replies to this topic

#1 User: LSM

User: LSM

  • Members
  • 31 posts
  • OFFLINE
  •  
  • Local time:04:11 AM

Posted 11 January 2018 - 01:24 AM

Summary: I've tried to mitigate an ongoing and progressive infection on a Windows 2003 Server with network involvement without success.   Scans with various anti-virus tools clearly point to virus activity along with Autoruns and Process Explorer info.  At least 1 workstation and a remote PC have also been affected.  After initial attempts to remove the virus were made some of the anti-virus tools used on the 2003 Server were deleted or uninstalled.  Today, I noticed similar activity on one of the workstations with problems, in which recently installed anti-virus tools, were partially removed or deleted.  Each time I think the machines are clean as everything runs smoothly, the threat/attack reappears and becomes more difficult to detect.  At this point, I've done all I know to do and need trained, professional help.  I probably needed trained, professional help before I got started. 
 
Server Setup: Windows 2003 SP2 running Exchange 2003 and SQL 2000 with Terminal Services and file sharing.
 
Note: I am aware of the security concerns of continuing to run Server 2003 and have had plans to migrate away from it completely but internal issues have prevent the change. I found out today that our primary but unsupported legacy application running on SQL 2000  can be migrated to SQL 2008, which was previously not thought possible. I am also aware that our current server design is very poor and way outside of best practices. 
 
Problem: Initially, we began experiencing network latency with an inability to connect to server resources.  I also began to notice strange intermittent network activity from my own workstation to the point I thought the NIC was failing. Using Process Explorer I discovered a Server 2003 process, w3wp.exe, with Monero miner involvement as it was connected to atlanta01.monero.hashvault.  Within this same period, e-mails to Gmail also started to bounce back and then other domains as well, resulting in being blacklisted by a few services. A review of Exchange turned up a lot of spam being sent out with high utilization from store.exe.  Also during this time, an at-home remote RDP/Terminal Services user reported "virus" activity.  I later figured out that connections by this user were also causing stop errors involving rdpdr,sys - SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (0x0000007e).  This user has since avoided any RDP connections.  (I don't want to imply this user as the source of the infection, but only at least as a symptom of the problem.)  Another in-office user, who connects to the Server through RDP and files shares started experiencing latency,  Ultimately, this machine would lock up trying to access Server resources.  Scans of this machine turned up Hamza.vbs using wscript.exe.  Initial attempts to remove this seemed successful, but it also has reappeared.  This is the same machine that also had anti-virus tools partially removed or deleted.         
 
As part of the mitigation process, I installed a trial of ESET File Security for Windows Server as it was one of the few I found which still supported Windows 2003.  Initially, it found infections but was later partially deleted or uninstalled at some point.  It also indicated Windows Update was not available and provided a list of recommended updates.  Attempts to fix Windows Update failed, so I installed Secunia PSI in an attempt to help because the list from ESET was no longer available.  PSI, however, could not connect to its update server update and run.  Surprisingly, after some continued hang ups and restarts, Windows Update "magically" started to run and I was able to install all suggested updates.  After reinstalling/repairing ESET, it has continued to find w3wp.exe infections every couple of days.  However, now I'm seeing high utilization from wuaueng.dll and resources/access to the Server is intermittent or hangs up and admin RDP sessions hang.  I've seen from other posts that wuaueng.dll can be problematic.  I can't help but to think the attack/hacker fixed the Windows Update issue to transition the attach to wuaueng.dll.  Virustotal is showing this hash clean.  
 
With the nature of our setup and the seemingly sophisticated attack, I'm not sure if this is the right place for assistance, but any removal support or guidance would be appreciated.  Thank you in advance for your time.  


BC AdBot (Login to Remove)

 


#2 SleepyDude

SleepyDude

  • Malware Response Team
  • 3,035 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Portugal
  • Local time:10:11 AM

Posted 11 January 2018 - 04:46 AM

Hi,

 

Besides the updates offered by Windows Update there are other updates released by Microsoft out of Server 2003 Support to patch several SMB security that needs to be installed manually. Did you install them?


• Please do not PM me asking for support. Post on the forums instead it will increases the chances of getting help for your problem by one of us.
• Posts in the Malware section that are not replied to within 4 days will be closed. PM me or a moderator to reactivate.
• Please post your final results, good or bad. We like to know! Thank you!

 
Proud graduate of GeekU and member of UNITE
___
Rui

 
 


#3 User: LSM

User: LSM
  • Topic Starter

  • Members
  • 31 posts
  • OFFLINE
  •  
  • Local time:04:11 AM

Posted 11 January 2018 - 10:48 AM

Thanks for the reply, SleepyDude.  I'm sure there are additional updates needed.  I was hoping to check for others through Windows Update, but I received an error to add update to Internet Options-->Trusted Sites.  Adding the recommended sites to the Trusted Sites zone didn't initially work.  I then added http://*.microsoft.com and https://*.microsoft.com.  This cleared the error but then Windows Update redirected to Microsoft Update.  I assume this is one in the same?

 

Now, I'm not sure Microsoft Update is actively working; perhaps it is.  It does indicate "Checking for the latest updates for your computer..." but I'm not sure if it is actively searching and working or simply hung.  Is there a way to check this?  Also, I assume Windows Update upgrading to Microsoft Update for Server 2003 is not a problem?

 

By the way, I was able to screen shot recommended updates by ESET FS.  I can compare the Microsoft Update history with what ESET FS recommended to compare and then download and install manually if needed.  Is there a more efficient way to determine what is still needed and installed?



#4 SleepyDude

SleepyDude

  • Malware Response Team
  • 3,035 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Portugal
  • Local time:10:11 AM

Posted 11 January 2018 - 11:20 AM

Hi,

 

Check your PM.


• Please do not PM me asking for support. Post on the forums instead it will increases the chances of getting help for your problem by one of us.
• Posts in the Malware section that are not replied to within 4 days will be closed. PM me or a moderator to reactivate.
• Please post your final results, good or bad. We like to know! Thank you!

 
Proud graduate of GeekU and member of UNITE
___
Rui

 
 


#5 SleepyDude

SleepyDude

  • Malware Response Team
  • 3,035 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Portugal
  • Local time:10:11 AM

Posted 11 January 2018 - 11:38 AM

 

Also, I came across this BC article.  Is RDP the primary culprit?  How effective would port change be?

 

Yes RDP is big risk specially if the server accepts connections from the outside and users can use any insecure password.

 

Changing the default RDP port to something else will help at least against automated scripts that only search for the default port and try to brute force login.


• Please do not PM me asking for support. Post on the forums instead it will increases the chances of getting help for your problem by one of us.
• Posts in the Malware section that are not replied to within 4 days will be closed. PM me or a moderator to reactivate.
• Please post your final results, good or bad. We like to know! Thank you!

 
Proud graduate of GeekU and member of UNITE
___
Rui

 
 


#6 User: LSM

User: LSM
  • Topic Starter

  • Members
  • 31 posts
  • OFFLINE
  •  
  • Local time:04:11 AM

Posted 12 January 2018 - 02:43 AM

1. I successfully ran the Windows Update package without any problems.  

 

2. Because there are so many Windows updates, I didn't get a chance to compare with the ESET list and don't anticipate having the time to do so.  I still don't know if Automatic Updates is working to check for any pending updates. Thoughts? 

 

3. I have identified a few Exchange 2003 updates that still need to be installed.  I will backup Exchange and then install these.

 

4. I continue to see wuaueng.dll running as a thread for svchost (C:\WINDOWS\System32\svchost.exe -k netsvcs).  I have been suspending this most recently because it seemed to significantly increase the page faults for the process, and I've suspected it may have been related to declining server performance as the day progressed.  This was noticed prior to recent updates.

 

5. I'm also seeing activity from process, store.exe (E:\Program Files\Exchsrvr\bin\store.exe) with an ESE.dll thread consistently at 5-6%.  Is this normal given no employee/user activity?  There are 20-30% interrupts as well..  This store.exe activity was noticed prior to recent updates as well.  I'm not seeing any spam in Exchange Message tracking, only a handful of legit mail.  



#7 SleepyDude

SleepyDude

  • Malware Response Team
  • 3,035 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Portugal
  • Local time:10:11 AM

Posted 12 January 2018 - 04:34 AM

Hi,

 

1 - Good

 

2 - I don't remember of problems with checking for updates on Server 2003 only on Windows 7 and Windows 8 due to the changes made by Microsoft because of Windows 10...

 

Check the Windows Event Viewer for errors and c:\windows\windowsupdate.log to see if Windows Update is working

 

3 - update all that you can

 

4 - wuaueng.dll is related to Windows Update if Windows is searching for updates its normal to see that component active and the svchost parent using memory and cpu. If you stop the Windows Update service this should stop immediately.

 

5 - I'm not familiarize with Exchange server... that store.exe file and ese.dll seems related to Exchange. Do you have Veeam Backup installed or other backup program compatible with Exchange?


• Please do not PM me asking for support. Post on the forums instead it will increases the chances of getting help for your problem by one of us.
• Posts in the Malware section that are not replied to within 4 days will be closed. PM me or a moderator to reactivate.
• Please post your final results, good or bad. We like to know! Thank you!

 
Proud graduate of GeekU and member of UNITE
___
Rui

 
 


#8 User: LSM

User: LSM
  • Topic Starter

  • Members
  • 31 posts
  • OFFLINE
  •  
  • Local time:04:11 AM

Posted 13 January 2018 - 02:52 PM

Hi,

 

1 - Good

 

2 - I don't remember of problems with checking for updates on Server 2003 only on Windows 7 and Windows 8 due to the changes made by Microsoft because of Windows 10...

 

Check the Windows Event Viewer for errors and c:\windows\windowsupdate.log to see if Windows Update is working

 

3 - update all that you can

 

4 - wuaueng.dll is related to Windows Update if Windows is searching for updates its normal to see that component active and the svchost parent using memory and cpu. If you stop the Windows Update service this should stop immediately.

 

5 - I'm not familiarize with Exchange server... that store.exe file and ese.dll seems related to Exchange. Do you have Veeam Backup installed or other backup program compatible with Exchange?

 

2. Ok.  The windowsupdate.log is definitely active as it is quite lengthy.  Any benefit for you to see?  If so, should I post here?

 

3. Will do.  Exchange updates are running now. 

 

4. I will check correlation to service actively searching for updates.  

 

5. Yes, Exchange is backed up using NTbackup to a network share.  It's relatively small.  What are your specific thoughts?  

 

FYI, server ran great yesterday without any noticeable issues.  The infected workstation was taken offline Thursday so there might be a direct correlation.  At this point, either the Windows Updates and/or taking the workstation offline have been positive.  What would you advise about the workstation at this point?



#9 SleepyDude

SleepyDude

  • Malware Response Team
  • 3,035 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Portugal
  • Local time:10:11 AM

Posted 13 January 2018 - 05:09 PM


2. Ok.  The windowsupdate.log is definitely active as it is quite lengthy.  Any benefit for you to see?  If so, should I post here?
 
3. Will do.  Exchange updates are running now. 
 
4. I will check correlation to service actively searching for updates.  
 
5. Yes, Exchange is backed up using NTbackup to a network share.  It's relatively small.  What are your specific thoughts?  
 
FYI, server ran great yesterday without any noticeable issues.  The infected workstation was taken offline Thursday so there might be a direct correlation.  At this point, either the Windows Updates and/or taking the workstation offline have been positive.  What would you advise about the workstation at this point?

 

2 - If you go to the end of the file and there are lines with the current time stamp then its probably ok. Not really only in case of errors.

 

5 - Just because searching for ESE.dll seems to indicate that is something that the backup programs must use to "communicate" with Exchange during backups. I suppose its normal to see it active.

 

Keep the workstations off-line until checked for malware and make sure that all Windows updates are installed. You should create a specific topic for each affected machine on this section or on the "Virus, Trojan, Spyware, and Malware Removal Logs" section of the forum.

 

There are several malware that move laterally in the network, some uses the credentials stored on the machine to access other machines, others will use SMB exploits to gain access to any machine on the network that doesn't have all the windows security updates.


• Please do not PM me asking for support. Post on the forums instead it will increases the chances of getting help for your problem by one of us.
• Posts in the Malware section that are not replied to within 4 days will be closed. PM me or a moderator to reactivate.
• Please post your final results, good or bad. We like to know! Thank you!

 
Proud graduate of GeekU and member of UNITE
___
Rui

 
 


#10 User: LSM

User: LSM
  • Topic Starter

  • Members
  • 31 posts
  • OFFLINE
  •  
  • Local time:04:11 AM

Posted 15 January 2018 - 10:22 AM

 


2. Ok.  The windowsupdate.log is definitely active as it is quite lengthy.  Any benefit for you to see?  If so, should I post here?
 
3. Will do.  Exchange updates are running now. 
 
4. I will check correlation to service actively searching for updates.  
 
5. Yes, Exchange is backed up using NTbackup to a network share.  It's relatively small.  What are your specific thoughts?  
 
FYI, server ran great yesterday without any noticeable issues.  The infected workstation was taken offline Thursday so there might be a direct correlation.  At this point, either the Windows Updates and/or taking the workstation offline have been positive.  What would you advise about the workstation at this point?

 

2 - If you go to the end of the file and there are lines with the current time stamp then its probably ok. Not really only in case of errors.

 

5 - Just because searching for ESE.dll seems to indicate that is something that the backup programs must use to "communicate" with Exchange during backups. I suppose its normal to see it active.

 

Keep the workstations off-line until checked for malware and make sure that all Windows updates are installed. You should create a specific topic for each affected machine on this section or on the "Virus, Trojan, Spyware, and Malware Removal Logs" section of the forum.

 

There are several malware that move laterally in the network, some uses the credentials stored on the machine to access other machines, others will use SMB exploits to gain access to any machine on the network that doesn't have all the windows security updates.

 

 

2. There were active time stamps thru Saturday (12/13).  I'm guessing its found all of them.  Each time I've run it though, it just hasn't stopped searching and never confirms the system is up to date.  I started it on Saturday, and it's still running as of this morning.  

 

5.  I see.  FYI, the backup for Exchange was not running at the time store.exe/ese.dll was running.

 

I will start a new thread for the other affected machine.  Once a machine has been updated, it still has to be properly cleaned, correct?



#11 SleepyDude

SleepyDude

  • Malware Response Team
  • 3,035 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Portugal
  • Local time:10:11 AM

Posted 15 January 2018 - 11:26 AM

Hi,

 

2 - Try to reset Internet Explorer. What is the IE version installed on the server?

 

5 - I have zero experience with Exchange so I can't comment on that, sorry...

 

In some cases the malware block the installation of updates, both things must be done...  Its probably better to keep then off-line or isolated on network until you clean them and update.

 

If you feel that the machines got hacked it's recommended to revise all the passwords.


• Please do not PM me asking for support. Post on the forums instead it will increases the chances of getting help for your problem by one of us.
• Posts in the Malware section that are not replied to within 4 days will be closed. PM me or a moderator to reactivate.
• Please post your final results, good or bad. We like to know! Thank you!

 
Proud graduate of GeekU and member of UNITE
___
Rui

 
 


#12 User: LSM

User: LSM
  • Topic Starter

  • Members
  • 31 posts
  • OFFLINE
  •  
  • Local time:04:11 AM

Posted 15 January 2018 - 12:56 PM

2

 

Hi,

 

2 - Try to reset Internet Explorer. What is the IE version installed on the server?

 

5 - I have zero experience with Exchange so I can't comment on that, sorry...

 

In some cases the malware block the installation of updates, both things must be done...  Its probably better to keep then off-line or isolated on network until you clean them and update.

 

If you feel that the machines got hacked it's recommended to revise all the passwords.

 

2. Version: IE 8.  I reset the the IE settings from Internet Options-->Advanced-->Reset but it didn't seem to change anything after restarting IE.  Also, is this at a user level or machine level?  Would a system restart typically be necessary?

 

5. I understand.  At this point I'm going to assume this activity is normal as I'm no longer seeing any abnormal spam through Exchange Message Tracking.  

 

Passwords: We will definitely reset passwords for workstations.  

 

Infected vs. Hacked - When I originally submitted this thread, I suspected a hack (i.e. direct human control) due to specific programs being partially deleted or uninstalled.  Now, I'm not so sure.  Based on what you know, would you say this was an infection or a hack?  What other symptoms or signs should I look for to distinguish between the two?    



#13 User: LSM

User: LSM
  • Topic Starter

  • Members
  • 31 posts
  • OFFLINE
  •  
  • Local time:04:11 AM

Posted 15 January 2018 - 01:00 PM

SMB Updates - With the SMB updates that were run, I know there are SMB 1.0 vs 2.0 compatibility issues between this 2003 server.  I'm trying to connect to a Server 2012 share from 2003, but unable to do so.  I can connect to the share from my Windows 7 workstation.  I can also ping the 2012 server from the 2003 machine.  



#14 User: LSM

User: LSM
  • Topic Starter

  • Members
  • 31 posts
  • OFFLINE
  •  
  • Local time:04:11 AM

Posted 15 January 2018 - 01:01 PM

Follow-up Scans / Cleanup - Would you recommend any follow-up scans or clean-up now that all Windows Updates are seemingly completed?



#15 SleepyDude

SleepyDude

  • Malware Response Team
  • 3,035 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:Portugal
  • Local time:10:11 AM

Posted 15 January 2018 - 05:07 PM

2. Version: IE 8.  I reset the the IE settings from Internet Options-->Advanced-->Reset but it didn't seem to change anything after restarting IE.  Also, is this at a user level or machine level?  Would a system restart typically be necessary?

 

The IE reset is at user level. Try changing the Windows Updates configuration to Never then restart the machine and manually search for Updates using the browser.

 

Infected vs. Hacked - When I originally submitted this thread, I suspected a hack (i.e. direct human control) due to specific programs being partially deleted or uninstalled.  Now, I'm not so sure.  Based on what you know, would you say this was an infection or a hack?  What other symptoms or signs should I look for to distinguish between the two?

 

It could be a not so easy task! On infected machines you will find usually several malware on a hacked machines it can have new user accounts and other system changes may have new stuff installed or not, many times they use normal programs to do bad things and that's why may not all detected by AV programs.

 

 

SMB Updates - With the SMB updates that were run, I know there are SMB 1.0 vs 2.0 compatibility issues between this 2003 server.  I'm trying to connect to a Server 2012 share from 2003, but unable to do so.  I can connect to the share from my Windows 7 workstation.  I can also ping the 2012 server from the 2003 machine.

 

AFAIK the updates fix the security problems without changing the SMB protocols in use.
 

Follow-up Scans / Cleanup - Would you recommend any follow-up scans or clean-up now that all Windows Updates are seemingly completed?

 

Don't know the AV program that you have I would scan with Malwarebytes and Eset On-line Scanner.


Edited by SleepyDude, 15 January 2018 - 05:09 PM.

• Please do not PM me asking for support. Post on the forums instead it will increases the chances of getting help for your problem by one of us.
• Posts in the Malware section that are not replied to within 4 days will be closed. PM me or a moderator to reactivate.
• Please post your final results, good or bad. We like to know! Thank you!

 
Proud graduate of GeekU and member of UNITE
___
Rui

 
 





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users