Foire Aux Questions Delixir.com

Several days ago I've got a message from our hosting company. It stated that one of our IP addresses was used in massive UDP DoS attack. Since that IP address does not belong to any VM and is actually assigned to VMware ESXi host itself, I started my investigation. As it turned out:

• VMware ESXi 4.1 was installed by hosting company (probably with default settings)
• ESXi is based on Linux and contains ntpd 4.2.4p6
• By some unknown reason, default ntpd settings allow anyone to query daemon status using "monlist" command
• Monlist is a remote command that sends the requester a list of the last 600 clients who have connected to that NTP server
• NTP uses UDP as an underlying protocol, therefore source address can be easily spoofed

By scanning internet address space, attackers gathered a list of NTP servers which allow querying their status. Then these servers were flooded with NTP monlist command sent from spoofed source address. This spoofed address belongs to the victim being attacked. Each NTP server responds with a list of the last 600 clients which is significantly larger than original request (one 40-byte-long request generates 18252 bytes worth of response traffic). This leads to significant amount of UDP traffic which can be directed by attacker to any destination.

How to verify if your server is vulnerable

$ ntpdc -c monlist 1.2.3.4

remote address          port local address      count m ver rstr avgint  lstint
===============================================================================
example.com            64059 1.2.3.4               52 7 2    590      0       0
host.example.com         123 1.2.3.4                1 4 4    590      0      57
....

How to fix

1. Get access to local or remote (ssh) ESXi console

Launch VMware vSphere Client and connect to your ESXi server. On the "Configuration" tab open "Security Profile" and start "Local Tech Support" or "Remote Tech Support (SSH)" service. To use local mode you need either physical server access or remote KVM. For remote mode, SSH client is sufficient.

2. Change NTP settings

Edit /etc/ntp.conf and add "noquery" to a line which contains "restrict default":

restrict default kod nomodify notrap noquery nopeer
restrict 127.0.0.1
server pool.ntp.org
driftfile /etc/ntp.drift

Do not forget to restart NTP daemon:

$ /etc/init.d/ntpd restart

And make sure it is running after restart:

$ ps ax | grep ntp

1234 1234 ntpd /sbin/ntpd

3. Verify that monlist command no longer works

$ ntpdc -c monlist 1.2.3.4

1.2.3.4: timed out, nothing received
***Request timed out

Other possible countermeasures

• Upgrade NTP to the latest version which do not contain "monlist" command (this is problematic on ESXi)
• Add ACL to inbound firewall or NTPD itself to allow only trusted clients
• Rate-limit responses to individual IP addresses

In an ideal world every provider should filter spoofed packets at their source (spoofed address does not belong to provider's networks). This is described in detail in BCP38

Source : http://ar0.me/blog/en/posts/2014/01/howto-prevent-malicious-usage-of-vmware-esxi-in-ntp-reflection-attacks.html