Is this you? As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.
Claim your profile
Articles by Marek Majkowski
Memcached Stats Amplification Scanner
## # This module requires Metasploit: https://metasploit.com/download # Current source: https://github.com/rapid7/metasploit-framework ## class MetasploitModule < Msf::Auxiliary include Msf::Auxiliary::Report include Msf::Exploit::Capture include Msf::Auxiliary::UDPScanner include Msf::Auxiliary::DRDoS def initialize super( 'Name' => 'Memcached Stats Amplification Scanner', 'Description' => %q( This module can be used to discover Memcached servers which expose the unrestricted UDP port 11211.
The Linux Crypto API for user applications
In this post we will explore Linux Crypto API for user applications and try to understand its pros and cons. The Linux Kernel Crypto API was introduced in October 2002. It was initially designed to satisfy internal needs, mostly for IPsec. However, in addition to the kernel itself, user space applications can benefit from it. If we apply the basic definition of an API to our case, we will have the kernel on one side and our application on the other.
Cloudflare servers don't own IPs anymore — so how do they connect to the Internet? | APNIC Blog
A lot of Cloudflare’s technology is well documented. For example, how we handle traffic between the eyeballs (clients) and our servers has been discussed many times on the Cloudflare blog: ‘A brief primer on anycast (2011)‘, ‘Load Balancing without Load Balancers (2013)‘, ‘Path MTU discovery in practice (2015)‘, ’Cloudflare’s edge load balancer (2020)‘, ‘How we fixed the BSD socket API (2022)‘.
Cloudflare servers don't own IPs anymore - so how do they connect to the Internet?
Get Started Free | Contact Sales: +1 (888) 274-3482 Subscribe to receive notifications of new posts: Subscription confirmed. Thank you for subscribing! Loading... A lot of Cloudflare's technology is well documented.
How to stop running out of ephemeral ports and start to love long-lived connections
Loading... Often programmers have assumptions that turn out, to their surprise, to be invalid. From my experience this happens a lot. Every API, technology or system can be abused beyond its limits and break in a miserable way. It's particularly interesting when basic things used everywhere fail. Recently we've reached such a breaking point in a ubiquitous part of Linux networking: establishing a network connection using the connect() system call.
Branch predictor: How many "if"s are too many? Including x86 and M1 benchmarks!
Some time ago I was looking at a hot section in our code and I saw this: if (debug) { log("..."); } This got me thinking. This code is in a performance critical loop and it looks like a waste - we never run with the "debug" flag enabled[1]. Is it ok to have if clauses that will basically never be run? Surely, there must be some performance cost to that... Just how bad is peppering the code with avoidable if statements?
The revenge of the listening sockets
Loading... Back in November we wrote a blog post about one latency spike. Today I'd like to share a continuation of that story. As it turns out, the misconfigured rmem setting wasn't the only source of added latency. It looked like Mr. Wolf hadn't finished his job. After adjusting the previously discussed rmem sysctl we continued monitoring our systems' latency. Among other things we measured ping times to our edge servers.
Kernel bypass
Loading... In two previous posts we've discussed how to receive 1M UDP packets per second and how to reduce the round trip time. We did the experiments on Linux and the performance was very good considering it's a general purpose operating system. Unfortunately the speed of vanilla Linux kernel networking is not sufficient for more specialized workloads. For example, here at CloudFlare, we are constantly dealing with large packet floods. Vanilla Linux can do only about 1M pps.
Unimog - Cloudflare’s edge load balancer
Loading... As the scale of Cloudflare’s edge network has grown, we sometimes reach the limits of parts of our architecture. About two years ago we realized that our existing solution for spreading load within our data centers could no longer meet our needs. We embarked on a project to deploy a Layer 4 Load Balancer, internally called Unimog, to improve the reliability and operational efficiency of our edge network. Unimog has now been deployed in production for over a year.
Sandboxing in Linux with zero lines of code
Loading... Modern Linux operating systems provide many tools to run code more securely. There are namespaces (the basic building blocks for containers), Linux Security Modules, Integrity Measurement Architecture etc. In this post we will review Linux seccomp and learn how to sandbox any (even a proprietary) application without writing a single line of code.
The real cause of large DDoS - IP Spoofing
Loading... A week ago we published a story about new amplification attacks using memcached protocol on UDP port 11211. A few things happened since then: Github announced it was a target of 1.3Tbps memcached attack. OVH and Arbor reported similar large attacks with the peak reported at 1.7Tbps. Let's take a deep breath and discuss why such large DDoS attacks are even possible on the modern internet.
Speeding up Linux disk encryption
Loading... Data encryption at rest is a must-have for any modern Internet company. Many companies, however, don't encrypt their disks, because they fear the potential performance penalty caused by encryption overhead. Encrypting data at rest is vital for Cloudflare with more than 200 data centres across the world.
A gentle introduction to Linux Kernel fuzzing
For some time I’ve wanted to play with coverage-guided fuzzing. Fuzzing is a powerful testing technique where an automated program feeds semi-random inputs to a tested program. The intention is to find such inputs that trigger bugs. Fuzzing is especially useful in finding memory corruption bugs in C or C++ programs. Image by Patrick Shannon CC BY 2.0 Normally it's recommended to pick a well known, but little explored, library that is heavy on parsing.
Cloudflare architecture and how BPF eats the world
Recently at Netdev 0x13, the Conference on Linux Networking in Prague, I gave a short talk titled "Linux at Cloudflare". The talk ended up being mostly about BPF. It seems, no matter the question - BPF is the answer. Here is a transcript of a slightly adjusted version of that talk. At Cloudflare we run Linux on our servers.
Every 7.8μs your computer’s memory has a hiccup
Modern DDR3 SDRAM. Source: BY-SA/4.0 by Kjerish During my recent visit to the Computer History Museum in Mountain View, I found myself staring at some ancient magnetic core memory. Source: BY-SA/3.0 by Konstantin Lanzet I promptly concluded I had absolutely no idea on how these things could ever work. I wondered if the rings rotate (they don't), and why each ring has three wires woven through it (and I still don’t understand exactly how these work).
The rise of multivector DDoS attacks
Loading... It's been a while since we last wrote about Layer 3/4 DDoS attacks on this blog. This is a good news - we've been quietly handling the daily onslaught of DDoS attacks. Since our last write-up, a handful of interesting L3/4 attacks have happened. Let's review them. Gigantic SYN In April, John tweeted about a gigantic 942Gbps SYN flood: It was a notable event for a couple of reasons. First, it was really large. Previously, we've seen only amplification / reflection attacks at terabit scale.
How to drop 10 million packets per second
Internally our DDoS mitigation team is sometimes called "the packet droppers". When other teams build exciting products to do smart things with the traffic that passes through our network, we take joy in discovering novel ways of discarding it. Being able to quickly discard packets is very important to withstand DDoS attacks. Dropping packets hitting our servers, as simple as it sounds, can be done on multiple layers. Each technique has its advantages and limitations.
Today we mitigated 1.1.1.1
On May 31, 2018 we had a 17 minute outage on our 1.1.1.1 resolver service; this was our doing and not the result of an attack. Cloudflare is protected from attacks by the Gatebot DDoS mitigation pipeline. Gatebot performs hundreds of mitigations a day, shielding our infrastructure and our customers from L3/L4 and L7 attacks.
Abusing Linux's firewall: the hack that allowed us to build Spectrum
Loading... Today we are introducing Spectrum: a new Cloudflare feature that brings DDoS protection, load balancing, and content acceleration to any TCP-based protocol. CC BY-SA 2.0 image by Staffan Vilcans Soon after we started building Spectrum, we hit a major technical obstacle: Spectrum requires us to accept connections on any valid TCP port, from 1 to 65535. On our Linux edge servers it's impossible to "accept inbound connections on any port number".
eBPF, Sockets, Hop Distance and manually writing eBPF assembly
A friend gave me an interesting task: extract IP TTL values from TCP connections established by a userspace program. This seemingly simple task quickly exploded into an epic Linux system programming hack. The result code is grossly over engineered, but boy, did we learn plenty in the process! You may wonder why she wanted to inspect the TTL packet field (formally known as "IP Time To Live (TTL)" in IPv4, or "Hop Count" in IPv6)?
Memcrashed - Major amplification attacks from UDP port 11211
Over last couple of days we've seen a big increase in an obscure amplification attack vector - using the memcached protocol, coming from UDP port 11211. CC BY-SA 2.0 image by David Trawin In the past, we have talked a lot about amplification attacks happening on the internet. Our most recent two blog posts on this subject were: SSDP amplifications crossing 100Gbps. Funnily enough, since then we were a target of an 196Gbps SSDP attack. General statistics about various amplification attacks we see.
SYN packet handling in the wild
Here at Cloudflare, we have a lot of experience of operating servers on the wild Internet. But we are always improving our mastery of this black art. On this very blog we have touched on multiple dark corners of the Internet protocols: like understanding FIN-WAIT-2 or receive buffer tuning. One subject hasn't had enough attention though - SYN floods. We use Linux and it turns out that SYN packet handling in Linux is truly complex. In this post we'll shine some light on this subject.
Why does one NGINX worker take all the load?
Scaling up TCP servers is usually straightforward. Most deployments start by using a single process setup. When the need arises more worker processes are added. This is a scalability model for many applications, including HTTP servers like Apache, NGINX or Lighttpd. Increasing the number of worker processes is a great way to overcome a single CPU core bottleneck, but opens a whole new set of problems.
The Porcupine Attack: investigating millions of junk requests
We extensively monitor our network and use multiple systems that give us visibility including external monitoring and internal alerts when things go wrong. One of the most useful systems is Grafana that allows us to quickly create arbitrary dashboards. And a heavy user of Grafana we are: at last count we had 645 different Grafana dashboards configured in our system! This post is not about our Grafana systems though.
How and why the leap second affected Cloudflare DNS
At midnight UTC on New Year’s Day, deep inside Cloudflare’s custom RRDNS software, a number went negative when it should always have been, at worst, zero. A little later this negative value caused RRDNS to panic. This panic was caught using the recover feature of the Go language. The net effect was that some DNS resolutions to some Cloudflare managed web properties failed.
Say Cheese: a snapshot of the massive DDoS attacks coming from IoT cameras
Over the last few weeks we've seen DDoS attacks hitting our systems that show that attackers have switched to new, large methods of bringing down web applications. They appear to come from an IoT botnet (like Mirai and relations) which were responsible for the large attacks against Brian Krebs. Our automatic DDoS mitigation systems have been handling these attacks, but we thought it would be interesting to publish some of the details of what we are seeing.
The curious case of slow downloads
Some time ago we discovered that certain very slow downloads were getting abruptly terminated and began investigating whether that was a client (i.e. web browser) or server (i.e. us) problem. Some users were unable to download a binary file a few megabytes in length. The story was simple—the download connection was abruptly terminated even though the file was in the process of being downloaded. After a brief investigation we confirmed the problem: somewhere in our stack there was a bug.
400Gbps: Winter of Whopping Weekend DDoS Attacks
Over the last month, we’ve been watching some of the largest distributed denial of service (DDoS) attacks ever seen unfold. As CloudFlare has grown we've brought on line systems capable of absorbing and accurately measuring attacks. Since we don't need to resort to crude techniques to block traffic we can measure and filter attacks with accuracy. Our systems sort bad packets from good, keep websites online and keep track of attack packet rates and bits per second.
The story of one latency spike
A customer reported an unusual problem with our CloudFlare CDN: our servers were responding to some HTTP requests slowly. Extremely slowly. 30 seconds slowly. This happened very rarely and wasn't easily reproducible. To make things worse all our usual monitoring hadn't caught the problem. At the application layer everything was fine: our NGINX servers were not reporting any long running requests. Time to send in The Wolf. First, we attempted to reproduce what the customer reported—long HTTP responses.
CloudFlare "Interview Questions"
Loading... For quite some time we've been grilling our candidates about dirty corners of TCP/IP stack. Every engineer here must prove his/her comprehensive understanding of the full network stack. For example: what are the differences in checksumming algorithms between IPv4 and IPv6 stacks?
Show More
loading
Actions
Is this you?
As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.Get in touch with Marek
Contact Marek, search articles and posts on X, monitor coverage, and track replies from one place.
Learn more about Muck Rack