Blocking SYN-Flood Attacks on macOS?

I did see that. Unfortunately I don't have any experience with pf and especially not with synproxy. I also couldn't find anything about synproxy and mac os on the interwebs; not even like stackexchange posts about it not working with no solutions (like i saw for pf and divert). That said... I did see this post about it not working on NetBSD in a similar way. The described symptoms sound similar to your experienced symptoms, so I wonder if their workaround will work, too. They've suggested adding a second loop back interface with a new RFC1918 address, but I don't see why you couldn't just redirect to 127.0.0.1, or add a RFC1918 alias to lo0. So your rule might be pass in quick proto tcp from any to X.98 port { 80 443 } flags S/SA rdr-to 127.0.0.1 synproxy state Or whatever other IP you add. If you're doing IP based virtual hosts, you'd need to tweak your apache config a smidge to manage, but not too bad.

So I just added "rdr-to 127.0.0.1" to my web/mail port rule on my dummy server and just like that, it appears to be working. I can't really tell if the proxy is working because its a legit connection when I load a dummy web page, so if it's proxied, it's only proxied for a ms. If only I knew how to execute a syn flood so I could do it locally and see if the proxied ports pile up.

Also, before I enable this on my real server, can you -if you know- explain why this is working? I want to make sure I fully understand what is going on.

Also, is rdr-to the same thing you'd use to forward connections from a server, to a VM running on that server, and then back to the server again? Well I guess not exactly because the redirect BACK from the VM to the real server would have to be also redirected to a custom port number otherwise it would get stuck in an infinite loop. If I had more free time, I'd play around with some infinite loops like that

Hopefully I don't need the VM approach, hopefully a working synproxy will be sufficient.
Also my data centered offered me free access to their big ddos mitigating service, so I can fall back on that if I can't eventually get a working solution done on my own.

l008com said:

So I just added "rdr-to 127.0.0.1" to my web/mail port rule on my dummy server and just like that, it appears to be working. I can't really tell if the proxy is working because its a legit connection when I load a dummy web page, so if it's proxied, it's only proxied for a ms. If only I knew how to execute a syn flood so I could do it locally and see if the proxied ports pile up.

Also, before I enable this on my real server, can you -if you know- explain why this is working? I want to make sure I fully understand what is going on.

Also, is rdr-to the same thing you'd use to forward connections from a server, to a VM running on that server, and then back to the server again? Well I guess not exactly because the redirect BACK from the VM to the real server would have to be also redirected to a custom port number otherwise it would get stuck in an infinite loop. If I had more free time, I'd play around with some infinite loops like that

Hopefully I don't need the VM approach, hopefully a working synproxy will be sufficient.
Also my data centered offered me free access to their big ddos mitigating service, so I can fall back on that if I can't eventually get a working solution done on my own.

Click to expand...

I don't know why this is working; I saw it on that mailing list post, and figured, if it works there, it'll work for you too. If I had to guess, it's probably something like the response packets aren't getting properly mangled, but redir-to helps them get flagged for mangling. I'm not going to dig into the firewall code and figure it out; I'm retired

rdr-to is "destination NAT", what people would normally use to forward a port from a firewall machine to a machine inside with only an RFC1918 address. You could use this for the VM, but I'd recommend route-to instead; it's a bit more work to setup, but it forwards the packets as-is, and doesn't need state tracking on the forwarder. (no sense tracking state in both places)

Incidentally, and unrelated, forwarding this way with a load balancer is called 'direct server return', as the servers response packets don't have to go back through the load balancer, which enables a simpler load balancer to handle way more connections. Of course, if the destination server is a VM on the host, packets are going to have to go back through the host.

So i'm back trying to get synproxy running on my dummy server.

There may have been some unrelated problems that caused previous attempts to fail, so they weren't necessarily reliable tests.

So now I'm back, trying to enable it with the following rule:


The rdr-to 127.0.0.1 is preventing the rules from being loaded. When I try to load the rule, it gives me a syntax error. If I delete that one portion, it loads the rules file. I'm not sure if I was doing something different when I was trying this a few weeks ago or if I just didn't notice.

Hmmm... I found a FreeBSD forum post that looks similar. I pulled config examples from OpenBSD, but it looks like it's a newer syntax, and you'll probably have to use older syntax like they said for FreeBSD.

pf is definitely not my strong suit. I'm pretty sure you should be able to adjust rules while the system is running, and I certainly have experience doing that with ipfw, but ipfw has rule numbers that make it a lot easier to turn rules on or off etc (although, I had a lot of issues with crashing while tweaking ipfw rules on FreeBSD <= 9 systems under load... stopped happening when we upgraded, but never figured out what change fixed it )

So some good news. I have a script up and running that resets my TCP stack if the server is offline for more than 5 minutes. It has been working very well, I haven't had to do any hard server reboots. Even during severe attacks, once the server gets knocked offline, after 5 minutes, it shuts down all ethernet interfaces for 10 seconds, then brings everything back on.

But synproxy would still be helpful here. I was poking around with that some more today but had no luck. It seems like the completion of the handshake proxy is getting hung up in the firewall somehow. So I tried commenting out the "Block in all" rule, and turning on synproxy. But that still didn't work. The synproxy counter still goes up when you try to load web pages through it, but still none of the pages actually load. I'm still convinced theres a way to make it work, I just need to figure out the secret

Strangely, not only can I not find any examples of people successfully using this specific feature on their Macs, I can't find anyone using this features in `pf` on any system. I think there's some specific way you have to do it that is missing from the docs. But at least while I keep searching, the server is still online. I had some bad days where the server was offline for most of the day. Now it's generally no more than 10 minutes, and usually less than that, then it resets itself and is back up and ready to go.

The easy SA answer would have been to subscribe any and all servers to a properly maintained blacklist to auto filter out malicious IPs.
Block all ports you don't actively use to catch new IPs.

Get a firewall or sec appliance in front of the mac.
You are running in a colo not your house.
Any pressure on compute resources, even if successful in filtering, is considered a successful attack bc it's eating time and compute to deal with.

I used to contract for an ISP that rented colo space to companies, I know if 1 client that has a pair of older Mac Pros that serve documents for North America that had to go this route bc they lacked in house engineering to maintain their presence.

I can see a use case for them to migrate to AWS since publishing static content in 2020 shouldn't require server resources at this point for anything but an origin to feed a CDN.

l008com said:

Strangely, not only can I not find any examples of people successfully using this specific feature on their Macs, I can't find anyone using this features in `pf` on any system. I think there's some specific way you have to do it that is missing from the docs. But at least while I keep searching, the server is still online. I had some bad days where the server was offline for most of the day. Now it's generally no more than 10 minutes, and usually less than that, then it resets itself and is back up and ready to go.

Click to expand...

From what I saw in the docs and mailing lists, it seems like synproxy was expected to be used by a firewall machine in between the internet and the server machine; people weren't getting it to work on the same machine, that I could see. If you have time to fiddle around with it, I'd try to get tcpdumps of the syn received and syn + ack sent, and any response (in a firewall machine in the middle case, you'd be able to see also the modified syn and the syn + ack sent to that, as well as if the syn + ack was modified, but alas). If the syn + ack doesn't match the syn, that's a smoking gun. Be sure to use -S on the tcpdump so it doesn't rewrite sequence numbers.

somebrains said:

The easy SA answer would have been to subscribe any and all servers to a properly maintained blacklist to auto filter out malicious IPs.
Block all ports you don't actively use to catch new IPs.

Click to expand...

blocking IPs doesn't work against synfloods, because most synflooders are able to spoof source addresses. The only way that could work is a whitelist approach, if the whitelist is pretty small (like only allow from corp IPs or something).

somebrains said:

Get a firewall or sec appliance in front of the mac.
You are running in a colo not your house.
Any pressure on compute resources, even if successful in filtering, is considered a successful attack bc it's eating time and compute to deal with.

Click to expand...

He tried using the colo's security appliance, and it didn't really work. Sticking a decent OS in front of the mac would probably work, but comes at a cost of hardware and rack space. Assuming there's a good reason to run on mac (maybe some automation that uses software only for mac), and there's no budget for a firewall, going from a 30 second synflood takes the box offline until someone gets around to rebooting it to 5 minutes later, it fixes itself is a win. Whatever the service is, it doesn't seem to need super high availability, so this is fine. Some tuning could probably cut the time to resolve down, but there's also an escalation game with whoever is flooding the thing; if you get better at handling it, they get better at flooding you, and there's no way mac os is going to win at DDoS escalation, so taking a smoke break instantly is a good way to make it unfun for the flooders. In a mixed customer colo, it could just be some idiot going sequentially through all the IPs because someone they don't like is in the same neighborhood; who knows.

Whitelist only is a lot like treating your public subnet like they're private, but they're still exposed.

I'm not sure what appliances are available in this specific cade, but heuristic scanning kicking off an automated block of attack I take for granted. I'm used to the colo itself filtering the traffic and sending nasty emails to owners of that cage trying to upsell services.

Things are getting weirder.
I was playing around with my dummy server and found out that the `ifconfig en0 down` command wasn't quite working. It was bringing the interface down but the interface would pop back up after a few seconds on it's own, not waiting for my own up command. Turns out the system was restoring itself to the settings configured in system prefs. So instead of taking down the interfaces that way, I have to use apple's own tools to take it down and keep it down:

networksetup -setnetworkserviceenabled Ethernet_1 on

Click to expand...

So that works by using on/off. I guess. But now I'm starting to get incidents where my server suddenly goes down like before. But my script doesn't restore it. When I reboot it remotely and log in, there are no SYN_RCVD packets, not even at the start of the incident. Furthermore, my script did in-fact toggle the network off and on several times, but it didn't do the trick. This is very frustrating because it's so hard to tell what is actually going on.

So I'm not sure if this is a different kind of outage that my old script was still able to reset and mitigate, or if this is a totally different situation. These things happen infrequently enough that it's really hard to troubleshoot. I might go a week without an issue. Then have a bunch of issues in short succession.

Intermittent issues are almost worse than hourly incidents. ugh.

If you have the storage space, I'd really try to get some form of continuous tcpdump. If you have enough space to do something like tcpdump -s 40 -G 60 -w /tmp/tcpdump%H%M.pcap goes a long way to seeing what's going on. If you are sure you can catch the system within an hour, you can do -w tcpdump%M.pcap and only keep 60 files. -s 40 isn't always enough, you can experiment a bit, in case you get a lot of important tcp options you want to check out.

Of course, there's a danger. Once you spend enough time looking at wireshark, you start looking at all problems like network problems, and you get drawn into posts about syn floods p) and you get an ISP tech to come out on Dec 30th because going to http://icmpcheckv6.popcount.org/ resets the DSL modem (every time ; thankfully i't s a modern Linux kernel 2.6.30, so it certainly doesn't have any bugs or anything like that).

It would be really hard trying to catch an incident with tcpdump. Plus there is so much data in and out, this is a very high traffic web server.

So I contacted my data center, they checked their logs for the time in question and didn't see anything. I looked through my own logs that save certain things any time my server is offline. What I saw with ifconfig is that after a few minutes, both of my interfaces dropped their IPs and never got them back. Now, my script does take the interfaces offline every 3 minutes. But the logs are taken every 1 minute. So worst case should be that every 3rd log of ifconfig has no IPs, then they get them back. But that's not happening, there is a stretch of ~15 minutes where there are no IPs on either interface. Makes me think maybe the router at the data center was attacked, or that there was some glitch in their system. Maybe this whole incident was a fluke. I guess for now I need to run it as-is for a while, and potentially implement system-rebooting into my script when repeated TCP resets aren't doing the trick.

The size and complexity of my bandaid continues to grow. . .

l008com said:

It would be really hard trying to catch an incident with tcpdump. Plus there is so much data in and out, this is a very high traffic web server.

Click to expand...

Data is big is why you only store headers, not the whole packet. Maybe you can store 5-15 minutes, and save the buffer the first time your script triggers in a day? Maybe even just 90 seconds. It'll be a lot to sift through in wireshark, but often there's a big change in packet rate, and there's something fishy within a couple screens before the change.

You could maybe capture only packets that aren't tcp with just ack or ack+push. Those are likely less interesting than most, and more numerous than most would be my guess. Although, certainly not always: two of the nasty tcp bugs I ran into involved excess sending of data or just empty acks (sometimes up to 10gbps worth, fun times).

Hi toast0

I was just poking around my firewall after getting the idea in my head that maybe i can mitigate SOME of this problem by blocking malformed packets in my firewall. But as usual, `pf` is the secret firewall and theres not much out there on it. Googling around, I found these three rules which are supposed to block specific kinds of malformed packets. But I don't want to implement it until I fully understand what it is doing. Do YOU know what these flags all mean?

# block in quick all flags SF/SFRA
# block in quick all flags S/SA
# block in quick all flags FUP/FUP

Click to expand...

More reading and it looks like just doing `scrub all in` will drop all packets with invalid flag combinations. I don't know if this will help but it shouldn't hurt, I'll try to test this out soon and see if it does help at all. I suspect it may.

ALSO toast0 i remember recently we crossed paths somewhere else but I don't remember where that was, was it irc, or reddit?

Any web server should not be direct on the internet anyways an a perimeter firewall or similar (reverse proxy like NGINX) in front of it, to handle all of this as that is what they are for and can be configured to do what you need to drop said packets.

l008com said:

But as usual, `pf` is the secret firewall and theres not much out there on it

Click to expand...

There's a lot out there about pf. But most of of it is about OpenBSD's current version of pf. Those who run pf on FreeBSD or the ancient version on MacOS have a lot harder time.

I would try the scrub all, it shouldn't hurt. If I recall off the top of my head, the block flags is looking at tcp flags

S/SA means looking at the syn and ack flags, block if only Syn is set. which you wouldn't want.

SF/SFRA is blocking packets with syn and fin set, but not Ack and whatever R is.

l008com said:

ALSO toast0 i remember recently we crossed paths somewhere else but I don't remember where that was, was it irc, or reddit?

Click to expand...

I think we bumped into each other on Hacker News? I have a limited profile on reddit, and I've mostly stayed off IRC since I went to college, so I would stay out of trouble (it worked! at least one of my IRC buddies did time, and I had a good wholesome career!)

MrGuvernment said:

Any web server should not be direct on the internet anyways an a perimeter firewall or similar (reverse proxy like NGINX) in front of it, to handle all of this as that is what they are for and can be configured to do what you need to drop said packets.

Click to expand...

Guess what, NGINX is a web server and you just suggested he run that direct on the internet. Mac OS really is missing some features to be a competent internet connected server, but you should be able to just run a web server on the internet. Yeah, it's good to be careful about what you run on anything connected to the internet, especially if it's direct; but only run the daemons you need, local firewall to limit inbound ports if you are extra paranoid should be enough. Synfloods were generally solved in the early 2000s (sometimes you might need a bit of tuning... IIRC, I got hit with synfloods around 2017-2018 where they took down my 2x10G boxes with closer to 1-2gbps of syns; but with a bit of tuning, I could handle line rate syns, and I could have dropped syns with a same host firewall), but Mac OS pulled FreeBSD's tcp code early and refreshed it a few months before syncookies were added and it's now been about 24 years; Apple did add multipath TCP, and merging FreeBSD's stack again after so long would be a fair amount of work, but it would make a much better server OS if they did... it might make a better client OS too, but at reasonable socket counts, nobody would notice the difference.

If you configure it to be yes, in the sense of all your website content is hosted directly from said instance vs configuring it to be a proxy, which is in front of the actual source of said content.

IIS, Apache, NGINX, no matter what I had running websites, it always had a NGINX reverse proxy sitting in front doing the SSL offloading and direction to internal webserver, and I would lock down my config files for each site, in front of that was my firewall which NAT'd to said NGINX proxy, this was also because I had said webservers on a separate VLAN as well from the proxy to even further limit access and traffic.

I was a little over the top back then

lol 5 year necro and you guys pick up like it didnt even happen

pendragon1 said:

lol 5 year necro and you guys pick up like it didnt even happen

Click to expand...

That's the beauty of BSD. Everything you learned 5 or 30 years ago is still relevant. I'm sure systemd is replacing syncookies as we speak.

When you get old, 5 years goes by in the blink of an eye. Feels like we just left this thread hanging a month ago.

So this is curious. On my test server, when I do scrub all in at the top, then I can do synproxy state on my port 80 rule and NOW synproxy seems to be working.
I'm afraid to actually enable this on my real server yet but that is a very interesting development that needs more testing.

... more testing done. On test server, i disabled scrub all in, kept the synproxy setting, and no access. This is crazy! After all these years!

I'm still a little afraid to go live with this and risk locking myself out of my server, but this problem may finally be fixed!

LOL of course not, its the exact opposite of fixed. scrub all in is causing a syntax error that is preventing the firewall from starting up at all.

Ok so scrub all in is incorrect, apparently it's scrub in all

But it's still throwing a syntax error even when worded properly.

Rules must be in order: options, normalization, queueing, translation, filtering

Click to expand...

I put my scrub in all rules underneath set skip on lo0. The only stuff above that is anchor points added by apple that I don't *think* do anything. All my block and pass rules are below all that. I might have to call it quits for the night but if you think you know whats going on, do tell! Getting synproxy working was likely a false alarm, but even getting the scrubbing working would be a bonus while I work on freebsd.

Ok I made a little progress anyway. The error goes away if I comment all of the apple anchor points above the rules. But I'm not even sure what these lines mean.

# com.apple anchor point
#
scrub-anchor "com.apple/*"
nat-anchor "com.apple/*"
rdr-anchor "com.apple/*"
dummynet-anchor "com.apple/*"
anchor "com.apple/*"
load anchor "com.apple" from "/etc/pf.anchors/com.apple"

Click to expand...

Comment that out, and it goes away.

The only uncommented line in /etc/pf.anchors/com.apple is:

anchor "200.AirDrop/*"
anchor "250.ApplicationFirewall/*"

Click to expand...

Appears to be some kind of syntax for including rules from another file/zone, but I'm a complete fw noob and have no osx/bsd knowledge.

Ok so final update on scrubbing, I commented out all the external anchors added by the OS. They appeared to be doing nothing anyway but they were causing the scrub command to throw a syntax error. With that gone, I was able to get scrubbing working. Synproxy still does not work. I'd love to get an apple engineer to look at that and figure out why. I suspect theres some extra rule needed to complete the proxy but who knows what it is. But I should be good for now, we'll see if this scrubbing actually helps or not.

Sadly, it does not look like the scrubbing is having any effect at all. Oh well, it was worth a shot.

Oh hey, did you see this? https://photon.codes/blog/we-found-a-ticking-time-bomb-in-macos-tcp-networking

Not sure if this lines up though, but interesting to see another issue with tcp on macos.

Never heard of that. I'm also very skeptical. They don't say its any specific macos version, but "ANY" macos version. I've been running multiple Mac servers at a time for over 25 years and they're almost always over 49 days of uptime and I have never experienced this problem. I wish they gave more info about what version they tested on. If this is a new bug in much newer macos versions that i have not yet run as a server, that could make sense. But they make ti sound like its EVERY macOS version and thats clearly not the case.

Under normal circumstances, I would send you a link to one of my many mac servers (aside from the primary one this post is all about) that have been up for much much longer than 49 days. But alas, 28 days ago we had a power outage so everything got rebooted at home.

l008com said:

Never heard of that. I'm also very skeptical.

Click to expand...

Fair enough. I'm on vacation, so I only skimmed the article, and it seemed plausible, but weird. Seems like something they could have tracked down to version ranges; all versions since the dawn of os x is a lot, and someone probanly would have noticed before now.

Yeah I think the newest macos i've run on mac servers is.... Monterey. So if its a new problem then thats possible but they make it sounds like its every macos in that article and its definitely not. Who knows. Theres a lot of content in that article but not a lot of actual information.