Hi all,

I have a RB3011 with ROS7.5 running as an edge router. Behind it is a pfsense FW via a transit vlan (172.16.100.0/24), and behind that FW, a mail server on one of the PFsense's legs (mailserver=172.16.15.11). The Mikrotik does the NAT-ing between the internal 172.16.0.0/12 networks and the public internet. It looks like this (well, it looks fancier in real life):

(Internet)----Mikrotik-----Pfsense-----Mailserver

In order to be completely honest, I wanted to be completely transparent by telling you that there is indeed a Pfsense FW in between, but please don't point it to be the responsible device:
- this FW has been in place (with different upgrades of course) for 15+ years
- the mail server has been in place for 15+ years (also with upgrades of course)
- all that has changed now is the edge router as I am migrating from Cisco (+ WAN link behind it) to Mikrotik (+new WAN link behind it).
- the workaround I have put in place (see below) and packet captures (also see below) put the PFsense out of cause.

Now to my issue.

Oubound邮件工作完全从我们的服务器(the outside world). However, we are unable to receive mails from outside with simple DST-NAT (with the old cisco it worked flawlessly), that is what makes me wonder). I am pretty sure there is an additional FW rule that needs to be set on the MT, but I can't figure out which...

As a temp workaround, I had to implement DST-NATandSRC-NAT on the mikrotik. It works in terms of receiving emails, however, this is not a viable solution as the mail server sees all inbound mails coming from the MT's LAN IP, which means that the IP-based ACL's (for mitigating brute-force attacks) had to be deactivated and most spam filters are also useless now.

Here's what is currently in place:the last line from the config above (src-nat) is the one I configured as a workaround, but which I need to get rid of asap in order to reactivate the ACL's and spam filters.
172.16.100.2 is the Mikrotik IP on the transit VLAN to the Pfsense

Now, here are 2 packet captures on the FW (on the 172.16.100.0/24 transit VLAN from the MT), both times for an inbound mail coming from the internet:

- without the SRC-NAT rule ( I replaced the external's server IP with 1.1.1.1 as it is a client's server...):- with the SRC-NAT rule (172.16.100.2 is the MT's Transit interface IP)Could you help me out and help me find the rule which I guess is missing on the MT so that the mail reception could work?

谢谢!
Denis

NAT rule seem to be fine.

你没有说任何关于PFse之间路由nse and MT though ... to me it seems as if return traffic (from internal SMTP server towards internet) somehow takes another path, not via MT. Which then likely upsets the remote client. And MT (NAT requires connection tracking machinery)

all vlans behind the pfsense have their default route pointing to the Pfsense

The Pfsense has 2 WAN links, the old one over the cisco router (waiting to be decommissioned), and the other one over the Mikrotik. The PFsense has no dynamic routing, only a few static toutes for sites connected over VPN. (which point to the Mikrotik by the way)

traceroute from the mailserver to the internet is: mailserver=>pfsense=>mikrotik=>ISP. And access from the mailserver to any all websites on the internet works perfectly over the Mikrotik.

On the Pfsense, for the moment, the default route points to the old cisco router, with a route to the mikrotik specified for the mailserver. As part of the tshoot, I tried to define the MT as the default gateway, but the result are exactly the same: no problem for outbound connections (i.e. checking a website), but incoming connections (receiving SMTP traffic from an external server for instance) only works with dst-nat AND src-nat on the MT...

Well, to me everything points at some glitches in how pfsense routes return traffic of DST-NATed traffic (seems like not towards MT). Concentrate on src-address shown by NAT logs ... that should be dst-address set by internal mail server on return packets and whatever routing settings on pfsense these should pass via MT at all costs. Verify that none of routes on pfsense can be the cause (not only according to src-address but also according to dst-address, possibly route rule order matters ... disclaimer: I've no idea about how pfsense does it).

That was my first clue, too, but the fact that the traffic over the old WAN/Cisco (also DST-NATed) works flawlessly. this is what I don't understand.
I think I will put a test web server in on the transit VLAN, so that I can test reaching it from outside over both WAN links (MT and Cisco), without the pfsense in between.

If I can access it from the MT WAN as well as the Cisco WAN, then it will be settled, there is something wrong with the pfsense config. If I can reach it from the Cisco WAN but not from the MT WAN, then we'll have to look into it...

I'll try to get this done today.

I think I will put a test web server in on the transit VLAN, so that I can test reaching it from outside over both WAN links (MT and Cisco), without the pfsense in between.

When working with two WANs it frequently comes down to proper routing without any unnecessary "detours" to improper WAN router. And it's not exactly trivial to get things right, specially so if one tries to configure things for concurrent use of both/all WAN links. I can understand your hesitance to commit to using MT as sole WAN router but it might be you're overcomplicating your life by trying to make sure things work in multi-WAN scenario. If you setup test server in front of pfsense, you'll have to make sure it uses correct WAN gateway (either MT or Cisco) every single time .. depending on which WAN was used for initial connection packets. As I wrote, that's not exactly trivial thing in most scenarios.

I agree. This is why I tried 2 different ways (but with same results). As a starting point, I had the Cisco as the def GW from the PFsense's perspective. Then I did:
1) specifying MT as the gateway for packets coming FROM a specific source (mailserver). The traceroute + packet captures did show that the traffic went through the MT indeed (both directions). Note: for incoming packets (Internet to LAN) it is not a problem since both routers (MT + Cisco) have a static route towards the MT for all local subnets (such as the mailserver vlan). Browsing and outgoing mails did work correctly from mailserver to the internet, but incoming mails ran into the issue mentionned.
2) I defined the MT as the default gateway on the pfsense and removes the specific source routing. the results were identical.

If my next test (web server on the transit vlan) shows that the issue is on the pfsense, the best option might be to it the hard way by disconnecting the Cisco and touring all traffic to the MT at once, as you suggested...

I managed to find a couple of minutes to create a test VM on the transit vlan between the MT and pfsense. And indeed, it is perfectly reachable from the outside world using only dst-nat, and no src-nat. When I move the VM on a vlan behing the pfsense, I need dst-nat to be able to reach it from the outside world...

Therefore, as you pointed out, the issue seems to lay on the side of the pfsense. I will dig deeper into this and report here as I might not be the only one facing this situation.

I made some additional investigations, and found something interresting... the setup is as follows:

• Client is somewhere on the internet (another public IP)

• Test server behind the PFsense

• Captures are made on the PFsense, transit VLAN to the Mikrotik

1) Capture with DST-NAT + SRC-NAT on the Mikrotik:

We see here that everything is working as expected. The client's public IP is replaced by the MT's IP address (thanks to SRC-NAT) on the transit VLAN, and the Pfsense sees the SYN, SYNACK, ACK handshake between the MT and the server (172.16.50.69). It is followed by a GET request from the MT, and the content of the webpage is loaded.


2) Capture with only DST-NAT on the Mikrotik:

Here, what is really, really weird is that we see the SYN request from the client, followed by the SYN, ACK from the server (which validates routing is OK, otherwise the packet would have been dropped), but then, the MT never sends the final ACK to the server, so we get into TCP retransmits until the request times out:

From what you've said so far, you didn't replace anything.
You just added extra stuff to the existing setup (which was already double nat):
- you configured the existing pfSense to handle dual wan;
- you added the extra WAN to pfSense via that new MikroTik that also does NAT;
Now somehow, something doesn't work with these changes and there's nothing else to blame than the new MikroTik router becauseit worked fine for 15 years in a different setup.
Fishy.
Show all the configs, not just 3 firewall rules, and try to make a more complete diagram of the setup.
Also what is the purpose of the first firewall rule you've shown? the `accept to 172.16.15.11 port 25 from wan` rule inforwardchain?

Hi Znevna,
thanks for your input, I might need to add some more details to give you more insights here indeed:

you didn't replace anything. You just added extra stuff to the existing setup (which was already double nat):

Not exactly. The MT will eventually replace the Cisco router. The MT is the new device that was added to the infrastructure. Both routers need to stay online only during the migration, the Cisco will be decommissioned afterwards. The Pfsense does not do any NAT, just simple routing and firewalling. NAT is done on the edge routers only (Currently on the Cisco, to be replaced by the Mikrotik).

you configured the existing pfSense to handle dual wan

是的,我不幸的是必须添加一个second gateway (to the Mikrotik) as a temporary solution because I cannot migrate all services in one shot. It is a slight config change on the pfsense indeed (new additional gateway + specifying this specific GW for selected rules)

you added the extra WAN to pfSense via that new MikroTik that also does NAT

Not really. The Pfsense doesn't do any NAT. The Cisco router does, as also does the MT.

something doesn't work with these changes and there's nothing else to blame than the new MikroTik router because it worked fine for 15 years in a different setup.
Fishy.

Well, what I meant was that on the Cisco, it works perfectly with only DST-NAT, and on the MT, I face the issue that the ACK packer is never received by the Pfsense. As you could read in my previous posts, it seemed to me that the Pfsense was responsible for this behaviour, but my packet captures clearly show that the ACK packet is not received by the Pfsense when only DST-NAT is configured on the MT. If I add SRC-NAT on the MT, I get the packets on the inbound Pfsense interface without the slightest config change on the Pfsense. I'm not blaming or badmouthing the Mikrotik here (I like MT!), but clearly there is something wrong if the SYN packets reach the pfsense but the ACK does not. All that we can conclude here is that it cannot be a routing issue. The link between the MT and the PFsense is a "directly connected" transit vlan.

I will prepare a diagram in the next couple of minutes and upload it.

Also what is the purpose of the first firewall rule you've shown? the `accept to 172.16.15.11 port 25 from wan` rule in forward chain?

172.16.15.11 is the mail server. The "INBOUND SMTP" firewall filter rule you are mentioning is needed to accept inbound traffic (mail sent from a distant SMTP server to my server) after it was been DSTNAT-ed. Initially, I configured this rule on theoutputchain, but it turns out it is the forward chain that needs to be used instead. The logic seems to be that despite the NAT-ing done by the MT, the device still considers that it is just forwarding the packet from src= to dst=.

I created the exact same rule for my test LAMP server (172.16.50.59, port tcp/80) for my last test of course.

I'll prepare the diagram...
Cheers!

Here's a quick drawing of the setup:
while looking into the captures again, I think I found the cause of the issue.
This one is with sdst-nat and src-nat on the MT. From the dst field of this frame (this is the SYN,ACK frame) we can see that the frame is sent to the MT as it should (which is to be expected)...but without the src-Nat on the MT, the pfsense sends the SYN,ACK frame to the Cisco instead of the MT:
...so we have assymetric routing. I need to figure out why the pfsense sends it to the Cisco instead of the MT.

That diagram you've posted looks broken to me, i don't understand how that could work reliably with the details provided.
Could be that I just don't understand networking that well, who knows.
Maybe someone else will take a stab at it.

...so we have assymetric routing. I need to figure out why the pfsense sends it to the Cisco instead of the MT.

That's what I was trying to point out (without being a smartass).

Check pfsense routing ... go through whole config and try to see where that particular SYNACK packet weered towards Cisco. I'm guessing that pfsense is simply not properly configured to handle dual WAN with regard to incoming connections (i.e. it doesn't mark connections via MT as such and then uses default route, which seems to be via Cisco, instead the correct route via MT).

@Znevna: well it's quite easy. VLAN100 is a transit VLAN. On the Pfsense, I have 2 gateways set. GW1 to the Cisco router and GW2 to the MT. Each rule has automatically GW1 set as the default route (that is default Pfsense behaviour), unless you specify it otherwise (on a rule-by-rule basis, or via policy-based routing). In my case, this enables me to migrate services one after the other from the old Cisco router to the new MT. Which part do you feel confused about?

@mkx: Yes you are absolutely right. I know dual WAN can be tricky, especially for incoming connections. Thing is that I thought I had configured the PFsense properly, and some services work fine on the MT already (but these are outbound connections). It appears that I need additional config on the PFsense (setting the "reply-to" variable) which requires a specific config on the Interface first... I'll give it a try tonight. But clearly we now know from the packet captures that it is the pfsense that doesn't send back the packets as it should.

I know dual WAN can be tricky, especially for incoming connections.

It's not tricky even for incoming connections if incoming connections for certain service are only allowed via single WAN, in this case static routing setup is enough. But it gets tricky if incoming connections are allowed via both (all) WANs ... then the ultimate router (in your case that's pfsense) needs to identify the used WAN on first ingress packet and then use same WAN for sending all replies. In MT that's done using mangle rules (routing rules are not good enough), I've no idea what has to be done on pfsense. And according to your network layout it might be that pfsense actually can't identify correct WAN because dst-address is already changed (DST NAT) and src-address doesn't define WAN being used (SMTP client can use any of your WAN addresses to connect). If you used different physical interface to connect pfsense to both routers, you could use pfsense interface as differentiator between WANs, but not in this case. The only difference between packets arriving through both routers will be src MAC address of packets (it will either be mikrotik's or cisco's) ...

As I already wrote: you're complicating your life by going into dual WAN config too deep as it's only temporary. I know that it'd be nice to know actual remote IP address of SMTP client but since configuring src-nat on MT solves all other problems you have, you could simply go this way. After you get rid of Cisco-routed WAN (and/or set default route on pfsense towards MT), simply remove/disable all SRC-NATs created to solve the routing triangle problem.

I've solved it last night. It is working like a charm now.

即使它有点跑题,交易更智慧h the PFsense config than the MT, I'd like to share my findings in case someone else is running into a similar issue. The issue in my setup (2 gateway IP's on the same transit subnet) is, as you stated, that the Pfsense does not know where to route packets in certain cases. Indeed, as you also stated, a better way, especially for a production setup, would be to implement 2 separate WAN interfaces on the Pfsense, each one having a dedicated subnet to 1 router. Of course, in my case that was not possible as it was just a temporary setup while migrating from one router to another.

If incoming packets (from the Internet) are DST-NATed on a router, it is not a problem for the Pfsense as it sees the packets coming from a "directly connected" IP. This even supersedes static routes. For incoming packets that are only SRC-NATed on one of the routers, the return packet is a problem because PFsense, by design, uses thedefault gateway which is specified on its WAN interfaceto send outgoing packets to. Specifying the gateways on the FW rules does not make the trick when you have only 1 WAN interface like I do. There is a workaround using the "reply-to" setting on PFsense's rules, but I didn't need to go that far. In the end, I defined the MT as the default route on PF's Wan interface, and added some routes on the MT for services that needed to be handled by the Cisco router.

I then removed the SRC-NAT on my MT, and migrated the services one after the other from Cisco to MT (I still have telephony to migrate, but short of that, I'm done). Since this morning, 90% of production is now running over the MT.

Regarding your suggestion with SRC-NAT, one has to be very careful. I tried that at the beginning of this topic, but while it is suitable for certain types of service, you definitely can't use it for SMTP for instance: Antispam filters will not work anymore since they will see all mails coming from the SRC-NAT IP. Same for dynamic access-lists needed to secure webmail and imap services against brute-force attacks (e.g. IP gets blocked for 30 days if you have 5 failed authentication attempts within 10 minutes).

Thanks to all of you for your time and suggestions, it definitely helped me figuring out the issue! Cheers!