Title:
MultiWAN with RouterOS

Welcome:
This article aims to bring clarity to the daunting and confusing task of routing multiple WAN and ISP provider connections in and out of your network. We will be using RouterOS version 7 firmware to accomplish these techniques. Examples will include Static IP wan connections, DHCP assignments, PPPoE encapsulations, and using LTE as a backup or main internet connection. Each style has its own unique requirements to get the most out of the WAN availability. What happens when one goes down? Do you want to dedicate one ISP to a certain type of traffic? How does incoming traffic reach internal targets? These solutions and more will be discussed.

MultiWAN is not SD-WAN:
Utilizing more than one internet provider is a way to bring more capacity and uptime to your network. If you have latency sensitive traffic, it can be helpful to put it on another WAN connection. However, multiple wan connections by themselves don't necessarily create a seamless experience for end applications because of the way current TCP protocols are designed. An ISP failover may result in temporary application disruption.

Advanced applications can cope if they are programmed a certain way but streaming style applications are very suspect of background changes. There is a way, using a layer of external servers and routers, to achieve that high level of ISP float technique, where the applications are not aware of what is happening below. Illustrating this is not the purpose of this document which instead focuses on a more rudimentary way to manage multiple internet providers given you only have one router to manage.

Why MultiWAN?
This sounds like a trick question. If a single provider had amazing latency, enormous bandwidth, and never experienced any downtime, why indeed would there be the need for another network? I suppose there would not be much of a need. But in the real world networks aren't everything we wish they were. Sometimes internet providers can only offer so much latency, or they go down from events taking place in the area.

Sometimes it is more cost effective to put an application on one ISP and the main bulk of your network on another. There is also the possibility that you have a physically different connection to a network resource and pathway you own and control. Still another scenario is for when networks come and go based on where you might be. The router maybe a moving device and thus different wireless networks come and go. They don't fail exactly, they just are not there for awhile.


MultiWAN Overview:
There is a lot to cover so make sure you read slowly. This article series will illustrate a scenario that will have several moving parts. You will not implement everything because your environment is different and custom to you. This means you will not be able to simply copy/past the examples. You will need to add, remove, or customize what has been presented. To do that, you will need to understand the mono sized router example.

示例显示了一个路由器有很多广域网connections. You will not have this exact arrangement. Do you have LTE? If not, do not implement that section of the example. A best effort has been made to create a single example that can easily shed various parts that are not needed. This article assumes aVLANenvironment. So, consider that a requirement in your learning journey.


Disclaimer:
What follows is my best understanding of how to implement the stated goals in RouterOS v7 based on the generous feedback from many forum community members. I am a student and spend time in the forums to learn and give back when I can. I am not an expert nor am I even aforum guruas my forum title humorously states. There are far more qualified persons in attendance. My skill is taking what others have shared and then building a presentation around their thoughts and techniques. Thank you. Note that this article is new and has many mistakes. It will be updated many times until the community considers it the gold standard.

Example1:
We will lead into this topic using a generic example that covers the majority of situations. We have four WAN connections and need a method to determine when a primary connection is down. When primary is deemed down, the router should switch the environment over to another WAN network. Because we have multiple WAN connections, some traffic is configured to use a specific WAN. Static, DHCP, PPPoE, and LTE style WAN types are shown. Remove or rearrange assignments that don't apply to you.


Are we down?
Finding out if a network is down is tricky. The absolute best way would be an application inside your network that makes connections to resources outside your network for every type of application you deem important. That would be a very special application. Lacking that, a low cost, easy to use, andincluded withRouterOS option, is to use a technique calledrecursive nexthop lookup(aka recursive routing). This just means that we validate theentire pathto a host, instead of just the one connected to our router directly. This way, your WAN's connection to the internet and thus outside of your ISP network, can be verified.

A caveat of course is that the host you are checking is outside your ISP and is itselfnotdown. Because a single host could be down it is therefore highly recommended to check two separate hosts. You could check as many as you feel warrant a decision. Here we show two.

Gateway Ping
路由验证执行ping检查。Every ten seconds a ping is sent to a remote Host. Failing that, another ping is attempted. Two failed ping replies will set the route as invalid and unreachable. Thecheck-gatewayparameter helps us to accomplish this but not on its own. It is necessary to link two route commands with each other.

Scope and Target Scope
To validate a path to a remote host, we use route entries that are connected to each other over aScopeandTarget Scopearrangement. These two parameters are an unfortunate abstraction that we as network administrators must deal with. I'll attempt an explanation. Think ofscopeas the area of your concern. How much of an area, how far, how deep, and how wide of an area do you care about? This is your scope size.


Target Scope would then be thenextarea you care about. Since we are only concerned with next hop paths, the way we tell RouterOS to use a particular route as anext hopto be validated, is to set theTarget-Scope higherthan theScope, effectively increasing the size of your standard scope. It takes two command lines to show this awkward representation and linking.

Forum memberanavhas beautifully hacked this concept by always setting the defaultScopeto 10 and using theTarget Scopeparameter to change therelationship. Note that two linked route entries is enough for validation as shown in the simplified diagram. But in our examples, you will note that we also add one more route for return traffic. This is because we want the ability to use the other ISP connections even when primary is up.

If we are up
When both networks are up, we show other networks being utilized instead of leaving them always idle. We also enable traffic to connect remotely into the network from any available WAN. Also shown is having certain traffic always leave out of a specific WAN.



DHCP WAN Type
If you have a DHCP connection to your ISP, you will note that we use a static route in the example. Attached to your DHCP client, is a curious script provided byrextended(who is author?). With a static route, the script becomes necessary if the ISP changes the IP Address or gateway for any reason. The script keeps your dhcp client and manually added route in sync.

When this event occurs, the script will fire and compare the client gateway value with any route that has a comment ofISP2_Monitor. The script reads like so:if this dhcp client has an ip address, search through all routes for a route in which the gateway does not match our own and which has a comment of "ISP2_Monitor". If both conditions are true, change the gateway value.

LTE WAN Type
If you have LTE availability in your area, the best way to utilize this service is with an LTE enabled MikroTik router running ROS v7 which has better support for modems and behavior. To us, these hockey pucks are radios and that's about all. To that end, enable thepassthrough interfacefeature in the APN configuration. However, if the LTE hardware only has one ethernet interface (or you only want to use one interface), you'll loose the ability to manage the LTE unit. This is not a problem, however with a simple VLAN between the two devices. Our example shows this arrangement. See theLTE Router Examplelinked below.


MultiWAN Router Example1
LTE Router Example

Reserved

Reserved

Reserved

Excellent pcunite..........if only my posts were so well put together as your approach LOL.

@anav,

That means a lot. Your help on the forums is felt and you've helped me personally. You know more than me! These long form articles take a long time to produce. So, don't feel bad about that. We all have our own strengths. This is a way for me to give back.

Give me a few weeks to get this article in order and then we can hammer on it. I'm writing, producing content as I go. Then I'll publish out the syntax.

If it is the same quality as your vlan series, it will be a great contribution !
Subscribed to get the updates.

Good idea! Some significant number of posts on the forum involve some kinda "MultiWAN situation". And @anav was going to run out of letters in his "new user" post.

I like the approach starting with the 4 WAN types you have. Think focus on "recursive lookup" (vs "netwatch techniques") for upstream failure detection seems a good call... I use RRs (with PBR) today, but with all the new netwatch detection mechanism in 7.7, I can see a well-design script doing better than RR – but as generic/"general purpose" approach might be tough to explain. Now I guess, depending on when and what form BFD comes, that may offer a 3rd failure detection option – but that's still not in V7 to even consider.

Anyway, I'll follow with interest. Glad someone is taking this on – MT's docs have long lacked on any kinda of respectable "user guide", and offer very limited cookbook/recipe style docs to even hint at some canonical network architectures/designs.

@pcunite

Bravo.

... focus on "recursive lookup" (vs "netwatch techniques") for upstream failure detection seems a good call ... with the new netwatch detection mechanism in 7.7, I can see a well-design script doing better than RR ... when BFD comes, that may offer a 3rd failure detection option.

I agree. I hope to see a new netwatch routine make it into this series. In the opening graphic, the text bubble states: "recursive lookup or netwatch techniques". I wanted to lead with RR because it seems more popular on the forums. So, I speak to that. It is a mess to make sense of in the route menu, however. I would personally prefer a script, but that is going pretty deep into the woods. Actually, I would prefer an application in RouterOS to identify and process a failed network. I do hope to engage yourself andrextendedto create something easy to follow as an alternative to RR. I'll have some ideas to offer when we get to that section. BFD will take the forum by storm because sockets are the best way to identify and process downed networks.

Is PCC also going to be part of the scope ?

Is PCC also going to be part of the scope ?

A perfectly cooked beef tenderloin served with buttered mashed potatoes, lemon roasted asparagus, and ranch covered salad spears is not enough for you? You want ... dessert too?

Pssst he's Belgium has a France complex, dont mention desserts!!!

1 i don't like dessert
2 isn't pcc suitable for multi wan ?

I'm sure they'll be time for a debate about load balancing, e.g. PCC vs ECMP vs .... But in reality you almost always want some failure detection, regardless if a failover or load balancing case.

Perhaps a nod to the "check-gateway=ping" that's required in routes, before explaining the scopes and recursive routing? That's actually what triggers/blocks the route recursion.

pcunite I noticed on ex1, if I am not mistaken, you are using 1.1.1.1 for an ISP address?? This is not an ideal choice as that is the IP address for clouldflare DNS services and I happen to use this as a host to check my recursive routes......... very confusing when I saw that next to ISP............ ?????? Please change...

you are using 1.1.1.1 for an ISP address?

Opps, yes I had that incorrect in the diagram. Thank you. The rsc file was correct, however. DNS Host1 is 1.1.1.1, it is what the example is pinging. The ISP is 10.1.1.1 and the recursive route is naturally 1.1.1.1.

Can you explain how your script works ( ip dhpc script in first example)
a. what each command is invoking but in english and not script language, in other words right the script in words,
b. what does it do functionally
c. why is it needed.

Okay, Example1 is ready for intensive criticism. What could be more clear? Are there any errors?

A note about Example1:
This is a recursive routing example. It is supposed to stand on its own as a deliverable. It answers the question: "How do I wire up multiple WANs and make sending/receiving, custom routes in/out, and fail-over just work?“这很简单,将让人们basics. However, let's see the kinda of questions we get.

For Example 2, maybe we have a scripting version (Netwatch) instead of recursive? Example3 would be WAN Load Balancing perholvoetnrequest?

I dont understand your mangling on example 1. You mark in-interface WAN traffic and then you mangle it again???? The second one makes no sense to me.
It thought you want to ensure in-interface=LAN1 and interface=LAN2 go out the right WAN correct............

For Example 2, maybe we have a scripting version (Netwatch) instead of recursive?

I still think you devalue just a basic "check-gateway=ping" as an option 1. That does something for little effort (e.g. just the next hop – better than either 0 checking) – want more? move on to recursive routing...

I still wish MT do something like this:
viewtopic.php?t=192844
as it allow better scripting of liveness routing & avoid recursive routes in more cases. Netwatch is become quite sophisticated of late. And, if there were a feature to link a route to a netwatch status, that let you define the monitored host in netwatch. A poor man's BFD.

Agreed. As for the feature request, I concur. Regarding example 2, why not add PCC and perhaps a @Sindy optimization to minmized the cpu load when the number of mangle rules becomes sgnificant. 1)viewtopic.php?t=134048#p659676, 2)viewtopic.php?p=962647#p962647

Hmm... I asked about adding pcc before and my nose was practically gone

Haha ... don't be sorry, what can I do to ease your pain?

Caress holvoe with your wings to ease his pain, but don't accidentally nip him in the nose with those fangs....... or are those front hanging testicles, hard to tell, maybe just dripping globules of phlegm. =-)

Oh man, this is gold dust! Please continue development of this topic.

Hi,

I encounter a strange issue today.
I had a failure on my primary link and another failure on my dynamic ip update script for my secondary link...

When the primary link came back, i tested my secondary link input rules and routes (when my 2 links are up and running).
A simple curl from an internet test server to my secondary ip link (LTE) took very long time (at least 20-30s) to answer.
However, output (NAT) from LAN to internet via the secondary link do not have any issue and work as expected.
我当然没有ny issue on my primary link (input/output).

I defined a simple route in main table to force output to the server where i tested input (curl) and after that, it worked as expected. The requests on the secondary link answer very quick as expected.
In conclusion, there seems to be an routing issue, but i do not understand where i made a mistake.

My router is a CCR2116 with RouterOS 7.7.
Here is the config in attachment, could you please check if everything is fine ?

EDIT1 : i tried to use tools/Torch to check, and as soon as i enable Torch, everything work perfectly ! I stop Torch issue come back again !
EDIT2 : upgraded to the last 7.10.2, no change. I even tried to completely disable recursive monitoring for primary link, (and only use distance) no change
EDIT3 : in real failover (when primary link is really down), no issue, the secondary ip (failover link) work and answer very quickly

thanks !

Cyayon DO NOT DOUBLE POST. Besides you should know better, this thread is to discuss the user article not solve your issues which CAN BE FOLLOWED HERE:

viewtopic.php?p=1012786#p1012786


( I would say our newly minted mods are asleep on the job)