Hi,
I'm running 7.1rc4 on an RB4011 since a while.
At some point I noticed that it reboots periodically (really always somewhen after 3,5 - 4 hours runtime).
Unfortunately I do not have much more information than that. I obviously could provide a supout but the forum says to report issues here and not to support@.
Overall my configuration is not too complex. The device is just my central home network router and capsman device for two additional hAP-ac^2. Those are running 7.1rc4 as well but do not crash. They have almost no configuration besides running as CAPs.
Currently I do not have logs from just before the crashes.
Any pointers how to debug further?
Periodic crashes in 7.1rc4
You do not have the required permissions to view the files attached to this post.
Re: Periodic crashes in 7.1rc4
send a supout to Mikrotik. they should be interested........
Re: Periodic crashes in 7.1rc4
What I find intriguing here is that after each crash it goes back to more or less the same "base" time.
Oct 1 2021, 21h58 and some seconds.
I would expect the internal clock goes up as the device operates normally ?
Oct 1 2021, 21h58 and some seconds.
I would expect the internal clock goes up as the device operates normally ?
Re: Periodic crashes in 7.1rc4
So that looks fine at least, right?
You do not have the required permissions to view the files attached to this post.
-
-
joegoldman
Forum Veteran
- Posts: 756
- 加入:
Re: Periodic crashes in 7.1rc4
The config is fine - its obviously being set by ntp - but the curious point the other poster was making is that the boot time is always the same to ~2 days ago, all your 'reboot without proper shutdown messages' default back to oct 01 23:57:49 - is this when you upgraded/installed 7.1rc4? Sounds like some kind of memory leak and nothing is really saving on the underlying system so resets back to that point like it was refresh.Bildschirmfoto_2021-10-03_11-59-20.png
So that looks fine at least, right?
Have you tried a proper reboot cycle? Let it shutdown gracefully and come back up..
Alternatively I'd say do a net-install fresh install of rc4 and use a recent backup to restore config. Take a supout and sent it to support though in case its an error with the upgrade process or similar.
Re: Periodic crashes in 7.1rc4
No idea why after boot it falls back to that old date.
I performed a clean shutdown (actually more than one) in the meantime yesterday.
Also I installed 7.1rc4 a while ago not only at Oct 1st.
I'm also not totally sure but I think the problem appeared not immediately after the 7.1r4 update. I noticed it the first time after I installed a few extra packages. Not sure if I didn't notice before or it really started after that.
I installed container, zerotier and user-manager because I wanted to start playing with those. I didn't touch anything for those yet. First thing when I noticed my reboot issue I disabled and eventually removed those packages one after the other to see if it helps but it did not.
A supout I have sent today to support and a netinstall I can try (as soon as people around me can live a while w/o internet connection
)
I performed a clean shutdown (actually more than one) in the meantime yesterday.
Also I installed 7.1rc4 a while ago not only at Oct 1st.
I'm also not totally sure but I think the problem appeared not immediately after the 7.1r4 update. I noticed it the first time after I installed a few extra packages. Not sure if I didn't notice before or it really started after that.
I installed container, zerotier and user-manager because I wanted to start playing with those. I didn't touch anything for those yet. First thing when I noticed my reboot issue I disabled and eventually removed those packages one after the other to see if it helps but it did not.
A supout I have sent today to support and a netinstall I can try (as soon as people around me can live a while w/o internet connection
)Re: Periodic crashes in 7.1rc4
After an reboot I have a warning in Terminal that the time has been adjusted by the NTP-client.
Normal on a reboot the that is 1970-01-01 00:00 and that was on v6.
V7 could just like a RaspberryPI save the last know date and time when doing a reboot and use that as start from time. However if you boot your device weeks later that save data is well out of date.
It could be the same here that the last known time is saved and reused on boot in the time before a NTP sync or the user manually setting date and time.
In your case it seems that the saved time isnever (re)setbecause the router hard crashed.
Normal on a reboot the that is 1970-01-01 00:00 and that was on v6.
V7 could just like a RaspberryPI save the last know date and time when doing a reboot and use that as start from time. However if you boot your device weeks later that save data is well out of date.
It could be the same here that the last known time is saved and reused on boot in the time before a NTP sync or the user manually setting date and time.
In your case it seems that the saved time isnever (re)setbecause the router hard crashed.
Re: Periodic crashes in 7.1rc4
That was exactly the point I was making with my earlier remark.In your case it seems that the saved time isnever (re)setbecause the router hard crashed.
Going on a stretch ... the crash might be related to this internal time not advancing as it should ? Some code not reading the 'real' time, or something related.
Speculation, I know.
For fun did a reset on one of my mAPLite's running 7.1rc4
Code:Select all
[holvoetn@mAPLite91] > log print 16:21:10 system,info router rebooted 16:21:13 interface,info wireguard link up 16:21:15 script,warning NETWATCH - WG Down 16:21:15 system,info led trigger changed 16:21:16 wireless,info 2C:C8:1B:47:0B:3D@wlan1 established connection on 2452000, SSID HomeLab 16:21:17 dhcp,info dhcp-client on wlan1 got IP address 192.168.92.27 16:21:23 system,critical,info ntp change time Oct/03/2021 14:21:23 => Oct/03/2021 14:21:41 16:21:55 system,info,account user holvoetn logged in from 10.255.255.1 via winbox 16:22:00 system,info led trigger changed 16:22:00 script,info NETWATCH - WG Up 16:22:22 system,info,account user holvoetn logged in from 10.255.255.1 via telnet [holvoetn@mAPLite91] >So why does the device of OP go back to Oct 1st ?
Something to try: go back to a previous version (downgrade) and then upgrade again to 7.1rc4 ?
Only the analysis of supout might tell what's happening ...
Re: Periodic crashes in 7.1rc4
Do you have the container package installed?
I'm starting to think that the container package is to blame. I had it installed on my hAP-ac2, it was rebooting all the time, but, seems that just removing does not fix it, just a full config reset.
I'll test my theory on my RB5009 that is not in "production" yet
I'm starting to think that the container package is to blame. I had it installed on my hAP-ac2, it was rebooting all the time, but, seems that just removing does not fix it, just a full config reset.
I'll test my theory on my RB5009 that is not in "production" yet
Re: Periodic crashes in 7.1rc4
Yes, above I wrote that it feels like my problem started after I installed container, zerotier and user-manager. But removing those did not change anything. So this experience seems to match.
Re: Periodic crashes in 7.1rc4
/系统reset-configurationI'm starting to think that the container package is to blame. I had it installed on my hAP-ac2, it was rebooting all the time, but, seems that just removing does not fix it, just a full config reset.
I'll test my theory on my RB5009 that is not in "production" yet
worked for you?
Re: Periodic crashes in 7.1rc4
on the hAPac2 in the end I had to do a netinstall, because it was crashing too much, but there is another issue with the hapac2 that is l2MTU, that also make it crash, so, I had 2 issues that were locking up the device./系统reset-configurationI'm starting to think that the container package is to blame. I had it installed on my hAP-ac2, it was rebooting all the time, but, seems that just removing does not fix it, just a full config reset.
I'll test my theory on my RB5009 that is not in "production" yet
worked for you?
I just installed it on my RB5009 to test, lets see
Re: Periodic crashes in 7.1rc4
When reboot it will always sync just as when it crashes. The difference is that on a reboot the time is not that far apart, this because of when you reboot the time is written.
On a hard crash the current time is not written and is much more in the past. I assume Mirotik left this kind of 'canary' in there to detect normal reboots or crashes.
Big difference, crash an small difference, a manual reboot/triggered reboot.
On a hard crash the current time is not written and is much more in the past. I assume Mirotik left this kind of 'canary' in there to detect normal reboots or crashes.
Big difference, crash an small difference, a manual reboot/triggered reboot.
Re: Periodic crashes in 7.1rc4
My RB5009 is up for 18h after installing the container package, but it is arm64.
I'll install on my hAP-ac2 (that uses the same arch than RB4011) to see if it crashes...
I'll install on my hAP-ac2 (that uses the same arch than RB4011) to see if it crashes...
Re: Periodic crashes in 7.1rc4
I understand where you are going with your thinking but it is not logical.When reboot it will always sync just as when it crashes. The difference is that on a reboot the time is not that far apart, this because of when you reboot the time is written.
On a hard crash the current time is not written and is much more in the past. I assume Mirotik left this kind of 'canary' in there to detect normal reboots or crashes.
Big difference, crash an small difference, a manual reboot/triggered reboot.
Why on subsequent crashes always going back to Oct 1st even though OP said he did downgrade/upgrade after Oct 1st ? Upgrading also involves rebooting at least twice, no ? Package and firmware.
I did another test this morning when leaving home: took my mAP, simply pulled the plug and took it with me to work. I think that classifies as an unexpected shutdown.
Upon reboot 2 hours later, the log file correctly showed this unexpected event and upon NTP-start a correction of just a bit under 4 hours. More then almost 2 hours which my commute took this morning but not a difference of days.
Re: Periodic crashes in 7.1rc4
That four looks to a pivotal point here. The crashes are about fours apart and your last time set also went back four hours. Then maybe the updating of the time triggered the crashes of TS.
Re: Periodic crashes in 7.1rc4
Just an update because I tried to do a netinstall of the device a few times now.
I really don't get netinstall to do anything. I followed the manual really very much in detail. I brought the device into netboot via /system routerboard settings set boot-device=try-ethernet-once-then-nand and had prepared netinstall. When I try to install, it shortly tells me "updating" but no progress bar or anything. Goes into "ready" state quickly. After a reboot everything is back as before.
I tried that often and now I'm scared to not be able to recover my device when doing the next step using the reset button to reset and enter netboot.
Tried on a spare RB2011 doing the same and it worked flawlessly with the same laptop and setup.
Will continue trying but need to have some Plan B to have connectivity... (to be continued)
BTW, support reacted but apparently no idea. Asked for a serial console connection which I cannot provide easily due to missing hardware.
I really don't get netinstall to do anything. I followed the manual really very much in detail. I brought the device into netboot via /system routerboard settings set boot-device=try-ethernet-once-then-nand and had prepared netinstall. When I try to install, it shortly tells me "updating" but no progress bar or anything. Goes into "ready" state quickly. After a reboot everything is back as before.
I tried that often and now I'm scared to not be able to recover my device when doing the next step using the reset button to reset and enter netboot.
Tried on a spare RB2011 doing the same and it worked flawlessly with the same laptop and setup.
Will continue trying but need to have some Plan B to have connectivity... (to be continued)
BTW, support reacted but apparently no idea. Asked for a serial console connection which I cannot provide easily due to missing hardware.
Re: Periodic crashes in 7.1rc4
I have the same problem that Netinstall statred but does continue. Have you tried to run it as Administrator on Win10.
I solved the my 4011 being unusable with RC4 was to go back to 6.48.4 and fromntherecupgrade to v7RC4. It running smoothly now.
I solved the my 4011 being unusable with RC4 was to go back to 6.48.4 and fromntherecupgrade to v7RC4. It running smoothly now.
Re: Periodic crashes in 7.1rc4
Last night I downgraded to 6.48.4 and back to 7.1rc4 as mentioned by msatter.
Unfortunately still seeing exactly the same behavior.
Unfortunately still seeing exactly the same behavior.
-
-
intrepidsilence
刚刚加入了
- Posts: 4
- 加入:
Re: Periodic crashes in 7.1rc4
I don't know if this is relevant to this discussion but I wanted to post it here anyway in case it leads to something useful.
After upgrading to 7.1rc4 it takes my RB4011GS somewhere between 6 and 18 hours to corrupt itself to the point that it must be reset and rolled back to rc3 before it starts working again.
For some reason it stop routing DNS traffic to my DNS server. It also seems to cause port flapping. The base routing out to the Internet still works. Very strange behavior and I was unable to capture anything that helped me pinpoint what was going on. I have gone through this a few times now and I feel that i have proven that it happens after the upgrade to rc4 so I won't be doing that again. I guess I will give it another go in the next rc if the relase notes have anything resembling an acknowledgement and fix.
After upgrading to 7.1rc4 it takes my RB4011GS somewhere between 6 and 18 hours to corrupt itself to the point that it must be reset and rolled back to rc3 before it starts working again.
For some reason it stop routing DNS traffic to my DNS server. It also seems to cause port flapping. The base routing out to the Internet still works. Very strange behavior and I was unable to capture anything that helped me pinpoint what was going on. I have gone through this a few times now and I feel that i have proven that it happens after the upgrade to rc4 so I won't be doing that again. I guess I will give it another go in the next rc if the relase notes have anything resembling an acknowledgement and fix.
Re: Periodic crashes in 7.1rc4
I got same issue. Rb5009 v7.1rc4
Re: Periodic crashes in 7.1rc4
Same as in crashing/rebooting every 3h45m or same as in "starts dying after 6-18 hours"?I got same issue. Rb5009 v7.1rc4
Just for the record. Finally connected a serial console a few minutes ago to possibly see something around the next crash/reboot.
Re: Periodic crashes in 7.1rc4
Just for the record. This is all which shows up through the serial console after the "unexpected reboot":
Seems quite useless unless the 00000050 says anything.
Code:Select all
MikroTik 7.1rc4 (testing) MikroTik Login: :00000050 RouterBOOT booter 7.1rc4 RB4011iGS+5HacQ2HnD CPU frequ.. loading kernel... OK setting up elf image... OK jumping to kernel codeRe: Periodic crashes in 7.1rc4
mine rb5009, crashed too, every 3-5 hours
-
-
edward0488
刚刚加入了
- Posts: 8
- 加入:
Re: Periodic crashes in 7.1rc4
mine rb5009 random reboot on v7.0.5
Re: Periodic crashes in 7.1rc4
Using cake or codel? Thats what's crashing my 4011.
Re: Periodic crashes in 7.1rc4
Updated to 7.1 stable from 6.49.2, my rb4001 still crashing.
Re: Periodic crashes in 7.1rc4
Is fqcodel or cake still a crash garuantee in 7.1.1 STABLE?
Re: Periodic crashes in 7.1rc4
having the same issue, but on the 7.2rc1 on an RB5009... not even waiting 2-3 hours, does it nearly every 30 min in some cases...
Re: Periodic crashes in 7.1rc4
My device was working with with the stable 7.1.x versions.
But updated two days ago to 7.2.3 and crashes seem back
"router was rebooted without proper shutdown"
But updated two days ago to 7.2.3 and crashes seem back
"router was rebooted without proper shutdown"
Re: Periodic crashes in 7.1rc4
我最终rma Mikrotik……雷竞技网站请注意,the company I purchased it from had no stock (RB5009) and now waiting for a new one... moved to a CHR on Hyper-V as a stop-gap and has been stable since... So, it might be hardware..."router was rebooted without proper shutdown"
Re: Periodic crashes in 7.1rc4
FWIW, Mikrotik support asked me to try 7.3 since it has a fix for a "very similar issue".
It seems to have fixed mine as well. At least the device is running since around 10 hours w/o a reboot.
I don't know which item in the changelog it was though.
It seems to have fixed mine as well. At least the device is running since around 10 hours w/o a reboot.
I don't know which item in the changelog it was though.
Who is online
Users browsing this forum:duripand 3 guests