Based on my metrics and logs, around 1 am on the 15th of May, 2023, my (cable modem-based) internet provider just stopped providing IPv6 addressing to me. According to the logs, the first indication of trouble was 03:34 or so, when odhcpd started logging messages about no default route being present (IPv6 one) and my home network hosts lost IPv6 connectivity subsequently.
Then I looked at stored netflows:
mstenber@fw ~>nfdump -R /flows/2024/05/ -t 2024/05/15.00:00:00 ipv6 2>&1 | egrep -v '(fe80|fd42|ff02):'
Date first seen Duration Proto Src IP Addr:Port Dst IP Addr:Port Packets Bytes Flows
...
2024-05-15 01:20:45.525 00:00:00.000 TCP 2001:14..79:115c.60513 -> 2a01:b7..01::2:4.443 1 84 1
The last failed attempt to even use IPv6 was apparently around 1:20. So it most likely broke two, three hours earlier than what was shown in the logs, and just the default route lifetime expired at the point when odhcpd started being unhappy.
Unfortunately I do not have explicit IPv6 availability monitoring, so I can just look for log entries and netflows. (Note: I do now - Uptime Kuma for the win.)
When I noticed the situation on the following day, 16th of May (I don’t obviously use IPv6 enough), I recreated my virtualised OpenWrt router (yellow -> green colour change) but behaviour did not change.
Try #1: Email my ISP (IPv6 downtime ~1d 12h)
I sent a query yesterday evening. I stated clearly that that
- I had restarted my cable modem
- I had reset my home router as well and.
The response (IPv6 downtime ~2d 6h)
I got a response sometime the following morning which was more or less ‘did you try resetting it harder’, and ‘please call us if the problem persists. I figured I would call them once I have time later on in the day.
Bit more investigation
tcpdump showed IPv6 router advertisements on the link, with M bit set; therefore my home router was using DHCPv6 client to try to configure it (and normally it would get /56 prefix delegated, and life would be good, but this not happen anymore for some reason):
~ # tcpdump -n -i eth0 'udp port 546 and port 547'
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:26:57.200937 IP6 fe80::216:3eff:fe66:dc8.546 > ff02::1:2.547: dhcp6 solicit
14:29:06.220920 IP6 fe80::216:3eff:fe66:dc8.546 > ff02::1:2.547: dhcp6 solicit
14:31:13.196761 IP6 fe80::216:3eff:fe66:dc8.546 > ff02::1:2.547: dhcp6 solicit
Try #2: Call my ISP around 16:30 (IPv6 downtime ~2d 15h)
After a ~5 minute wait time, I got to talk to a person. Unfortunately the first line support questions were more or less what I had in the earlier email exchange, also along the lines ‘have you tried resetting your cable modem’ (yes), ‘have you looked at its UI’ (I don’t use the UI, it is in bridging mode).
Perhaps five minutes of that later the request got forwarded to their ops (presumably), where they detected that some kind of outage covered more than my account (no idea how large area). It is somewhat unfortunate that it took 2,5 days to notice that, though, but they promised to look into it and send me an SMS once it gets fixed.
The plot thickens - IPv6 started working (IPv6 downtime 7d 8h)
I had more or less given up on this, except for checking my monitoring every day or so. But this morning IPv6 started actually working around 09:30. Unfortunately the ISP has still to notify me about it having been fixed.. 😏