WiFi Mesh unstable when parent offline (IDFGH-13875)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
v5.3.0
Espressif SoC revision.
Chip is ESP32-S3 (QFN56) (revision v0.2)
Operating System used.
Windows
How did you build your project?
VS Code IDE
If you are using Windows, please specify command line type.
PowerShell
Development Kit.
ESP32-S3-WROOM-1
Power Supply used.
USB
What is the expected behavior?
I expect the ESP32 to continue to run the application without crashing when the WIFI Mesh parent disappears.
If the MESH_ROOT was powered off, I expect a MESH_NODE to assume the role of MESH_ROOT
If the WIFI Router is powered off, when I restore it, I expect the mesh network to establish itself
What is the actual behavior?

Sometimes these tests work perfectly.  The Mesh network goes down and the nodes start scanning. If I restore the WiFi router,  the Mesh network is reestablished
Sometimes the Mesh network goes down, and can't recover. It doesn't crash but it doesn't scan properly and reestablish the Mesh network.
Very regularly, if I power off the WiFi router the MESH_ROOT intermittently crashes OR if I power off the MESH_ROOT a MESH_NODE intermittently crash

Steps to reproduce.

Power on system comprising 2 x ESP32-S3 dev boards and a Wifi router
Connect a serial terminal (I am using PUTTY) to each serial port for monitoring
Let the Mesh network get established and verify MESH_ROOT and MESH_NODE connected.
Power off the WiFi Router
In the example (logs below) the MESH_ROOT crashed

Debug Logs.
I (00:58:03.336) aWifiMesh: <MESH_EVENT_MESH_STARTED>ID:77:77:77:77:77:76
I (136546) mesh: <MESH_NWK_LOOK_FOR_NETWORK>need_scan:0x3, need_scan_router:0x0, look_for_nwk_count:1
I (00:58:03.336) aWifiMesh: This node MAC:48:ca:43:9b:53:d8
I (00:58:03.354) aWifiMesh: WiFi Mesh started successfully, heap:141084, root not fixed
WIN> I (140766) mesh: [S6]VONETS, 00:17:13:20:bd:74, channel:8, rssi:-12
I (140776) mesh: find router:[ssid_len:6]VONETS, rssi:-12, 00:17:13:20:bd:74(encrypted), new channel:8, old channel:0
I (140786) mesh: [FIND][ch:0]AP:11, otherID:0, MAP:1, idle:1, candidate:0, root:0[00:17:13:20:bd:74]router found
I (140796) mesh: [FIND:1]find a network, channel:8, cfg<channel:0, router:VONETS, 00:00:00:00:00:00>

I (00:58:07.590) aWifiMesh: <MESH_EVENT_FIND_NETWORK>new channel:8, router BSSID:00:00:00:00:00:00
W (140796) wifi:<MESH AP>adjust channel:1, secondary channel offset:1(40U)
W (140816) wifi:<MESH AP>adjust channel:8, secondary channel offset:1(40U)
I (141126) mesh: [SCAN][ch:8]AP:1, other(ID:0, RD:0), MAP:0, idle:0, candidate:1, root:0, topMAP:0[c:0,i:0][00:17:13:20:bd:74]router found<>
I (141126) mesh: 1330[SCAN]init rc[48:ca:43:9b:53:d9,-9], mine:0, voter:0
I (141136) mesh: 1368, vote myself, router rssi:-9 > voted rc_rssi:-120
I (141146) mesh: [SCAN:1/10]rc[128][48:ca:43:9b:53:d9,-9], self[48:ca:43:9b:53:d8,-9,reason:0,votes:1,idle][mine:1,voter:1(1.00)percent:1.00][128,1,48:ca:43:9b:53:d9]

I (141456) mesh: [SCAN][ch:8]AP:2, other(ID:0, RD:0), MAP:1, idle:1, candidate:1, root:0, topMAP:0[c:0,i:1][00:17:13:20:bd:74]router found<>
I (141466) mesh: [SCAN:2/10]rc[128][48:ca:43:9b:53:d9,-8], self[48:ca:43:9b:53:d8,-8,reason:0,votes:1,idle][mine:1,voter:2(0.50)percent:1.00][128,1,48:ca:43:9b:53:d9]

I (141776) mesh: [SCAN][ch:8]AP:2, other(ID:0, RD:0), MAP:1, idle:0, candidate:1, root:1, topMAP:0[c:0,i:0][00:17:13:20:bd:74]router found<>
I (141776) mesh: 7391[selection]try rssi_threshold:-78, backoff times:0, max:5<-78,-82,-85>
I (141796) mesh: [DONE]connect to parent:ESPM_3372B8, channel:8, rssi:-15, 30:30:f9:33:72:b9[layer:1, assoc:0], my_vote_num:0/voter_num:0, rc[00:00:00:00:00:00/-8/0]
I (141806) mesh: set router bssid:00:17:13:20:bd:74
I (142596) mesh: <MESH_NWK_MIE_CHANGE><><><><ROOT ADDR><><><>
I (142596) mesh: <MESH_NWK_ROOT_ADDR>from assoc, layer:2, root_addr:30:30:f9:33:72:b9, root_cap:1
I (142616) mesh: <MESH_NWK_ROOT_ADDR>idle, layer:2, root_addr:30:30:f9:33:72:b9, conflict_roots.num:0<>
I (00:58:09.409) aWifiMesh: <MESH_EVENT_ROOT_ADDRESS>root address:30:30:f9:33:72:b9
I (142616) mesh: [scan]new scanning time:600ms, beacon interval:300ms
I (142636) mesh: 2012<arm>parent monitor, my layer:2(cap:6)(node), interval:7286ms, retries:1<normal connected>
I (00:58:09.436) aWifiMesh: <MESH_EVENT_PARENT_CONNECTED>layer:1-->2, parent:30:30:f9:33:72:b9<layer2>, ID:77:77:77:77:77:76
I (00:58:09.451) mesh_netif: It was a wifi station removing stuff
Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.

My application integrates a number of IDF example programs including ip_internal_network
I went back to the example project ip_internal_network and built it unmodified, and can reproduce the same problems quite readily.
Also, for when the ESP32 nodes don't completely crash, I would like to know how to restart the Mesh network in software.
I have tried stopping the Mesh network with:
ESP_ERROR_CHECK(esp_mesh_stop());
ESP_ERROR_CHECK(esp_mesh_deinit());
ESP_ERROR_CHECK(mesh_netifs_destroy()); // I have tried with and without this line. Without it, the logs continually report:
I (135746) mesh: mesh is not started
E (00:58:02.547) mesh_netif: Received with err code 16388 ESP_ERR_MESH_NOT_START
I then try to restart the Mesh network with:
/*  mesh initialization /
ESP_ERROR_CHECK(esp_mesh_init());
ESP_ERROR_CHECK(esp_mesh_set_max_layer(CONFIG_MESH_MAX_LAYER));
ESP_ERROR_CHECK(esp_mesh_set_vote_percentage(1));
ESP_ERROR_CHECK(esp_mesh_set_ap_assoc_expire(10));
/ set blocking time of esp_mesh_send() to 30s, to prevent the esp_mesh_send() from permanently for some reason /
ESP_ERROR_CHECK(esp_mesh_send_block_time(5000)); // was 30 seconds
mesh_cfg_t cfg = MESH_INIT_CONFIG_DEFAULT();
cfg.crypto_funcs = NULL;
/ mesh ID */
memcpy((uint8_t ) &cfg.mesh_id, MESH_ID, MAC_SIZE);
/ router */
cfg.channel = CONFIG_MESH_CHANNEL;
cfg.router.ssid_len = strlen(meshProvisionData.ssid);
memcpy((uint8_t *) &cfg.router.ssid, meshProvisionData.ssid, cfg.router.ssid_len);
memcpy((uint8_t *) &cfg.router.password, meshProvisionData.password, strlen(meshProvisionData.password));

ESP_ERROR_CHECK(esp_mesh_set_ap_authmode((wifi_auth_mode_t) CONFIG_MESH_AP_AUTHMODE));
cfg.mesh_ap.max_connection = CONFIG_MESH_AP_CONNECTIONS;
cfg.mesh_ap.nonmesh_max_connection = CONFIG_MESH_NON_MESH_AP_CONNECTIONS;
memcpy((uint8_t *) &cfg.mesh_ap.password, CONFIG_MESH_AP_PASSWD, strlen(CONFIG_MESH_AP_PASSWD));
ESP_ERROR_CHECK(esp_mesh_set_config(&cfg));
ESP_ERROR_CHECK(esp_mesh_start());

Doing the above when the system is running normally, often causes the ESP32's to crash with various errors
eg after start, MESH_NODE does a scan and then crashes with Guru Meditation Error: Core  0 panic'ed
I (00:22:02.752) aWifiMesh: <MESH_EVENT_FIND_NETWORK>new channel:8, router BSSID:00:00:00:00:00:00
W (1323864) wifi:adjust channel:1, secondary channel offset:1(40U)
W (1323874) wifi:adjust channel:8, secondary channel offset:1(40U)
I (1324184) mesh: [SCAN][ch:8]AP:2, other(ID:0, RD:0), MAP:1, idle:0, candidate:1, root:1, topMAP:0[c:0,i:0][00:17:13:20:bd:74]router found<>
I (1324184) mesh: 7391[selection]try rssi_threshold:-78, backoff times:0, max:5<-78,-82,-85>
I (1324204) mesh: [DONE]connect to parent:ESPM_3372B8, channel:8, rssi:-14, 30:30:f9:33:72:b9[layer:1, assoc:0], my_vote_num:0/voter_num:0, rc[00:00:00:00:00:00/-120/0]
I (1324214) mesh: set router bssid:00:17:13:20:bd:74
I (1324834) mesh: <MESH_NWK_MIE_CHANGE><><><><><><>
I (1324834) mesh: <MESH_NWK_ROOT_ADDR>from assoc, layer:2, root_addr:30:30:f9:33:72:b9, root_cap:1
I (1324844) mesh: <MESH_NWK_ROOT_ADDR>idle, layer:2, root_addr:30:30:f9:33:72:b9, conflict_roots.num:0<>
I (1324854) mesh: [scan]new scanning time:600ms, beacon interval:300ms
I (00:22:03.744) aWifiMesh: <MESH_EVENT_ROOT_ADDRESS>root address:30:30:f9:33:72:b9
I (1324854) mesh: 2012parent monitor, my layer:2(cap:6)(node), interval:4526ms, retries:1
I (00:22:03.771) aWifiMesh: <MESH_EVENT_PARENT_CONNECTED>layer:2-->2, parent:30:30:f9:33:72:b9, ID:77:77:77:77:77:76
I (00:22:03.785) mesh_netif: It was a wifi station removing stuff
Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.
Core  0 register dump:
PC      : 0x4212753c  PS      : 0x00060830  A0      : 0x82127613  A1      : 0x3fcc15d0
A2      : 0xffffffff  A3      : 0x00000000  A4      : 0x00000278  A5      : 0x00000001
A6      : 0x3fcc09d0  A7      : 0x00000278  A8      : 0x3c1505e4  A9      : 0x3fcd778c
A10     : 0x3fcc09d0  A11     : 0x00000000  A12     : 0x00000101  A13     : 0x3c1505e4
A14     : 0x00000007  A15     : 0x3fcaa7f4  SAR     : 0x00000004  EXCCAUSE: 0x0000001c
EXCVADDR: 0x00000284  LBEG    : 0x40056f5c  LEND    : 0x40056f72  LCOUNT  : 0xffffffff
Backtrace: 0x42127539:0x3fcc15d0 0x42127610:0x3fcc1620 0x4037e0aa:0x3fcc1640
I sometimes get MTX task stack overflows too when I try this, same as https://github.com/espressif/esp-idf/issues/13882

@michaelsimp
I have tested using the ip_internal_network example, but I didn't reproduce your problem. Can you provide the .elf file when the crash issue happen? Or can you provide the core dump decode file?

Hi

  Thanks for the response.

  I have a deadline for a project demo this week but I will go back
  and reinstall from scratch and rebuild and test and send you the
  files requested.

  One thing I note is that I originally installed IDF into vscode
  when 5.2.2 was the current build.  I have since upgraded to ver
  5.3.0 and selected this and did a clean build on my  project.

  Now I have gone back to "show examples" to recreate the
  ip_internal_network project from scratch, it asks which version of
  the IDF I want to use but only lists ver 5.2.2.  Only once I
  create the project I can change the version of IDF to 5.3.0

  Using winmerge I compared the directory structure and files from
  my target folder containing ip_internal_network
  from ver 3.2.2, with the
  ...\esp\v5.3\esp-idf\examples\mesh\ip_internal_network folder and
  the only change was in the partition.csv which now must be
  aligned.

  I also note there is now a version 5.3.1 marked as stable, and
  5.3.0 has gone?  I also found migration notes for going from 5.3
  to 5.4 although I can't find this in the installer. Is 5.4 the
  master (development branch).

  Which version would you recommend I use?  I have found everything
  in IDF to be stable and bug free except for the WiFi Mesh.

  Could you advise how I update IDF properly so it will enable me to
  select latest version of IDF when I create project from "Show
  Examples".
  Last time I followed the instructions from Visual Studio at:

https://marketplace.visualstudio.com/items?itemName=espressif.esp-idf-extension#:~:text=In Visual Studio Code%2C select,not supported inside configured paths
  Regards
  Michael

On 22/10/2024 10:57 pm, ZYJ wrote:

  @michaelsimp
    I have tested using the ip_internal_network
    example, but I didn't reproduce your problem. Can you provide
    the .elf file when the crash
    issue happen? Or can you provide the core dump decode file?
  —
    Reply to this email directly, view it on GitHub, or unsubscribe.

I created the ip_internal_network project from examples and configured as follows:
Set IDF version to 5.3.0
Set target device to ESP32-S3 with jtag integrated debugger
Set partition table Factory partition to 0x400000
Set device flash size to 16MB - matches my ESP32-S3
Set the Router SSID to "VONETS" and Password to "pass9999"
Set Panic Handler to "Print registers and halt"
Clean and Build project.
Load into 2 ESP32-S3 with serial terminals connected for monitoring.
One becomes MESH_ROOT and the other connects as MESH_NODE
Took turns at powering off the MESH_ROOT and watching the other become MESH_ROOT and then powering it back on and it connects as a MESH_NODE. This seemed to work ok today.
But what I did find easy to reproduce was:
Power off MESH_ROOT and power back on BEFORE other MESH_NODE becomes MESH_ROOT.
The original MESH_ROOT I power cycled, becomes MESH_ROOT again, but the MESH_NODE remains disconnected.
See attached files:
MESH_ROOT powered off at line 171
MESH_NODE loses connection around line 33 and never recovers
Also see .elf and .bin files in attachment.  I don't know what or where the "Core dump decode file is", but these tests don't show a CPU crash.  I cant run under the debugger due to the power cycle tests.

I did some more tests which can easily cause CPU crashes.
Power on both nodes, one becomes MESH_ROOT and one MESH_NODE.
Power off WIFI router.
MESH_ROOT crashes. See file attached MESH_ROOT "Router power off.txt" (crash at end)
Second test. Power up only one node, becomes MESH_ROOT
Power off Router
This time MESH_ROOT does not crash
Power on Router
MESH_ROOT does not crash
Power on a second node which connects to the first MESH_ROOT - see line 492
MESH_ROOT crashes.
See file attached "MESH_ROOT crash on MESH_NODE connect.txt"
MESH_ Testing 2.zip

MESH-Testing.zip
This is the attachment for the first tests, 2 entries up. It did not upload properly last before

Hi @michaelsimp
The esp-idf vscode extension allows you to save settings in multiple places: User (Global settings for vscode), Workspace and Workspace folder (your project's .vscode/settings.json). The ESP-IDF: Show Examples command shows you the current esp-idf path used in the current vscode window. You can change where to save settings with the ESP-IDF: Select where to save configuration settings command. It sounds confusing but it does allow to use multiple projects each with different esp-idf versions (even at the same time! Using vscode workspace) More information in here
It seems the example you are trying to use have some components with specific behavior in each esp-idf version. So building a v5.2.2 example using esp-idf v5.3 might produce some compilation problems. How about creating an example using esp-idf v5.3 ?

Open a vscode window.
Select esp-idf v5.3 from status bar (recommended) or the ESP-IDF: Configure ESP-IDF extension.
Run the ESP-IDF: Doctor command. Check that esp-idf is indeed using v5.3
Run the ESP-IDF: Show examples. The esp-idf path shown should be v5.3 now
Create your project from esp-idf example and try to build.

We will try to update the Show examples command to show all available esp-idf versions from esp-idf vscode extension to make this easier.

@michaelsimp The backtrace of the crash issue is here:
xtensa-esp32s3-elf-addr2line -piaf 0x4208c922:0x3fca7e60 0x4201c951:0x3fca7ee0 0x4201f96f:0x3fca7f20 0x4202345e:0x3fca7f50 0x420167bd:0x3fca7f70 -e ip_internal_network.elf

0x4208c922: parse_msg at C:\Users\micha\Documents\Cybertek\Software\ip_internal_network\build/C:/Users/micha/esp/v5.3/esp-idf/components/lwip/apps/dhcpserver/dhcpserver.c:993
 (inlined by) handle_dhcp at C:\Users\micha\Documents\Cybertek\Software\ip_internal_network\build/C:/Users/micha/esp/v5.3/esp-idf/components/lwip/apps/dhcpserver/dhcpserver.c:1190
 (inlined by) handle_dhcp at C:\Users\micha\Documents\Cybertek\Software\ip_internal_network\build/C:/Users/micha/esp/v5.3/esp-idf/components/lwip/apps/dhcpserver/dhcpserver.c:1106
0x4201c951: udp_input at C:\Users\micha\Documents\Cybertek\Software\ip_internal_network\build/C:/Users/micha/esp/v5.3/esp-idf/components/lwip/lwip/src/core/udp.c:404
0x4201f96f: ip4_input at C:\Users\micha\Documents\Cybertek\Software\ip_internal_network\build/C:/Users/micha/esp/v5.3/esp-idf/components/lwip/lwip/src/core/ipv4/ip4.c:746
0x4202345e: ethernet_input at C:\Users\micha\Documents\Cybertek\Software\ip_internal_network\build/C:/Users/micha/esp/v5.3/esp-idf/components/lwip/lwip/src/netif/ethernet.c:186
0x420167bd: tcpip_thread_handle_msg at C:\Users\micha\Documents\Cybertek\Software\ip_internal_network\build/C:/Users/micha/esp/v5.3/esp-idf/components/lwip/lwip/src/api/tcpip.c:174
 (inlined by) tcpip_thread at C:\Users\micha\Documents\Cybertek\Software\ip_internal_network\build/C:/Users/micha/esp/v5.3/esp-idf/components/lwip/lwip/src/api/tcpip.c:148

I think this issue is caused by the mismatch between your IDF version and the version in the example. Please update your version according to Brain's suggestion and test it again.

I installed IDF version 5.3.1
I had a problem at step 2 in your instructions to entries up states: "Select esp-idf v5.3 from status bar (recommended) or the ESP-IDF: Configure ESP-IDF extension."
When I try is reports, "Open a folder first."
So I open a folder of a the project I made in version IDF 3.0.  Then on the status bar I could select Version 5.3.1
When I run Run the ESP-IDF: Doctor command, I get the following errors:

Extension configuration report has been copied to the clipboard with errors.
Cannot open file ../report.txt. Detail: FIles above 50MB cannot be synchnrozied with extensions.
I checked my report,txt and found it was over 181MB
I tried continuing:
When I select Show examples it only shows 5.3.1 which is good.
I created ip_internal_network, but when completed the status bar reports ESP-IDF v5.2.2 again.
So I tried deleting the large report.txt and trying again.
Same problem, it created a report.txt of 181MB again

Delete this file:
%USERPROFILE%\.vscode\extensions\espressif.esp-idf-extension-VERSION\esp_idf_vsc_ext.log
and try to run ESP-IDF: Doctor command again. Seems that your extension log have been logging a lot and vscode limit.
About the ESP-IDF v5.2.2 again, it is because the newly created project does not set settings when created. You can select the v5.3.0 from status bar again.
Again sorry for this issue, will work to make it easier to use in the next release of esp-idf extension.

I need to make some real progress on this so I have completely uninstalled esp-idf and manually deleted all the ESP and espressif folders including all 3 IDF versions.
I have reinstalled ESP-IDF and only IDF version 5.3.1 to remove all doubt.
I will rebuild and test and report
Thanks

Hi again
Its hard to tell, but seems as if it might be a little more robust, especially with the router power off and on test.
Attachments.zip
But it still crashes, see files attached including my .elf
"Fail 1.txt" is taken from the Mesh_Root
Line 1458 Mesh_Node disconnected
Line 1495 Mesh_Node reconnects
Line 1496 crash
"Fails 2.txt" is taken from the Mesh_Node
Line 1789 Mesh_Root disconnected
Line 1822 Reconnect
Line 1832 Crash divide by zero

It's weird, I have tested it multiple times as you said (the following two cases) and it can connect normally without any crashing issues.

Power on both nodes, one becomes MESH_ROOT and one MESH_NODE. Power off WIFI router. MESH_ROOT crashes. See file attached MESH_ROOT "Router power off.txt" (crash at end)
Second test. Power up only one node, becomes MESH_ROOT Power off Router This time MESH_ROOT does not crash Power on Router MESH_ROOT does not crash Power on a second node which connects to the first MESH_ROOT - see line 492 MESH_ROOT crashes.

I'm using the Github IDF, and I will try to test with the vscode extension

FYI I use vscode for coding and building and
  sometimes JTAG debugging

  Most of the time in testing I am am monitoring serial com port
  using Putty terminals on com ports

  Thanks

On 30/10/2024 4:56 pm, ZYJ wrote:

  It's weird, I have tested it multiple times as you
    said (the following two cases) and it can connect normally
    without any crashing issues.

    Power on both nodes, one becomes MESH_ROOT and one
      MESH_NODE. Power off WIFI router. MESH_ROOT crashes. See file
      attached MESH_ROOT "Router power off.txt" (crash at end)
    Second test. Power up only one node, becomes
      MESH_ROOT Power off Router This time MESH_ROOT does not crash
      Power on Router MESH_ROOT does not crash Power on a second
      node which connects to the first MESH_ROOT - see line 492
      MESH_ROOT crashes.

  I'm using the Github IDF, and I will try to test
    with the vscode extension
  —
    Reply to this email directly, view it on GitHub, or unsubscribe.
    You are receiving this because you were mentioned.Message
      ID<EMAIL_ADDRESS>  [

{
@.": "http://schema.org",
@.": "EmailMessage",
"potentialAction": {
@.": "ViewAction",
"target": "https://github.com/espressif/esp-idf/issues/14720#issuecomment-2445782877",
"url": "https://github.com/espressif/esp-idf/issues/14720#issuecomment-2445782877",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
@.": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]

@michaelsimp
Are you using the completely unmodified example code during your testing?

Yes.  I did not change anything except:

set target device to ESP32-S3 - Internal JTAG debug - What device are you testing with?  Could this be a factor?
change partition table, factory partition size to 0x400000
Using vscode GUI menuconfig:

change device flash size to 15MB
set WiFi router SSID to VONETS and password to "pass9999"
See my source files attached where you can see most are untouched with the original install date 30/10/24 02:13pm NZ time.
Could someone please answer the questions in my message above.

FYI I use vscode for coding and building and sometimes JTAG debugging
Most of the time in testing I am am monitoring serial com port using Putty terminals on com ports
In addition to the crashing, sometimes a MESH_NODE will not reconnect to the MESH network.  As a work around for this, I want to stop and restart the wifi mesh network?  No matter what I try, my nodes intermittently crash on restart?  Is this perhaps related?
I am hoping you could please answer a few questions to help me.
What are the recommended steps to stop and then restart the WiFi Mesh network?  I currently have :
    ESP_ERROR_CHECK(esp_mesh_stop());
    ESP_ERROR_CHECK(esp_mesh_deinit());

but this causes a lot of error logging which stops if I add ...
    ESP_ERROR_CHECK(mesh_netifs_destroy());
My restart is as follow:
void wifiMeshStart() {
    ESP_LOGW(TAG, "Wifi Mesh switch on");
    /*  mesh initialization */
    ESP_ERROR_CHECK(esp_mesh_init());
    ESP_ERROR_CHECK(esp_mesh_set_max_layer(CONFIG_MESH_MAX_LAYER));
    ESP_ERROR_CHECK(esp_mesh_set_vote_percentage(1));
    ESP_ERROR_CHECK(esp_mesh_set_ap_assoc_expire(10));
    /* set blocking time of esp_mesh_send() to 30s, to prevent the esp_mesh_send() from permanently for some reason */
    ESP_ERROR_CHECK(esp_mesh_send_block_time(30000));
    mesh_cfg_t cfg = MESH_INIT_CONFIG_DEFAULT();
#if !MESH_IE_ENCRYPTED
    cfg.crypto_funcs = NULL;
#endif
    /* mesh ID */
    memcpy((uint8_t *) &cfg.mesh_id, MESH_ID, MAC_SIZE);
    /* router */
    cfg.channel = CONFIG_MESH_CHANNEL;

    cfg.router.ssid_len = strlen(meshProvisionData.ssid);
    memcpy((uint8_t *) &cfg.router.ssid, meshProvisionData.ssid, cfg.router.ssid_len);
    memcpy((uint8_t *) &cfg.router.password, meshProvisionData.password, strlen(meshProvisionData.password));

    ESP_ERROR_CHECK(esp_mesh_set_ap_authmode((wifi_auth_mode_t) CONFIG_MESH_AP_AUTHMODE));
    cfg.mesh_ap.max_connection = CONFIG_MESH_AP_CONNECTIONS;
    cfg.mesh_ap.nonmesh_max_connection = CONFIG_MESH_NON_MESH_AP_CONNECTIONS;
    memcpy((uint8_t *) &cfg.mesh_ap.password, CONFIG_MESH_AP_PASSWD, strlen(CONFIG_MESH_AP_PASSWD));
    ESP_ERROR_CHECK(esp_mesh_set_config(&cfg));
    /* mesh start */
    ESP_ERROR_CHECK(esp_mesh_start());
    ESP_LOGI(TAG, "WiFi Mesh started successfully");
}

I notice with my custom application:
I have 1 MESH_ROOT and 5 MESH_NODEs spread around the office.  Mesh_Node seem to connect to parents not based on the RSSI as documented.
The IDF documentation states "To prevent nodes from forming a weak upstream connection, ESP-WIFI-MESH implements an RSSI threshold mechanism for beacon frames."   Is this configurable and if so where?  I cant find it in the API or in MenuConfig.  What is the default RSSI threshhold value?
The IDF documentation states in Preferred Parent Node "The preferred parent node is determined based on the following criteria: Which layer the parent node candidate is situated on. The number of downstream connections (child nodes) the parent node candidate currently has".
Does this mean RSSI is not part of the parent selection process?
Is it recommend to use self-organized networking or for serious applications should I manually build the mesh network? I will only have a max of 10 mesh nodes altogether but they do a reasonable amount of MQTT5 communications to the cloud.

@michaelsimp  The following are the answers for your questions:

What are the recommended steps to stop and then restart the WiFi Mesh network? I currently have :
    ESP_ERROR_CHECK(esp_mesh_stop());
    ESP_ERROR_CHECK(esp_mesh_deinit());

Call esp_mesh_stop() is enough.

but this causes a lot of error logging which stops if I add ...
 ESP_ERROR_CHECK(mesh_netifs_destroy());
Where did you add the mesh_netifs_destory() function? What does the error log look like? Can you provide an example？

Where did you call the wifiMeshStart() function?

I have 1 MESH_ROOT and 5 MESH_NODEs spread around the office. Mesh_Node seem to connect to parents not based on
the RSSI as documented.

RSSI is not the only criterion for selecting the parent node, the layer and connections also need to be considered.

The IDF documentation states "To prevent nodes from forming a weak upstream connection, ESP-WIFI-MESH implements an RSSI threshold mechanism for beacon frames." Is this configurable and if so where? I cant find it in the API or in MenuConfig. What is the default RSSI threshhold value?

You can call this API:
https://github.com/espressif/esp-idf/blob/9106c43accd9f5e75379f62f12597677213f5023/components/esp_wifi/include/esp_mesh_internal.h#L216

The IDF documentation states in Preferred Parent Node "The preferred parent node is determined based on the following criteria: Which layer the parent node candidate is situated on. The number of downstream connections (child nodes) the parent node candidate currently has". Does this mean RSSI is not part of the parent selection process?

Same to the fourth point, selecting parent need to consider RSSI, layer and connections, the doc need to be updated.

Is it recommend to use self-organized networking or for serious applications should I manually build the mesh network? I will only have a max of 10 mesh nodes altogether but they do a reasonable amount of MQTT5 communications to the cloud.

You can use self-organized network.

@michaelsimp I can reproduce the crash using the vscode, I will check the difference between VSCode and standard IDF

Hi Zhangyanjiaoesp
This excellent news for me.  Hopefully it is just something simple you will find soon and be able to offer me a fix.
Thank you

Hi Zhangyanjiaoesp
Thanks so much for taking the time to answer all my questions.
Please note THESE tests are with MY application (NOT with example program ip_internal_network) running on a network of 6 nodes - all ESP32-S3.  My application has a CLI console integrated so I can trigger actions and see the responses on the COM port.
Q1 I will go back to just calling esp_mesh_stop() and see what happens.
To restart should I just be able to call ESP_ERROR_CHECK(esp_mesh_start());
Q2 Triggered from the CLI Console I was calling:
    ESP_ERROR_CHECK(esp_mesh_stop());
    ESP_ERROR_CHECK(esp_mesh_deinit());
    ESP_ERROR_CHECK(mesh_netifs_destroy());

Q3 wifiMeshStart() is also called from my CLI console
The CLI Console is started from my Mainline as is my WiFi Mesh application (built on top of the ip_internal_network source).
Triggering the Mesh Stop and Start would be called from the CLI Console thread.  I am assuming this is ok and does not need any mutex protection.  Please advise how I should call it if this is a problem.
Q4     I understand this
Q5 Thanks
Q6 Thanks for clarifying this but I am not finding this to be the case.  I distributed some MESH_Nodes across the office with the aim of creating a multi-hop network between the far extremes. But it does not form as expected or at all well for healthy RSSI.  I have nodes which are close to my Mesh_Root or 2nd layer Mesh_Nodes which are not at parent capacity numbers.  When my other Mesh_Nodes do connect to these parents they provide an RSSI on the child to the parent of -35dBm.  But they most frequently want connect to nodes a much longer distance away getting a RSSI of < -70dBm.
I read somewhere the default RSSI threshold is -120dBm, but I am finding nodes with RSSIs < -70dBm often lose their MQTT connection to the broker.  I have an office environment and I have located the nodes approx 10 to 20 meters apart with a max of one wall between but they are not all line of site.  I very much doubt I could even get a connection at RSSI less than -100dBm.  I am thinking it may be a signal to noise ratio issue so I have scanned the office for WiFi channel usage and selected channel 1 on my WIFI Router as nothing else is using this channel and no other channels overlap.  I know this is not an easy question to answer with precision and I appreciate the many influences, but realistic what is the ballpark min RSSI range at which I can expect a node to work reliably at what sort of distance range.
Q7  Because of my Q6 response above, I have started evaluating the example project "manual_networking" to make my MESH_NODES manually scan and select MESH_NODES with the healthiest RSSI.  It sounds like you are saying the IDF framework should already be doing this?
So I am now wondering if the vscode crashing issue is also causing this to not work properly and your fix might fix both.
Should I put manual scanning and parent selection changes to one side and wait for the outcome of the vscode crashing?
I guess I would prefer to use the self-organized network as much as possible, if it works as you describe.
Please advise / confirm manual scanning and parent selection should not be required and I might need you to look at the node parent selection for healthy RSSI next.
Best regards

@michaelsimp

According to the backtrace of the crash issue, it seems related to DHCP, it won't affect the mesh networking.
I'm sorry I didn't quite understand your question regarding the selection of the parent node. Can you draw a picture to explain it? For example, where are nodes A, B, and C located? What level? How many child nodes are connected below? What is the RSSI of A, B, and C scanning each other? Do you expect A to connect to B but actually connect A to C?

See attached:
MeshMaps.zip
"Target Network .png" shows the walls as black lines and nodes as circles.  The blue lines are approx what I was expecting to see.
It forms very randomly but with bad choices like the file "Actual example Network.png" with links and dBm in red.
My project is configured for up to 50 nodes and  3 children per node as I wanted to force some layers.

"According to the backtrace of the crash issue, it seems related to DHCP, it won't affect the mesh networking."
That may be the case with the crash issue you found, but the root cause of the vscode IDF environment may cause more than one issue.  I guess you will know better when you get to the bottom of the vscode IDF environment issue
Do the diagrams I sent help you understand my issue better now?

Yes, I now understand your question. Once the ROOT node is formed, the chances for other idle nodes to connect to the root node are the same; as long as they can scan the root node within the RSSI threshold range, they can connect and become second-layer nodes. Therefore, it is reasonable for C and D to connect to A and become second-layer nodes. What is the RSSI that B, E, and F receive from A? Since each node can connect to 3 child nodes, if they are within the RSSI threshold range and root A is not yet fully connected, at least one of B, E, or F should be able to connect to A.
I think you can call esp_mesh_set_rssi_threshold() to limit the RSSI threshold for optional parents, which would allow nearby nodes to connect as much as possible. However, nodes D and E are too far from node A. If you set the same RSSI threshold for all nodes, the connection results might still not meet your expectations, unless you configure different RSSI thresholds for each node. Alternatively, you could only call the esp_mesh_set_parent() function to specify the parent of each node.

My main concern is F connecting to C when it should go to E or A or B, and also E Connecting to D instead of A or B.
Raising the threshold should prevent this but will prevent connection is some nodes go off, causing more problems.
I think I should have my nodes scan manually for a list of eligible parents and select the one with the best RSSI.  Do nodes only become eligible to become parents once they have a connection to a parent?
Can I make a node scan after it has connected to a parent to look for a better connection if its RSSI is lower than a useful connection of say -65dBm.  If I only do this occasionally, is there a down side?
Is the RSSI an average value, or from the latest message received from the parent or something else?

Do nodes only become eligible to become parents once they have a connection to a parent?

Yes, only the idle nodes are eligible to become parent nodes.

Can I make a node scan after it has connected to a parent to look for a better connection if its RSSI is lower than a useful connection of say -65dBm. If I only do this occasionally, is there a down side?

Of cause you can switch the parent node as you want. In the self-organized mesh networking, the node can switch parent autonomously, see the doc.

Is the RSSI an average value, or from the latest message received from the parent or something else?
The RSSI is from the latest beacon from the parent.

@michaelsimp For the crash issue, can you please cherry-pick this branch and test again?

Hi
Today I merged some of the code from the example project "manual_networking" into my project.
I start everything up self organized.
Once connected, I have a task which checks if MESH_NODEs and MESH_LEAFs have a parent connection < 60dBm.  If so they run:
    ESP_ERROR_CHECK(esp_mesh_set_self_organized(0, 0));
    esp_wifi_scan_stop();
    wifi_scan_config_t scan_config = { 0 }; // mesh softAP is hidden
    scan_config.show_hidden = 1;
    scan_config.scan_type = WIFI_SCAN_TYPE_PASSIVE;
    esp_wifi_scan_start(&scan_config, 0);

On MESH_EVENT_SCAN_DONE I added mesh_scan_done_handler(scan_done->number);
I modified mesh_scan_done_handler() from "manual_networking" so that it will not stop on the first eligible parent.  It keeps scanning the entire list and takes the one with the highest RSSI.
The results are really good.  Please see attached my mesh mapping software output:

Left map is typical of self organized.
Right map is the same setup but with the new rescan to select better parents.

mapping.zip
Thanks for helping by answering my questions.

Its the end of my week but I will try to test the mesh_netif.c over the weekend

Hi Zhangyanjiaoesp
No its still crashes.
Quite reproducible if your get two nodes connected and then power off the root for a couple of seconds and back on.
See attached zip with my mesh_netif.c source file and the bin and elf files for you to check.
Also Crash1.txt with the MESH_NODE crashing line 182 IntegerDivideByZero after the test above.

Perhaps related, I regularly get "Send with err code 16392 ESP_ERR_MESH_ARGUMENT" from mesh_netif:
The source code has 3 instances which could print this error message. I added some diagnostics to prove it comes from the function: static esp_err_t mesh_netif_transmit_from_node_sta(void *h, void *buffer, size_t len)
The example below shows my code MQTT publishing a test message once / second cleanly and then I get a bunch of "Send with err code 16392 ESP_ERR_MESH_ARGUMENT" messages, and then it goes clean again.
What does this mean?
...
I (11:41:13.293) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/6/PumpStatus Payload: Status=12&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=6&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
I (11:41:14.302) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/7/PumpStatus Payload: Status=13&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=7&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
I (11:41:15.303) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/8/PumpStatus Payload: Status=14&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=8&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
I (11:41:16.313) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/9/PumpStatus Payload: Status=15&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=9&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
I (11:41:17.313) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/10/PumpStatus Payload: Status=16&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=10&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
I (11:41:18.313) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/1/PumpStatus Payload: Status=0&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=1&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
I (259339) mesh: 5130<assoc>parent layer:1, channel:1, rssi:-44, assoc:4, rssi threshold<-78,-82,-85>
E (11:41:21.973) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
E (11:41:21.999) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
I (11:41:22.323) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/5/PumpStatus Payload: Status=4&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=5&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
E (11:41:22.356) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
E (11:41:22.723) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
E (11:41:22.755) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
E (11:41:23.223) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
I (11:41:23.333) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/6/PumpStatus Payload: Status=5&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=6&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
E (11:41:23.362) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
E (11:41:23.723) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
E (11:41:23.765) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
I (11:41:24.333) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/7/PumpStatus Payload: Status=6&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=7&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
E (11:41:24.723) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
E (11:41:25.723) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
E (11:41:26.223) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
E (11:41:26.723) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
E (11:41:27.723) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
E (11:41:27.917) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
E (11:41:27.973) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
I (11:41:19.319) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/2/PumpStatus Payload: Status=1&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=2&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
START MQTT CONSOLE REDIRECT
I (11:41:34.675) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/8/PumpStatus Payload: Status=7&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=8&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
I (11:41:34.692) mqtt5_client: MQTT_MSG_TYPE_SUBACK return code is 1
I (11:41:34.709) aMQTT: MQTT_EVENT_SUBSCRIBED, msg_id=345  inFlightCounter: 0
I (11:41:20.323) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/3/PumpStatus Payload: Status=2&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=3&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
I (11:41:35.693) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/9/PumpStatus Payload: Status=8&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=9&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
I (11:41:36.693) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/10/PumpStatus Payload: Status=9&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=10&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
I (11:41:37.695) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/1/PumpStatus Payload: Status=10&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=1&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
I (11:41:38.713) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/2/PumpStatus Payload: Status=11&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=2&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
I (11:41:21.323) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/4/PumpStatus Payload: Status=3&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=4&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
I (11:41:39.713) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/3/PumpStatus Payload: Status=12&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=3&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
I (11:41:40.714) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/4/PumpStatus Payload: Status=13&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=4&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
E (11:41:21.585) mesh_netif: C Send with err code 16392 ESP_ERR_MESH_ARGUMENT
I (11:41:41.723) aMQTT: Console Publish: Topic: SiteID/00003030F93349BC/Fuelpoint/5/PumpStatus Payload: Status=14&TransType=1&State=3&TransactionSeqNo=123456&Hose=2&Product=5&Money=12345.67&Volume=987654.32&UnitPrice=12.34&LimitTyp=1&LimitAmount=100.00
...

Regarding Q1 and Q@ from 31 October (above), I still can't get stopping and restarting the wifi_mesh to work.
Per your advice I tried just calling function esp_mesh_stop() - this is triggered from my CLI console
This results in lots of ESP_LOG messages:  information warning and error .
and if I just call esp_mesh_start() the app generally crashes.  This is the same as when I called my original more complex restart procedure (see below):
See WifiStopStar.txt logs attached
WifiStopStart.txt
Can you have a rethink and test yourself if necessary and advise how I can make this work.  I have 2 needs:

to try to fix any wifi mesh issues without rebooting the entire app.  Sometimes a node will just not crash, but it just stops all wifi mesh functionailty
to suspend wifimesh coms when not required

void wifiMeshStart() {
    ESP_LOGW(TAG, "Wifi Mesh switch on");
    /*  mesh initialization */
    ESP_ERROR_CHECK(esp_mesh_init());
    ESP_ERROR_CHECK(esp_mesh_set_max_layer(CONFIG_MESH_MAX_LAYER));
    ESP_ERROR_CHECK(esp_mesh_set_vote_percentage(1));
    ESP_ERROR_CHECK(esp_mesh_set_ap_assoc_expire(10));
    /* set blocking time of esp_mesh_send() to 30s, to prevent the esp_mesh_send() from permanently for some reason */
    ESP_ERROR_CHECK(esp_mesh_send_block_time(30000));
    mesh_cfg_t cfg = MESH_INIT_CONFIG_DEFAULT();
#if !MESH_IE_ENCRYPTED
    cfg.crypto_funcs = NULL;
#endif
    /* mesh ID */
    memcpy((uint8_t *) &cfg.mesh_id, MESH_ID, MAC_SIZE);
    /* router */
    cfg.channel = CONFIG_MESH_CHANNEL;

    cfg.router.ssid_len = strlen(meshProvisionData.ssid);
    memcpy((uint8_t *) &cfg.router.ssid, meshProvisionData.ssid, cfg.router.ssid_len);
    memcpy((uint8_t *) &cfg.router.password, meshProvisionData.password, strlen(meshProvisionData.password));

    ESP_ERROR_CHECK(esp_mesh_set_ap_authmode((wifi_auth_mode_t) CONFIG_MESH_AP_AUTHMODE));
    cfg.mesh_ap.max_connection = CONFIG_MESH_AP_CONNECTIONS;
    cfg.mesh_ap.nonmesh_max_connection = CONFIG_MESH_NON_MESH_AP_CONNECTIONS;
    memcpy((uint8_t *) &cfg.mesh_ap.password, CONFIG_MESH_AP_PASSWD, strlen(CONFIG_MESH_AP_PASSWD));
    ESP_ERROR_CHECK(esp_mesh_set_config(&cfg));
    /* mesh start */
    ESP_ERROR_CHECK(esp_mesh_start());
    ESP_LOGI(TAG, "WiFi Mesh started successfully");
}

Hi Zhangyanjiaoesp No its still crashes. Quite reproducible if your get two nodes connected and then power off the root for a couple of seconds and back on. See attached 3Nov24-1.zip with my mesh_netif.c source file and the bin and elf files for you to check. Also Crash1.txt with the MESH_NODE crashing line 182 IntegerDivideByZero after the test above. 3Nov24-1.zip

@michaelsimp Because I am not always able to reproduce the problem when testing with VSode, I have provided you with the following debug version:

Replace the wifi lib in the idf/components/espw_ifi/lib/esp32s3 path with the wifi lib in the attachment
add the patch in dhcpserver.c
wifi_lib_s3_1104.zip
0001-fix-dhcp-add-debug-log-for-dhcp-server.zip

Send with err code 16392 ESP_ERR_MESH_ARGUMENT this log indicates that the parameter error passed into esp_mesh_send() function.

For mesh, stop ->init ->config ->start...
I am not sure what this means and how I stop it crashing.
The only WiFi comms I am doing during the test is via the MQTT API with an MQTT5 TLS connection.
Before I call esp_mesh_stop(), I set a flag to stop publishing any more MQTT messages.  MQTT is still active and I guess heartbeating with the broker, so I will try disconnecting from the broker tomorrow and retest.
The ->init ->config ->start code I call after esp_mesh_stop() is the exact same code as I call to initially start the WiFi mesh network.  But It is ALWAYS successful on initial startup and ALWAYS fails after calling esp_mesh_stop().
Would you be able to add some simple code to the example project "ip_internal_network" which after say 30 seconds of running, calls esp_mesh_stop() and after 10 seconds restarts the mesh again to show me how it should work. Note I can't stop and restart the mesh on either the MESH_ROOT or MESH_NODE.

Could this also be related to the crashing issue as it is also in mesh_netif.

I'm not sure, but I think it shouldn't matter.

For mesh, stop ->init ->config ->start... I am not sure what this means and how I stop it crashing. The only WiFi comms I am doing during the test is via the MQTT API with an MQTT5 TLS connection. Before I call esp_mesh_stop(), I set a flag to stop publishing any more MQTT messages. MQTT is still active and I guess heartbeating with the broker, so I will try disconnecting from the broker tomorrow and retest.
The ->init ->config ->start code I call after esp_mesh_stop() is the exact same code as I call to initially start the WiFi mesh network. But It is ALWAYS successful on initial startup and ALWAYS fails after calling esp_mesh_stop().
Would you be able to add some simple code to the example project "ip_internal_network" which after say 30 seconds of running, calls esp_mesh_stop() and after 10 seconds restarts the mesh again to show me how it should work. Note I can't stop and restart the mesh on either the MESH_ROOT or MESH_NODE.

I think it would be better if you could provide a demo to reproduce your problem, and we can help check if there are any issues with your code.

Hi
I installed the wifi_lib_s3_1104 folder you sent me today and did a clean build and installed in my two test nodes.  Does this just have diagnostics, or some improvements which chould fix the crashing?  The reason I ask is I could not reproduce the crashing after quite a few cycles approx 20.
To be sure I restored the original library to double test.  But I found it much harder to reproduce the crash today than previously with the original library. Eventually after quite a few power cycIes I caught one - see attached OrigLibraryCrash.txt
So I restored the new wifi_lib_s3_1104 library again and worked really hard at trying to crash it.  I tried many many times maybe 50+ and just could not break it.
5Nov-1.zip
This contains my "ip_internal_network.elf file with the new library installed for your reference.
What I did see from time to time was the MESH_NODE would start logging quite a bit of diagnostics form "wifi" in between the normal "mesh_main" messages. You might have expected this to occur, eg:
I (197688) mesh_main: Tried to publish layer:2 IP:<IP_ADDRESS>
I (197688) mesh_mqtt: sent publish returned msg_id=54636
I (197758) wifi:>>>intv = 102400, max = 0
I (197858) wifi:>>>intv = 102400, max = 0
I (197958) wifi:>>>intv = 102400, max = 0
I (198058) wifi:>>>intv = 102400, max = 0
I (198158) wifi:>>>intv = 102400, max = 0
I (198268) wifi:>>>intv = 102400, max = 0
I (198268) mesh_main: Received Routing table [0] 30:30:f9:33:72:b8
I (198268) mesh_main: Received Routing table [1] 48:ca:43:9b:53:d8
I (198368) wifi:>>>intv = 102400, max = 0
I (198468) wifi:>>>intv = 102400, max = 0
I (198568) wifi:>>>intv = 102400, max = 0
I (198678) wifi:>>>intv = 102400, max = 0
I (198778) wifi:>>>intv = 102400, max = 0
I (198878) wifi:>>>intv = 102400, max = 0

Two example scenarios of this are:
newdiagtrace 1.txt:
This is from the MESH_NODE
Power off and on the MESH_ROOT quickly
MeshNode starts logging heaps of stuff eg see line 1429 onwards
When MESH_ROOT comes back (same Root reestablishes itself after quick power cycle), the MESH_NODE keeps logging stuff.
Then when MESH_ROOT power cycled again, the extra logging on the MESH_NODE stops
newdiagtrace 2.txt:
This is MESH_NODE
Restart the MESH_ROOT multiple times until MESHNODE starts logging heaps of stuff
See line 312
Wait for this MESH_NODE to timeout, disconnect and then establish itself as MESH_ROOT line 475
Logging stops
Neither is causing a problem, I just wanted to give you all the information.
I am very interested to know whether the diagnostics had the potential to fix the crashing as it seems to have from my testing.  Please advise next steps.

@michaelsimp
Yes, I added a fix to the wifi lib I provided to you. wifi:>>>intv = 102400, max = 0 this log was added by me to locate the problem. I'm glad to hear that the problem has been solved.
Have you tested the router reboot case? Does the root node crash still exist?

Yes I have tested the router a few times.  I will need to do some more structured testing but it is looking very good now. Thank you for all your help.  Do you know what the forma release patch will be and when yet?
I also need to resolve the WIFI and MESH stop and restart but I am tied up for the next couple of days.
Can we keep this open until I am ready please.

The patch will merge into master branch firstly and then backported into release/v5.3, we will merge it ASAP.

Can we keep this open until I am ready please.

Of cause, we can keep it open.

Hi
I rebuilt my application with the wifi_lib_s3_1104 folder and did some more testing on my larger network of 6 nodes spread around the office. They all have the same software installed.
I tried resetting the MESH_ROOT node and watching the MESH_Network re-establish.
A lot of the time this worked perfectly but two nodes in particular would sometimes not come back online.
So I reinstalled the "ip_internal_network" app on 4 nodes spread over two locations in the office.
It is still much better than before, but it does still crash.
Crash 6.txt
Attached as an example on my MESH ROOT.
Reset MESH_ROOT
Starts up as MESH_ROOT again and then crashes at line 193.  You have my elf file from yesterdays post
Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.

Core  1 register dump:
PC      : 0x42013528  PS      : 0x00060430  A0      : 0x82013d17  A1      : 0x3fca8280
A2      : 0x00000006  A3      : 0x000034c7  A4      : 0x00060420  A5      : 0x3fca8324
A6      : 0x3fca56ec  A7      : 0x00000000  A8      : 0x00000000  A9      : 0x00000000
A10     : 0x00000006  A11     : 0x00000020  A12     : 0x00060a20  A13     : 0x00060623
A14     : 0x00000009  A15     : 0x0000abab  SAR     : 0x00000004  EXCCAUSE: 0x0000001c
EXCVADDR: 0x0000000d  LBEG    : 0x40056f5c  LEND    : 0x40056f72  LCOUNT  : 0xffffffff

Backtrace: 0x42013525:0x3fca8280 0x42013d14:0x3fca82a0 0x4201f6c1:0x3fca82d0 0x420234ba:0x3fca8300 0x42016819:0x3fca8320

ELF file SHA256: a6faa33f6

I need to find why my application is more prone to crashing than "ip_internal_network".  I suspect it is just busier coms wise.  The only communications it does in this crash test is MQTT5 through the MQTT API.  I subscribe to 3 topics on start up and then no more subscriptions (so none during the MESH_ROOT reboot.  I only publish using esp_mqtt_client_enqueue
So I am at a bit of a loss to know what to look at.  Do you have any suggestions for me or should I wait for another hopeful fix to the library.

@michaelsimp

2. 0001-fix-dhcp-add-debug-log-for-dhcp-server.zip

Have you added this patch?

Hi.
No sorry I missed this instruction your post.  I have just applied it now and am rebuilding.
I have found a reason for my application crashing, but am not sure if it is new as a result of the earlier patch.  I have not noticed up up until now, but these things happen some time.
The crash is caused by calling esp_wifi_sta_get_rssi(&currentRSSI); when the node is of type MESH_IDLE.  This causes a crash which I have not seen before.  The simple fix I have applied is:
if (esp_mesh_get_type() != MESH_IDLE) {
        ESP_ERROR_CHECK(esp_wifi_sta_get_rssi(&currentRSSI));
}

I understand why requesting RSSI if you don't have a parent doesn't make sense, but it shouldn't really cause a crash.  Maybe something at add to your backlog.

My application is looking a lot healthier now, but I still have a couple of issues to resolve.
I was developing and testing with a fixed MESH_ROOT as I can keep this at my desk, while my MESH_NODES are spread around the office.  However my field deployment will need to not have a fixed root for redundancy.
About 5 days ago I posted how I was having my MESH_NODE and MESH_LEAFs scan for closer parents if the RSSI < -60dBm.  This is using code taken from the "manual_networking" sample project.  This works perfectly and is a really good addition, when I have a fixed MESH_ROOT.
But for my field deployment where I need any node to become MESH_ROOT, my code has a problem.
After running my scan and parent swap, on only one MESH_NODE, if I restart the MESH_ROOT, none of my 6 nodes will become MESH_ROOT.  Some nodes might have 4 children showing in their routing table but their Type will be MESH_NODE or MESH_IDLE.
This is my manual scan initiate software
    ESP_ERROR_CHECK(esp_mesh_set_self_organized(0, 0));
    esp_wifi_scan_stop();
    wifi_scan_config_t scan_config = { 0 }; // mesh softAP is hidden
    scan_config.show_hidden = 1;
    scan_config.scan_type = WIFI_SCAN_TYPE_PASSIVE;
    esp_wifi_scan_start(&scan_config, 0);

On MESH_EVENT_SCAN_DONE I added mesh_scan_done_handler(scan_done->number) as in "manual_networking".
I modified mesh_scan_done_handler() from "manual_networking" so that it will not stop on the first eligible parent. It keeps scanning the entire list and takes the one with the highest RSSI.
I tried adding
`ESP_ERROR_CHECK(esp_mesh_set_self_organized(1, 0));`

after the MESH_NODE gets a new parent, but this still doesn't help.
I don't fully understand the API and this example.
Is it possible to start with a self organized network and then make MESH_NODES manually scan for better parents (higher RSSI) and use esp_mesh_set_parent like in mesh_scan_done_handler of manual_networking, and not have a fixed root.
Or do I have to manually scan and set the MESH_ROOT as well.
Can you give me a steer as to how to fix this problem or a better way to meet by goal of forming a MESH with the lowest possible RSSI figures and redundancy of any NODE being able to become MESH_ROOT.
Thanks.

@michaelsimp

This is my manual scan initiate software
    ESP_ERROR_CHECK(esp_mesh_set_self_organized(0, 0));
    esp_wifi_scan_stop();
    wifi_scan_config_t scan_config = { 0 }; // mesh softAP is hidden
    scan_config.show_hidden = 1;
    scan_config.scan_type = WIFI_SCAN_TYPE_PASSIVE;
    esp_wifi_scan_start(&scan_config, 0);

When do you call these operations?

After running my scan and parent swap, on only one MESH_NODE, if I restart the MESH_ROOT, none of my 6 nodes will become MESH_ROOT. Some nodes might have 4 children showing in their routing table but their Type will be MESH_NODE or MESH_IDLE.

Is your root node explicitly specified? Which functions have been called? Do you mean that after the root node restarts, it can no longer remain the root node? And that the original node cannot form the network anymore? Could you provide some logs?

I tried adding
`ESP_ERROR_CHECK(esp_mesh_set_self_organized(1, 0));`

What functionality are you trying to achieve by calling this function?
By the way, to better address the issue, please provide as much comprehensive log information as possible when describing the problem. It would be even more helpful if you could provide a demo.

Hi
I thought it best to go back to the Espressif example project "manual_networking".  These have been build with the 2 earlier updates to the IDF library.  This project does not use "mesh_netif.c"
I made no changes to the source.
I selected device ESP32-S3.
Using Menu Config I:

changed the Flash memory size to 16MB
set Example Configuration device type to "MESH_SET_NODE" as this matches my objectives with the network automatically assigning a MESH_ROOT.
entered my Router SSID and password

I did a clean build and tested.  See attachments for log files, line numbers and summary at top of each file.
Manual test 8Nov.zip
Test one, power 2 nodes up at the same time
Log file "manual test 1" Node crashes, reboots and then becomes root
Log file "manual test 1B" Node won't become root even though the other node is delayed with a crash and restart.  It displays disconnect message for a period of time until the other node becomes MESH_ROT and then it connects to it.
Test two, power 2 nodes one becomes MESH_ROOT and the other connects to it.
Restart the MESH_ROOT node for a couple of cycles until both nodes get in a state where neither becomes MESH_ROOT and so there are no connections made.
Log file "manual test 2A: Crashes once and then restarts.  Does not become MESH_ROOT and does not connect as there is no MESH_ROOT
Log file "manual test 2B: Start but does not become MESH_ROOT and does not connect as there is no MESH_ROOT
I stopped the log files after 72 seconds so I wouldn't lose the start. But I left the 2 terminals running while I wrote this up and they have now been going for 1416228ms or 1416 seconds or over 23 minutes
This pretty much mirrors what I am seeing with my application.
If I keep resetting nodes, they do start up with one becoming MESH_ROOT and the other connecting, but it is VERY easy to get it to fault as described above.  Please try it for yourself.
My objectives are to have a self organized network WITHOUT fixing my MESH_ROOT.  But if a MESH_NODE connects to a parent with an RSSI < -60dBm, I want them to scan and find eligible parents with better RSSI and switch to them using esp_mesh_set_parent().  The reason is I have found the mesh network comms to be flaky if my RSSI figures get less than -65dBm.  The stronger the better.
Most of the time (an for the testing I am doing and reporting to you) I am only using MQTT comms for nodes to an external broker,
But I have also developed and OTA solution, using HTTPS to update my MESH_ROOT and then have my MESH_ROOT push OTA updates to my MESH_NODEs using esp_mesh_send() and recv_cb.  I have developed a protocol and broadcast packets of 1024 bytes at a time, and the MESH_NODEs respond after a small delay X their node number to avoid collisions.  I have Ack messages and a retry mechanism for dropped packets and this works well if RSSI > -65dBm.  BUT less than -65dBm the dropped packets become to much.  I drop these nodes from the OTA process and complete the broadcast to the rest.  I then follow up with P2P messages to the nodes I dropped which works better over lower RSSI connections.  But none of the OTA stuff is active in my application during the tests I have been running, this is just for background information.

@michaelsimp

The crash issue is caused by the error check you added when calling the esp_mesh_scan_get_ap_ie_len() function. After the AP is retrieved, it returns -1.

Since you have set this node as "MESH_SET_NODE" in the menuconfig, it will only look for a mesh AP when searching for a parent. The mesh AP's mesh IE length is non-zero, and the parent of the root can only be a router,  router does not contain the mesh IE. Refer to the code in the example:

Log file "manual test 1" Node crashes, reboots and then becomes root

So, your description is incorrect. In "manual test1," the device (30:30:f9:33:72:b8) did not become the root; instead, after rebooting, it became a third layer node. The log is : I (7672) mesh_main: <MESH_EVENT_PARENT_CONNECTED>layer:0-->3, parent:48:ca:43:9b:5d:21, ID:77:77:77:77:77:76

In "manual test 1B", the node (48:ca:43:9b:53:d8, we call B) has designated 30:30:f9:33:72:b8 (we call A) as its parent, it will only connect to this parent and will not attempt to connect to any other node. When the node A crashes and reboots, the node B will fail to find the AP and will return a 201 error. It will only successfully reconnect once the parent has fully recovered.
W (6278) mesh_main: <PARENT>ESPM_3372B8, layer:2/23, assoc:0/6, 1, 30:30:f9:33:72:b9, channel:1, rssi:-19

In "manual test 2A", the node A initially selected ESPM_9B54C0 as its parent, I (27638) mesh_main: <MESH_EVENT_PARENT_CONNECTED>layer:0-->2, parent:48:ca:43:9b:54:c1<layer2>, ID:77:77:77:77:77:76.
After rebooting, it chose ESPM_9B53D8 as its parent instead. W (6258) mesh_main: <PARENT>ESPM_9B53D8, layer:3/22, assoc:0/6, 0, 48:ca:43:9b:53:d9, channel:1, rssi:-32. The connection fail shows the parent has changed to IDLE.

I (9888) wifi:state: init -> auth (0xb0)
I (9898) wifi:state: auth -> init (0x65c0)
I (9898) wifi:new:<1,0>, old:<1,1>, ap:<1,1>, sta:<1,1>, prof:1, snd_ch_cfg:0x0
I (9898) mesh: [wifi]disconnected reason:101(idle), continuous:1/max:12, non-root, vote(,stopped)<><>
I (9898) mesh_main: <MESH_EVENT_PARENT_DISCONNECTED>reason:101

In "manual test 2B", the node B selects node A as its parent.

Hi.  Thanks for your response.
My apologies, I thought I was working on the original "manual_networking project, but this was my test version which I did modify.
I thought MESH_SET_NODE meant I could set the MESH_NODES parent (to a closer node with RSSI), but I think you are saying it is designed set the MESH_TYPE as MESH_NODE only which I guess this explains why I don't later get a MESH_ROOT.  Please confirm.
So can you give me a lead on how I achieve my objective as I documented in my last post.  I am looking for a hybrid between example "ip_internal_networking" where everything is automatic and example "manual_networking" which is designed to manually set a fixed MESH_ROOT and MESH_NODES.
I am guessing my problem lies in my use of esp_mesh_set_parent() where this example always passes in a type of "my_type" so a MESH_NODE will be set to type MESH_NODE.  Does this then stop the node scanning to become MESH_ROOT.
I am guessing some how I need to restart the scanning for the router with highest RSSI for the voting process but I don't know how to tell when I have lost the MESH_ROOT in the network, and then how to trigger the rescan and vote to enable a MESH_NODE or MESH_IDLE to become MESH_ROOT again.  This all happens automatically on start up before my use of esp_mesh_set_parent().
I really appreciate your help to better understand this.

In the "manual_networking" example, the "MESH_SET_NODE" meant its type will be MESH_NONE, it can't be ROOT
To achieve your goal, I suggest initially using a self-organized network. Once the network is formed, if a node detects that its parent’s RSSI < -60dBm, it can initiate a scanning process to reselect a new parent. When the root node disappears, an MESH_EVENT_NETWORK_STATE event will be triggered. The nodes can then read this event and call esp_mesh_set_self_organized(true, true) to vote a new root.
Actually, in a self-organized network, after the initial network formation, non-root nodes will periodically scan the surrounding nodes and select a better parent, dynamically adjusting the network, as described in the document I mentioned previously. I believe this dynamic adjustment aligns with your requirements.
Have you observed this behavior? The related logs are as follows:

I (23076) mesh: 5170<active>parent layer:1(node), channel:5, rssi:-21, assoc:1(cnx rssi threshold:-120)my_assoc:0
I (24576) mesh: 5969<scan>parent layer:1, rssi:-20, assoc:1(cnx rssi threshold:-120)
I (24576) mesh: [SCAN][ch:5]AP:1, other(ID:0, RD:0), MAP:1, idle:0, candidate:1, root:1, topMAP:0[c:2,i:2][ac:84:c6:7e:31:2b]<weak>
I (24586) mesh: 7391[weak]try rssi_threshold:-120, backoff times:0, max:5<-78,-82,-85>
I (24596) mesh: 716[monitor]no change, parent:4c:11:ae:eb:83:45, rssi:-20
I (24596) mesh: 2012<arm>parent monitor, my layer:2(cap:6)(node), interval:5231ms, retries:3<>

if there is no better parent, the node will remain with its current parent;
I (24596) mesh: 716[monitor]no change, parent:4c:11:ae:eb:83:45, rssi:-20

if a better parent is found, it will switch to that parent.
mesh: 710[monitor]new parent:30:ae:a4:80:1e:ed<layer:2, rssi:-33, assoc:0>current parent:30:ae:a4:00:48:59<layer:3, rssi:-31, assoc:4>

Thank you for this information
Is the MESH_EVENT_NETWORK_STATE  event only triggered by a MESH_ROOT node disappearing or is there additional testing I need to do before calling esp_mesh_set_self_organized(true, true) to vote a new root. Its just the name of this event "MESH_EVENT_NETWORK_STATE" sounds pretty broad and not at all like MESH_ROOT disappeared?
I had read in the docs about the self-organized network non-root nodes periodically scanning the surrounding nodes and selecting a better parent, but I have had multiple cases where a node  at one end of the network location wise has connected with a node at the opposite end of the network location wise and resulting in RSSI figures < -70dBm when there are much closer nodes in between, and they have NOT automatically switched.  How often should it scan and switch?
When I added my manual scan and switching code (with the fixed MESH_ROOT) it formed a much better network and the performance was very noticeably better.
I will try this over the weekend and report back.

Hi
To recap,
I have created a task which runs every 60 seconds to check if the node RSSI < -60dBm and if so it calls wifiMeshScan(), to scan for a better parent.  For now I have commented this out and instead call wifiMeshScan() (see below) manually from my CLI console .
This task also monitors a selfOrganizeReactivateTimer which calls esp_mesh_set_self_organized(true, false) after the scan and switch parent is completed.
static uint8_t meshTaskPrescale = MESH_SCAN_INTERVAL / 2; // rescan in 10 seconds
void vWifiMeshTaskCode(void* args) {
    ESP_LOGI(TAG, "vWifiMeshTaskCode started");
    ESP_ERROR_CHECK(taskRegisterToWatch(xWifiMeshHandle, "appWiFiMesh")); // register task for watchdog
    while (true) {
        sleepMs(1 * MILLISECOND_TICKS_PER_SECOND);
        taskPatWdog(xWifiMeshHandle); // pat watchdog
#ifdef ENABLE_PARENT_SEARCH
        // if (meshTaskPrescale++ > MESH_SCAN_INTERVAL) {
        //     meshTaskPrescale = 0;
        //     if ((!esp_mesh_is_root()) && (currentRSSI < RSSI_RESCAN_THRESHOLD))
        //         wifiMeshScan();
        // }

        if (selfOrganizeReactivateTimer > 0) {
            selfOrganizeReactivateTimer--;
            if (selfOrganizeReactivateTimer == 0) {
                    ESP_LOGI(TAG, "Reactivate self organization");
                    ESP_ERROR_CHECK(esp_mesh_set_self_organized(true, false));
            }
        }
#endif
    }
}

Here is my wifiMeshScan():
void wifiMeshScan() { // for closer parent
    if (esp_mesh_is_root()) {
        ESP_LOGE(TAG, "Error: This node is MESH_ROOT");
        return;
    }
#ifndef ENABLE_PARENT_SEARCH
    ESP_LOGE(TAG, "WifiMeshScan for closer parent disabled");
    return;
#endif // ENABLE_PARENT_SEARCH

    if (esp_mesh_get_type() != MESH_IDLE) {
        esp_err_t err;
        err = esp_wifi_sta_get_rssi(&currentRSSI);
        if (err == ESP_OK) {
            ESP_LOGI(TAG, "WifiMeshScan for closer parent. Current parent RSSI: %d", currentRSSI);
        } else {
            ESP_LOGE(TAG, "WifiMeshScan for closer parent. Get RSSI error: %s", esp_err_to_name(err));
        }
    } else {
        currentRSSI = -100; // if not parent, default low RSSI
    }
    selfOrganizeReactivateTimer = SELF_ORGANIZE_REACTIVATE_LONG_TIME; // enable long reenable timer as backup in case scan does not complete
    ESP_ERROR_CHECK(esp_mesh_set_self_organized(false, false));
    esp_wifi_scan_stop();
    wifi_scan_config_t scan_config = { 0 }; // mesh softAP is hidden
    scan_config.show_hidden = 1;
    scan_config.scan_type = WIFI_SCAN_TYPE_PASSIVE;
    esp_wifi_scan_start(&scan_config, 0);
}

and in my mesh_event_handler I have added:
    case MESH_EVENT_SCAN_DONE: {
        mesh_event_scan_done_t *scan_done = (mesh_event_scan_done_t *)event_data;
        ESP_LOGI(TAG, "<MESH_EVENT_SCAN_DONE>number:%d", scan_done->number);
#ifdef ENABLE_PARENT_SEARCH
        findClosestParent(scan_done->number);  // now look for a closer parent in the results
#endif // ENABLE_PARENT_SEARCH
    }
    break;

and my findClosestParent (copied and modified slightly from manual_networking example, to scan all eligible node and take the one with the best RSSI)
void findClosestParent(int num) { // after a WiFi scan
    ESP_LOGW(TAG, "findClosestParent  Current RSSI: %d", currentRSSI);
    int i;
    int ie_len = 0;
    mesh_assoc_t assoc;
    mesh_assoc_t parent_assoc = { .layer = CONFIG_MESH_MAX_LAYER, .rssi = -120 };
    wifi_ap_record_t record;
    wifi_ap_record_t parent_record = { 0, };
    parent_record.rssi = currentRSSI; // has to be better than current RSSI to change parent
    bool parent_found = false;
    mesh_type_t my_type = MESH_IDLE;
    int my_layer = -1;
    wifi_config_t parent = { 0, };
    wifi_scan_config_t scan_config = { 0 };

    for (i = 0; i < num; i++) { // iterate through scan records looking for eligible closer parent node
        ESP_ERROR_CHECK(esp_mesh_scan_get_ap_ie_len(&ie_len));
        ESP_ERROR_CHECK(esp_mesh_scan_get_ap_record(&record, &assoc));
        ESP_LOGD(TAG, "ie_len: %d  sizeof(assoc): %d", ie_len, sizeof(assoc));
        if (ie_len == sizeof(assoc)) {
            ESP_LOGI(TAG,
                     "<MESH>[%d]%s, layer:%d/%d, assoc:%d/%d, %d, "MACSTR", channel:%u, rssi:%d, ID<"MACSTR"><%s>",
                     i, record.ssid, assoc.layer, assoc.layer_cap, assoc.assoc, assoc.assoc_cap, assoc.layer2_cap, MAC2STR(record.bssid),
                     record.primary, record.rssi, MAC2STR(assoc.mesh_id), assoc.encrypted ? "IE Encrypted" : "IE Unencrypted");

            // ESP_LOGI(MESH_TAG, "Type: %d  layer_cap %d:  assoc %d  assoc_cap: %d  rssi: %d", assoc.mesh_type, assoc.layer_cap, assoc.assoc, assoc.assoc_cap, record.rssi);
            if (assoc.mesh_type != MESH_IDLE && assoc.layer_cap && assoc.assoc < assoc.assoc_cap) { //  && record.rssi > -70) MBS removed
                // ESP_LOGI(MESH_TAG, "assoc.layer: %d  parent_assoc.layer %d:  assoc.layer2_cap %d  parent_assoc.layer2_cap: %d", assoc.layer, parent_assoc.layer, assoc.layer2_cap, parent_assoc.layer2_cap);
                if (assoc.layer < parent_assoc.layer || assoc.layer2_cap < parent_assoc.layer2_cap) {
                    if (record.rssi > parent_record.rssi) { // closer parent found
                        if (memcmp(&parent_assoc, &assoc, sizeof(assoc)) != 0) { // dont switch to same parent
                            ESP_LOGW(TAG, "Closer Parent found: %s  RSSI: %d", record.ssid, record.rssi);
                            parent_found = true;
                            memcpy(&parent_record, &record, sizeof(record));
                            memcpy(&parent_assoc, &assoc, sizeof(assoc));
                            if (parent_assoc.layer_cap != 1) {
                                my_type = MESH_NODE;
                            } else {
                                my_type = MESH_LEAF;
                            }
                            my_layer = parent_assoc.layer + 1;
                            // break; // MSB removed, keep searching for the closest parent
                        }
                    }
                }
            }
        } else {
            ESP_LOGD(TAG, "[%d]%s, "MACSTR", channel:%u, rssi:%d", i, record.ssid, MAC2STR(record.bssid), record.primary, record.rssi);
        }
    }

    esp_mesh_flush_scan_result();
    if (parent_found) { // parent: Both channel and SSID of the parent are mandatory
        parent.sta.channel = parent_record.primary;
        memcpy(&parent.sta.ssid, &parent_record.ssid, sizeof(parent_record.ssid));
        parent.sta.bssid_set = 1;
        memcpy(&parent.sta.bssid, parent_record.bssid, 6);
        if ((my_type == MESH_NODE) || (my_type == MESH_LEAF) || (my_type == MESH_IDLE)) {
            ESP_ERROR_CHECK(esp_mesh_set_ap_authmode(parent_record.authmode));
            if (parent_record.authmode != WIFI_AUTH_OPEN) {
                memcpy(&parent.sta.password, CONFIG_MESH_AP_PASSWD, strlen(CONFIG_MESH_AP_PASSWD));
            }
            ESP_LOGW(TAG,
                     "<PARENT>%s, layer:%d/%d, assoc:%d/%d, %d, "MACSTR", channel:%u, rssi:%d",
                     parent_record.ssid, parent_assoc.layer,
                     parent_assoc.layer_cap, parent_assoc.assoc,
                     parent_assoc.assoc_cap, parent_assoc.layer2_cap,
                     MAC2STR(parent_record.bssid), parent_record.primary,
                     parent_record.rssi);
            ESP_ERROR_CHECK(esp_mesh_set_parent(&parent, (mesh_addr_t *)&parent_assoc.mesh_id, my_type, my_layer));
            selfOrganizeReactivateTimer = SELF_ORGANIZE_REACTIVATE_TIME; // start self organize reactivation timer
        }
    } else {
        ESP_LOGE(TAG, "No eligible closer Parent found");
        if (currentRSSI == NO_RSSI) { // scan again if no connection yet
            esp_wifi_scan_stop();
            scan_config.show_hidden = 1;
            scan_config.scan_type = WIFI_SCAN_TYPE_PASSIVE;
            esp_wifi_scan_start(&scan_config, 0);
        }
    }
}

My problem was if I run the above code, and then later I restart my MESH_ROOT, I didn't get a new MESH_ROOT and the mesh network can't re-establish itself.
Note this is not a problem if the code above does not run.
Based on your last post advice I have added the following to mesh_event_handler()
    case MESH_EVENT_NETWORK_STATE: {
        mesh_event_network_state_t *network_state = (mesh_event_network_state_t *)event_data;
        if (network_state->is_rootless) {
            ESP_LOGW(TAG, "\n<MESH_EVENT_NETWORK_STATE> MESH_ROOT lost\n");
            esp_mesh_set_self_organized(true, true); // vote a new root
        } else {
            ESP_LOGW(TAG, "\n<MESH_EVENT_NETWORK_STATE> MESH_ROOT established\n");
        }
    }

Now when I restart the MESH_ROOT it works perfectly SOME of the time.  The event MESH_EVENT_NETWORK_STATE fires, and the voting process starts to find a MESH_ROOT. Everything reestablishes and starts working again.
But more than half of the time, the network gets in a mess.  The MESH_EVENT_NETWORK_STATE event does NOT fire to restart the MESH_ROOT polling.
I end up with no MESH_ROOT and MESH_NODES on deep layers all point directly or indirectly to a MESH_IDLE node.  See my logs attached.
10Nov.zip

rootrestart.txt starts at the reboot of my MESH_ROOT.  At the end my console command "mesh" shows this node 48:ca:43:9b:53:d8 as a MESH_NODE layer 4 to parent 48:ca:43:9b:5c:69
node1.txt  shows 48:ca:43:9b:5c:68 as a MESH_NODE layer 3 to parent 30:30:f9:33:49:bd
node2.txt  shows 30:30:f9:33:49:bc as a MESH_IDLE
node 3.txt shows 30:30:f9:33:72:b8 as a MESH_NODE layer 4 to parent 48:ca:43:9b:5c:69

None of these log files show the MESH_EVENT_NETWORK_STATE  firing and calling esp_mesh_set_self_organized(true, true)
So as a test I added a CLI command to call  esp_mesh_set_self_organized(true, true) and tried to use it to rectify the problem above.  It does work some of the time creating a MESH_ROOT and bringing some nodes back, but not all nodes.
In summary:
    esp_mesh_set_self_organized(false, false);
    esp_wifi_scan_stop();
    wifi_scan_config_t scan_config = { 0 }; // mesh softAP is hidden
    scan_config.show_hidden = 1;
    scan_config.scan_type = WIFI_SCAN_TYPE_PASSIVE;
    esp_wifi_scan_start(&scan_config, 0);

When I find a better parent:
    esp_mesh_set_parent(&parent, (mesh_addr_t *)&parent_assoc.mesh_id, my_type, my_layer)
    ....
    esp_mesh_set_self_organized(true, false);

When root disappears:
   case MESH_EVENT_NETWORK_STATE:  (and if is_rootless)
   esp_mesh_set_self_organized(true, true);

The problem:
MESH_EVENT_NETWORK_STATE often does not fire and call esp_mesh_set_self_organized(true, true)

Thank you for this information Is the MESH_EVENT_NETWORK_STATE event only triggered by a MESH_ROOT node disappearing or is there additional testing I need to do before calling esp_mesh_set_self_organized(true, true) to vote a new root. Its just the name of this event "MESH_EVENT_NETWORK_STATE" sounds pretty broad and not at all like MESH_ROOT disappeared?
I had read in the docs about the self-organized network non-root nodes periodically scanning the surrounding nodes and selecting a better parent, but I have had multiple cases where a node at one end of the network location wise has connected with a node at the opposite end of the network location wise and resulting in RSSI figures < -70dBm when there are much closer nodes in between, and they have NOT automatically switched. How often should it scan and switch?
When I added my manual scan and switching code (with the fixed MESH_ROOT) it formed a much better network and the performance was very noticeably better.
I will try this over the weekend and report back.

The MESH_EVENT_NETWORK_STATE event is triggered by a Layer 2 nodes and propagates progressively through the network to the final node. The event only records a "rootless" state underneath.

The scanning period is not fixed, but is at least 4.5 seconds.

I end up with no MESH_ROOT and MESH_NODES on deep layers all point directly or indirectly to a MESH_IDLE node. See my logs attached.
10Nov.zip

Could you provide a more complete log? I need to understand the network state before the root reboot. Specifically, how Node1, Node2, and Node3 were connected, and what the behavior was after the reboot.

Hi
See logs attached, from boot till end.
11Nov.zip
I have all 4 nodes in my office for convenience.
I have disabled the periodic scan for closer wifi and invoke this from my CLI console - see the command "wifi scan".  It looks like I only sent this on nodes 2 and 3 this time, but the results are pretty typical of when it goes wrong:

Event <MESH_EVENT_NETWORK_STATE> did not fire so I didn't call esp_mesh_set_self_organized(true, true);
ended up with broken mesh

Ignore the watchdog in MQTT. I think the library blocks a bit when WiFi is not up.

test2 root.txt: MAC: 30:30:f9:33:72:b8 MESH_ROOT powered off at the end of the file and left off
test2 node1.txt:  MAC: 48:ca:43:9b:53:d8 After MESH_ROOT powered of, this MESH_NODE becomes MESH_IDLE
test2 node2.txt:  MAC: 48:ca:43:9b:5c:68 After MESH_ROOT powered of, this MESH_NODE layer 2 becomes MESH_NODE layer 3 parent of 48:ca:43:9b:53:d9
test2 node3.txt:  MAC: 30:30:f9:33:49:bc After MESH_ROOT powered of, this MESH_NODE becomes MESH_NODE layer 4 parent of 48:ca:43:9b:5c:69

Hi Zhangyanjiaoesp
Just checking on the status of this to make sure you have everything you need and that you will get back to me with some ideas soon.  Thanks

@michaelsimp
Is this the expected behavior?

Initially, the network is self-organized, forming the initial network.
Then, non-root nodes disable self-organization and periodically scan for nearby APs, selecting a better parent.
When the root node disappears, the other nodes re-enable self-organization, reselect the root, and restore the network.

Answering your questions:
When would you invoke the scan via the CLI console? Is it at any given moment?
YES
Can you disable the MQTT task watchdog? I'm not sure if it will interfere with the entire process.
OK
Why are there so many reboot operations in the root log?
Sorry I should have deleted up until the last one.  I did a few tests, some successful before the final one I sent you.  Reference the last test run from line 1672 onwards which matches the node traces I sent.
Is this the expected behavior?
Initially, the network is self-organized, forming the initial network.
Yes
Then, non-root nodes disable self-organization and periodically scan for nearby APs, selecting a better parent.
Yes
When the root node disappears, the other nodes re-enable self-organization, reselect the root, and restore the network.
Yes.  This part is the problem as OFTEN MESH_EVENT_NETWORK_STATE does not fire and call esp_mesh_set_self_organized(true, true).  If I call call esp_mesh_set_self_organized(true, true) manually from my CLI, it doesn't fix the broken mesh.
If it helps, I could make a cut down project and send it to you to test and play with.  It doesn't need any special hardware, I am just using ESP32-S3-WROOM-1 dev boards with 16MB Flash

If it helps, I could make a cut down project and send it to you to test and play with. It doesn't need any special hardware, I am just using ESP32-S3-WROOM-1 dev boards with 16MB Flash

Sure, it would be better if you could provide a demo. That way, I can test and debug it locally.

Hi
See attached wincut folder structure of my project minus the build directory.
wincut.zip
I have cut out the application but left all of the platform stuff.  Its still quite big but you will probably only be interested in the source at:
..\wincut\main\platform\espidf\wifi where wifiMesh.cpp is developed out of the example project "ip_internal_network"
Typing "help" will bring up a menu.
Your targets probably won't be wifi provisioned yet but if it is you can run "wifi unprovision" to clear it out.  On restart it will display the provisioning screen on the console with the QRCode for the ESP provisioning app.
After provisioning (all your nodes), they will start up normally and form a self organized mesh.
Type "mesh" to display the nodes WiFi mesh attributes including the Mesh routing Table as follows:
W (00:14:26.671) aWifiMesh: MESH_ROOT  Layer: 1  MAC: 30:30:f9:33:49:bc  IP: <IP_ADDRESS>
I (00:14:26.674) aWifiMesh: Router VONETS  BSSID (MAC): 00:17:13:20:bd:74
I (00:14:26.686) aWifiMesh: Vote: 100%
I (00:14:26.687) aWifiMesh: Network is: self organized
I (00:14:26.688) aWifiMesh: Network Root is: not fixed
I (00:14:26.700) aWifiMesh: Number of nodes: 4
I (00:14:26.701) aWifiMesh: Max layers: 6
I (00:14:26.713) aWifiMesh: Mesh Parent MAC: 00:17:13:20:bd:74  RSSI: -45  Channel: 3
I (00:14:26.715) aWifiMesh: Mesh routing Table records: 4   Record size: 6   Table size: 24
I (00:14:26.728) aWifiMesh:   MAC: 30:30:f9:33:49:bc
I (00:14:26.729) aWifiMesh:   MAC: 30:30:f9:33:72:b8
I (00:14:26.742) aWifiMesh:   MAC: 48:ca:43:9b:5c:68
I (00:14:26.744) aWifiMesh:   MAC: 48:ca:43:9b:53:d8
The MQTT app will try to connect to a Mosquito Broker on <IP_ADDRESS>.  If you want to disable this the MQTT is in the file:
..\wincut\main\host\mqtt5app.cpp. Comment out the last line of code which starts my MQTT task.
Type "wifi scan" on a NON MESH_ROOT to get it to search for a better parent and switch.  It won't run on a MESH_ROOT.
Reboot the MESH_ROOT to reproduce the broken Mesh issue.
Type "wifi root" to run call esp_mesh_set_self_organized(true, true)

Have you manged to test my application yet and reproduce the problem?

Still in testing，I didn't use your demo as it involves too many components. Instead, I made some modifications based on the ip_internal_network example.

Ok, can you reproduce the problem?

@michaelsim
Here are the changes I made in the ip_internal_network example，you can take a look.
0001-add-user-scan-code.zip
There are the following points to note in your changes:

When determining if the parent is consistent, using the BSSID is sufficient，so I use memcmp(parent_record.bssid, record.bssid, 6) != 0 in the findClosestParent() function , the memcmp(&parent_assoc, &assoc, sizeof(assoc)) != 0 should not be used.
I encountered an issue during testing if I use memcmp(&parent_assoc, &assoc, sizeof(assoc)) != 0. The issue is like this: when the current connected parent was detected, but the RSSI was lower than when it was previously connected. The node still reselected the parent, even though it was not necessary to do so. It is normal for the RSSI values obtained when scanning the AP at different times to vary.

Why did you add a scan operation when the in the MESH_EVENT_PARENT_DISCONNECTED event? If the scan operation must be invoked, it is essential to call the esp_mesh_set_self_organized(false, false);  before the scan API. This ensures that the mesh self-organization is disabled before the application layer calling the scan.

same to here, also need to call esp_mesh_set_self_organized(false, false);

I think the invocation of the esp_mesh_set_self_organized(true, false) is unnecessary.

Hi Zhangyanjiaoesp
Thanks for the feedback and software
I backed up the original files, applied the patch and then compared the files.

esp_task.h had a change in #define ESP_TASKD_EVENT_STACK only. I have applied this but havn't seen stack related crashes.
internal_communication\main\mesh_main.c had many changes to compare.  Is this just adding the parent switch code, or are there other changes here which I need to apply to my application in addition to the 3 notes you gave me.

I will apply the 3 changes and retest and report, but I would like to know if you found and fixed anything or if your note 2 is my problem.
Thanks
Michael

Hi Zhangyanjiaoesp
Note 1: This makes sense, thanks implemented.
Note 2:
My thinking was if I previously called  ESP_ERROR_CHECK(esp_mesh_set_self_organized(false, false)); to scan for a closer parent, I don't have a self organized network, so I need to scan for a parent when the node gets disconnected. Is this not correct?
The second block you pointed out was taken from example project manual_networking inside function mesh_scan_done_handler().  In the else of parent_found (so !parent_found) it calls esp_wifi_scan_stop() and esp_wifi_scan_start() without esp_mesh_set_self_organized(false, false);
I have added esp_mesh_set_self_organized(false, false); before the esp_wifi_scan_stop() as per other instances of this.
Note 3: My thinking was to return the network to self organized again after the scan for better Parent.  I will remove it soon (don't want to make too many changes at once) if it is not causing a problem for now.  Please confirm.
I am not sure what the overall status of this is from your side, please advise:

Could you reproduce the broken mesh problem?
Did you find any problems in the library - I saw only a minor change to stack size in esp_task.h
Were there problems in my original main_mesh.c from internal_networking other than your test code
Is there a specific problem with my code in addition to not call esp_mesh_set_self_organized(false, false); before a mesh scan?

I did some tests.  It is much better at establishing a MESH_ROOT when the MESH_ROOT reboots or is powered off. But I still have a few problems:
14Nov.zip
Test1 root.txt: My Mesh_ROOT crashed line 395 Guru Meditation Error
Test 2 MESH_NODE.txt:   See notes at top of file.  After Wifi Scan for a new parent, Became a MESH_IDLE with a child node
Why does a node become a MESH_IDLE when there are MESH_ROOT and MESH_NODES very close (around -35dBM) which are well below capacity?  How do I fix them?  Another wifi scan request does not fix it as you can see in the log.
Why does a MESH_NODE connect to a MESH_IDLE when there are MESH_ROOT and MESH_NODEs available with strong RSSIs?  Surely this should not be allowed.  This MESH_NODE was otherwise healthy and so when I rebooted its parent which was MESH_IDLE, it connected to another MESH_NODE successfully.
I started looking though your many changes to internal_networking mesh_main file and found some changes which I think are important. eg in event <MESH_EVENT_PARENT_DISCONNECTED>

Would it be possible for you to highlight any other important changes I need to merge into my application

Also the change above adding to event <MESH_EVENT_PARENT_DISCONNECTED>
        if (!esp_mesh_get_self_organized()) {
            printf(">>>%d, set true, true\n",__LINE__);
            esp_mesh_set_self_organized(true, true); // vote a new root
        }

...undoes my switch to a close parent after manual scan.

After more testing and switching back modifications, I have the following summary:
In event <MESH_EVENT_PARENT_DISCONNECTED> if I have this code, the networks establishes itself again reliably and everything looks quite stable.  BUT: It immediately undoes my switch to a close parent after manual scan.
    case MESH_EVENT_PARENT_DISCONNECTED: {
        mesh_event_disconnected_t *disconnected = (mesh_event_disconnected_t *)event_data;
        ESP_LOGI(TAG, "<MESH_EVENT_PARENT_DISCONNECTED>reason:%d", disconnected->reason);
        mesh_layer = esp_mesh_get_layer();
        mesh_netifs_stop();
        wifiConnected = false;
        ESP_LOGW(TAG, "WiFi Disconnected");
        currentRSSI = NO_RSSI;

        printf(">>>last layer = %d, layer = %d\n", last_layer, mesh_layer);

        if (!esp_mesh_get_self_organized()) {
            printf(">>>%d, set true, true\n",__LINE__);
            esp_mesh_set_self_organized(true, true); // vote a new root
        }
    }
    break;

Whereas if I have this code, the MESH_NODE parent switch works nicely, but the mesh breaks after the MESH_ROOT reboots or powers off.
    case MESH_EVENT_PARENT_DISCONNECTED: {
        mesh_event_disconnected_t *disconnected = (mesh_event_disconnected_t *)event_data;
        ESP_LOGI(TAG, "<MESH_EVENT_PARENT_DISCONNECTED>reason:%d", disconnected->reason);
        mesh_layer = esp_mesh_get_layer();
        mesh_netifs_stop();
        wifiConnected = false;
        ESP_LOGW(TAG, "WiFi Disconnected");
        currentRSSI = NO_RSSI;

        printf(">>>last layer = %d, layer = %d\n", last_layer, mesh_layer);

        if (disconnected->reason == WIFI_REASON_ASSOC_TOOMANY) {
            esp_mesh_set_self_organized(false, false);
            esp_wifi_scan_stop();
            scan_config.show_hidden = 1;
            scan_config.scan_type = WIFI_SCAN_TYPE_PASSIVE;
            esp_wifi_scan_start(&scan_config, 0);
        }
    }
    break;

@michaelsimp
The following answers your question：

The patch I provided is the calling method I believe to be correct. I was unable to reproduce your issue locally, as the devices were relatively close to each other during my testing. I cannot deploy my test setup in the same way you have. However, I did test the scenario of whether other nodes can correctly elect a new root after a root power off, and the result was successful.

The second block you pointed out was taken from example project manual_networking inside function mesh_scan_done_handler(). In the else of parent_found (so !parent_found) it calls esp_wifi_scan_stop() and esp_wifi_scan_start() without esp_mesh_set_self_organized(false, false);
I have added esp_mesh_set_self_organized(false, false); before the esp_wifi_scan_stop() as per other instances of this.

In the manual_networking example, the esp_mesh_set_self_organized(false, false) API is called only once throughout the entire process. It ensures that the network remains non self-organized throughout. However, in your project, the esp_mesh_set_self_organized(false, false) , esp_mesh_set_self_organized(true, false), esp_mesh_set_self_organized(true, true) are all called. To ensure that the self-organized network is disabled during the user's scan, it is necessary to call esp_mesh_set_self_organized(false, false). See the doc

It looks like ESP_ERROR_CHECK is not a good strategy. I don't know which argument could be wrong?
Any advice?

Unless the return value check is critical, please avoid calling ESP_ERROR_CHECK, as it may cause the program to abort. This is something we discussed previously. Regarding the esp_mesh_set_parent() issue, could you provide how you have set the parameters? I can help check why the parameter error is occurring.

The crash issue is caused by the error check you added when calling the esp_mesh_scan_get_ap_ie_len() function. After the AP is retrieved, it returns -1.

In event <MESH_EVENT_PARENT_DISCONNECTED> if I have this code, the networks establishes itself again reliably and everything looks quite stable. BUT: It immediately undoes my switch to a close parent after manual scan.

Yes,  I’ve noticed that as well. I think I need to find a better solution.

Note 3: My thinking was to return the network to self organized again after the scan for better Parent. I will remove it soon (don't want to make too many changes at once) if it is not causing a problem for now. Please confirm.

I still think this part of the code is unnecessary. Returning to the self-organized network after finding a better parent doesn’t seem meaningful, especially since you need to periodically scan for nearby APs, and disable the self-organized network during scanning. As I understand it, you only need the self-organized network when selecting the root; at other times, you prefer to actively choose a better parent, correct?

I will check the log you sent and get back to you.

Hi Zhangyanjiaoesp
I'm really a bit lost now as to where I am and what to do next.
Yes I have applied all the patches you sent me and done a clean build.
I would like to park the Guru Meditation Errors for now as they are infrequent and focus on my main problem which is how to recover after the MESH_ROOT is rebooted.
In event <MESH_EVENT_PARENT_DISCONNECTED> if I have the code below, the networks establishes itself again reliably and everything looks quite stable. BUT: It immediately undoes my switch to a close parent after manual scan.
    case MESH_EVENT_PARENT_DISCONNECTED: {
        mesh_event_disconnected_t *disconnected = (mesh_event_disconnected_t *)event_data;
        ESP_LOGI(TAG, "<MESH_EVENT_PARENT_DISCONNECTED>reason:%d", disconnected->reason);
        mesh_layer = esp_mesh_get_layer();
        mesh_netifs_stop();
        wifiConnected = false;
        ESP_LOGW(TAG, "WiFi Disconnected");
        currentRSSI = NO_RSSI;

        printf(">>>last layer = %d, layer = %d\n", last_layer, mesh_layer);

        if (!esp_mesh_get_self_organized()) {
            printf(">>>%d, set true, true\n",__LINE__);
            esp_mesh_set_self_organized(true, true); // vote a new root
        }
    }
    break;

Whereas this, the MESH_NODE parent switch works nicely, but the mesh breaks after the MESH_ROOT reboots or powers off.
    case MESH_EVENT_PARENT_DISCONNECTED: {
        mesh_event_disconnected_t *disconnected = (mesh_event_disconnected_t *)event_data;
        ESP_LOGI(TAG, "<MESH_EVENT_PARENT_DISCONNECTED>reason:%d", disconnected->reason);
        mesh_layer = esp_mesh_get_layer();
        mesh_netifs_stop();
        wifiConnected = false;
        ESP_LOGW(TAG, "WiFi Disconnected");
        currentRSSI = NO_RSSI;

        printf(">>>last layer = %d, layer = %d\n", last_layer, mesh_layer);

//        if (!esp_mesh_get_self_organized()) {
//            printf(">>>%d, set true, true\n",__LINE__);
//            esp_mesh_set_self_organized(true, true); // vote a new root
//        }
    }
    break;

You acknowledged this replying "Yes, I’ve noticed that as well. I think I need to find a better solution."
My outstanding questions are:

Are you working on the better solution (immediate above). I don't mean to sound impatient, I just want to ensure we are on the same page.

The internal_communication\main\mesh_main.c you sent me had many changes to compare. Is this just adding the parent switch code, or are there other changes here which I need to apply to my application in addition to the 3 notes you gave me.

Why does a node become a MESH_IDLE when there are MESH_ROOT and MESH_NODES very close (around -35dBM) which are well below capacity? How do I fix them? Another wifi scan request does not fix it as you can see in the log.

Your replied "The device is idle because it has not yet connected to the network. Its child should return parent idle error."
But my problem is it stays this way.  Is there something I should do after a node goes MESH_IDLE, or should it fix itself?

Also when a node becomes MESH_IDLE and it has children (like the example above) why doesn't it drop it's children or why don't the children disconnect themselves and look for a proper connection?

Are these symptoms of the system breaking after the manual scan and set better parent followed by the MESH_ROOT reboot?
Is this something you are still working on?
Or do I have to manage this and if so, how?

Hi Zhangyanjiaoesp
I did spend time analyzing the changes you made to mesh_main.c with a full compare using winmerge.  I just wanted to check with you that I hadn't missed anything important.
I fully understand you have other tasks and cannot respond immediately.  The purpose of my question was just to check if you were still planning to help me resolve this.
See my source attached.
wincut2.zip
I have commented out the  esp_mesh_set_self_organized(true, false) per your advice.
I have commented out the task which automatically searches for close nodes as I found it better to control this manually from my console using the command "wifi scan"
My event <MESH_EVENT_PARENT_DISCONNECTED> has the following code per your update.  This stabilizes the mesh network when the MESH_ROOT reboots, but immediately undoes my manual switch to a closer parent.
        printf(">>>last layer = %d, layer = %d\n", last_layer, mesh_layer);

        if (!esp_mesh_get_self_organized()) {
            ESP_LOGW(TAG, "Reverting to Self Organised");
            printf(">>>%d, set true, true\n",__LINE__);
            esp_mesh_set_self_organized(true, true); // vote a new root
        }

I really do appreciate your assistance.

@michaelsimp
please use this change and test again.

@michaelsimp
I would like to discuss the rules for selecting a better parent with you. When choosing a better parent, do you only consider the RSSI value, or do you also take into account layer, assoc, and RSSI? If it’s the latter, which factor do you prioritize the most?
During my testing, I observed the following phenomenon:
Node A initially connects to the root node, becoming a layer2 node. Therefore, its parent’s layer should be 1, i.e., parent_assoc.layer = 1. However, parent_assoc is defined in the findClosestParent() function and is initially set to 6.
void findClosestParent(int num) { // after a WiFi scan
    ESP_LOGW(TAG, "findClosestParent  Current RSSI: %d", currentRSSI);
    int i;
    int ie_len = 0;
    mesh_assoc_t assoc;
    mesh_assoc_t parent_assoc = { .layer = CONFIG_MESH_MAX_LAYER, .rssi = -120 };
    wifi_ap_record_t record;
    wifi_ap_record_t parent_record = { 0, };
    parent_record.rssi = currentRSSI; // has to be better than current RSSI to change parent

If Node A scans an AP with a layer < 6 and an RSSI stronger than its current parent’s RSSI, it will adopt the new AP as a better parent. In my case, Node A found Node B, a layer2 node, as a better parent, and became a layer3 node.
I (18614) mesh_main: <MESH_EVENT_SCAN_DONE>number:2
W (18614) aWifiMesh: findClosestParent  Current RSSI: -6
I (18624) aWifiMesh: <MESH>[0]ESPM_A5B180, layer:2/4, assoc:0/2, 1, 34:85:18:a5:b1:81, channel:5, rssi:0, ID<77:77:77:77:77:76><IE Unencrypted>
>>>layer: 2,6, layer2_cap:1/0
W (18634) aWifiMesh: Closer Parent found: ESPM_A5B180  RSSI: 0
I (18644) aWifiMesh: <MESH>[1]ESPM_E0F6C0, layer:1/5, assoc:2/2, 0, 7c:df:a1:e0:f6:c1, channel:5, rssi:-6, ID<77:77:77:77:77:76><IE Unencrypted>
W (18654) aWifiMesh: <PARENT>ESPM_A5B180, layer:2/4, assoc:0/2, 1, 34:85:18:a5:b1:81, channel:5, rssi:0
I (18664) mesh: [IO]disable self-organizing<reconnect>
I (18674) wifi:state: run -> init (0x0)
I (18684) wifi:pm stop, total sleep time: 0 us / 12567464 us

I (18684) wifi:<ba-del>idx:0, tid:5
I (18684) wifi:new:<5,0>, old:<5,1>, ap:<5,1>, sta:<5,1>, prof:5, snd_ch_cfg:0x0
W (18704) wifi:<MESH AP>adjust channel:5, secondary channel offset:1(40U)
I (18704) wifi:Total power save buffer number: 16
W (18714) wifi:Password length matches WPA2 standards, authmode threshold changes from OPEN to WPA2
I (18754) mesh: [MANUAL]connect to parent:ESPM_A5B180, 34:85:18:a5:b1:81[layer:2], ID:77:77:77:77:77:76<>
I (18934) mesh_main: <MESH_EVENT_PARENT_CONNECTED>layer:2-->3, parent:34:85:18:a5:b1:81, ID:77:77:77:77:77:76

However, this actually resulted in a worse network condition for Node A, because the network path became longer by transitioning from a layer2 node to a layer3 node.
If I move the definition of parent_assoc outside the findClosestParent() function and assign it a value in the MESH_EVENT_PARENT_CONNECTED event, it would solve the issue described above.
    case MESH_EVENT_PARENT_CONNECTED: {
        mesh_event_connected_t *connected = (mesh_event_connected_t *)event_data;
        esp_mesh_get_id(&id);
        mesh_layer = connected->self_layer;
        memcpy(&mesh_parent_addr.addr, connected->connected.bssid, 6);
        parent_assoc.layer = mesh_layer - 1;

However, it would introduce a new problem: a layer 2 node would never switch its parent because its parent is the root node (layer = 1), and no other node has a layer smaller than 1.

Hi
In response to your 2nd to last post.
Q 1. Yes I am using 5.3.1.  I completed deleted earlier SDKs 5.3.0 and 5.2 a few weeks ago to be 100% sure.
Q 2. I have tried this but no success.
    case MESH_EVENT_PARENT_DISCONNECTED: {
        mesh_event_disconnected_t *disconnected = (mesh_event_disconnected_t *)event_data;
        ESP_LOGI(TAG, "<MESH_EVENT_PARENT_DISCONNECTED>reason:%d", disconnected->reason);
        mesh_layer = esp_mesh_get_layer();
        mesh_netifs_stop();
        wifiConnected = false;
        connectionStatusLed(Provisioned); // update LED flash pattern
        ESP_LOGW(TAG, "WiFi Disconnected");
        currentRSSI = NO_RSSI;

        printf(">>>last layer = %d, layer = %d\n", last_layer, mesh_layer);

        if (disconnected->reason == WIFI_REASON_BEACON_TIMEOUT) {
            printf(">>>%d, set true, true\n",__LINE__);
            ESP_LOGW(TAG, "Reverting to Self Organised");
            esp_mesh_set_self_organized(true, true); // vote a new root
        }
    }
    break;

I did some work around with the disconnect->reason codes.  These are named but not described so its hard to know what they mean and how to apply them. Some of them are not even defined eg 100 & 101
On Node reboot during shutdown I get disconnect->reason codes:
8 = WIFI_REASON_ASSOC_LEAVE
On Node startup I get disconnect->reason codes:
101 = not defined
100 = not defined
I found when the MESH_ROOT is lost I get disconnect->reason codes:
209 = WIFI_REASON_SA_QUERY_TIMEOUT
101 = not defined
then it scans
and after scan sometimes I get
2 = WIFI_REASON_AUTH_EXPIRE
105 = not defined
When I manually switch to a closer parent I get disconnect->reason codes:
8 = WIFI_REASON_ASSOC_LEAVE
201= WIFI_REASON_NO_AP_FOUND
sometimes
206=WIFI_REASON_AP_TSF_RESET
On event <MESH_EVENT_PARENT_DISCONNECTED> I thought I could test if not codes 8, 201 and 206 then esp_mesh_set_self_organized(true, true); but the reason codes are inconsistent
        // if (disconnected->reason == WIFI_REASON_BEACON_TIMEOUT) {
        //     printf(">>>%d, set true, true\n",__LINE__);
        //     ESP_LOGW(TAG, "Reverting to Self Organised");
        //     esp_mesh_set_self_organized(true, true); // vote a new root
        // }
        if ((disconnected->reason != WIFI_REASON_ASSOC_LEAVE) && (disconnected->reason != WIFI_REASON_NO_AP_FOUND) && (disconnected->reason != WIFI_REASON_AP_TSF_RESET)) {
            printf(">>>%d, set true, true\n",__LINE__);
            ESP_LOGW(TAG, "Reverting to Self Organised");
            esp_mesh_set_self_organized(true, true); // vote a new root
        }

I don't feel comfortable with this solution unless it is endorsed by you guys, but anyway after several successful cycles, it failed again with lots of:
W (210492) mesh: [mesh_schedule.c,3130] [WND-RX]max_wnd:2, 1200 ms timeout, seqno:589, xseqno:232, no_wnd_count:0, timeout_count:23
I (210992) mesh: [SCAN][ch:1]AP:5, other(ID:0, RD:0), MAP:3, idle:1, candidate:0, root:0, topMAP:0[c:2,i:2][00:00:00:00:00:00]<>
I (210992) mesh: [FAIL][53]root:0, fail:53, normal:0, <pre>backoff:0

ending in a broken mesh again.
Q3. Yes I had changed log_levels at the start of my project.  I have removed this now so everything starts at default levels.  You can set the levels in my console using "loglevel xxx l" where xxx is the ESP_LOG tag and l = the level.
See logs attached
Nov20.zip
Wifi manual scan switch shows MESH_NODE start up and connect,followed by a manual scan and switch to a closer parent. (No reboot of MESH_ROOT)
Test 1 is a reference with no manual scan for closer parent before reboot of MESH_ROOT and all nodes reconnect successfully
Test 2 is a number of cycles of manual scan and parent switch followed by MESH_ROOT reboot.  This was successful for about 3 cycles although it seemed the struggle, before failing on the last attempt.  The logs have wrapped but show the end when it gets broken
Test 3 is a more controlled failure:
Test3 Mesh_root showing power off after nodes connect and manual scan for better root
Test3 Node becomes root showing, MESH_NODE, Connects to root, Does manual scan and switch parent, Root powers off, This node becomes MESH_ROOT
Test 3 node becomes MESH_IDLE showing, MESH_NODE starts and connects to root, Power off root, MESH_NODE doesn't scan instead become MESH_IDLE
Test 3 node stays connected to parent which becomes MESH_IDLE, showing MESH_NODE connects to MESH_ROOT, Does manual scan and switch parent, Root powers off, This node doesn't scan, stays MESH_NODE connected same parent which has now become MESH_IDLE

Hi
In response to your last post:
At present I only consider the RSSI value.
The test in findClosestParent() does consider nodes already at capacity number of children, and will not try to switch to them.
My thoughts were, I am not wanting to build the mesh network from scratch as I start with a self configured network.  I am only planning to make changes to nodes with poor RSSIs.  So far my tests have been successful network architecture wise (when I have a fixed ROOT so I don't get the broken mesh problem).
I appreciate what you are saying and will certainly doo more testing and add more intelligence into the parent selection if necessary. I already send all my node network attributes to the MESH_ROOT where I have a table of all node and their parent, children, layers and RSSI.  I could broadcast this to all nodes if necessary to enable smarter logic at the selection.
But I can't keep the manual scan and parent switch code while it causes my network to break which leaves me in a real predicament performance wise.
I really need a resolution to this as my priority.
Thanks for your ongoing help

The definition of reason code 100/101 is here:
https://github.com/espressif/esp-idf/blob/f420609c332fbd2d2f7f188c6579d046c9560e42/components/esp_wifi/include/esp_mesh.h#L267

I don't feel comfortable with this solution unless it is endorsed by you guys, but anyway after several successful cycles, it failed again with lots of:

You can use the reason code to categorize the issues, but this is not entirely reliable, as different scenarios may generate the same reason code.

I have tried this but no success.

Are you saying that using disconnected->reason == WIFI_REASON_BEACON_TIMEOUT for judgment is completely ineffective, or it can work but can't work as well as (disconnected->reason != WIFI_REASON_ASSOC_LEAVE) && (disconnected->reason != WIFI_REASON_NO_AP_FOUND) && (disconnected->reason != WIFI_REASON_AP_TSF_RESET)?

The test 2 1/2/3/4 refer to the four devices in a single round of testing?
where is test 3?

Yes Test 2 1.txt through test 2 4.txt were the 4 devices on a single round of testing
Sorry here are the missing test logs from yesterday
Nov22.zip
Tests 1, 2, 3 were done on the software with:
disconnected->reason == WIFI_REASON_BEACON_TIMEOUT
test 4 was done with"
(disconnected->reason != WIFI_REASON_ASSOC_LEAVE) && (disconnected->reason != WIFI_REASON_NO_AP_FOUND) && (disconnected->reason != WIFI_REASON_AP_TSF_RESET)
Are you saying that using disconnected->reason == WIFI_REASON_BEACON_TIMEOUT for judgment is completely ineffective, or it can work but can't work as well as (disconnected->reason != WIFI_REASON_ASSOC_LEAVE) && (disconnected->reason != WIFI_REASON_NO_AP_FOUND) && (disconnected->reason != WIFI_REASON_AP_TSF_RESET)
Neither work reliably.  after 2 or 3 cycles some nodes will fail and go MESH_IDLE and not scan and the network is broken.

I just reviewed the log for test3, and the device behavior is normal. The device being in the MESH_IDLE state is not permanent; it is a temporary state. Below is my analysis:

At the beginning, the self-organizing network formed the following topology:
root(53:d8) --- node A (39:d4)
|--- node B (72:b8)
|--- node C (5c:68)
node C call manual scan, select A as the better parent, change to layer3 node (self-organized disabled, set parent)
node A call manual scan, still select root as the better parent, still layer2 node (self-organized disabled, set parent)
root power off
node B found root leave, beacon timeout, parent disconnect, enable self-organized, change to be root
node A found root leave, beacon timeout, parent disconnect, enable self-organized. However, at that moment, it was sending data, and it is trying to reconnecting, when you queried, the device was shown as in the MESH_IDLE state. I believe that if the device remains in an idle state and cannot recover, then this is an issue. However, if there are no subsequent logs, I don't consider it a problem. You cannot expect the device to always be in a non-idle state whenever the application layer checks the mesh status.

In the test4 log, the device eventually connected successfully.

The log you referred to is just a part of the intermediate process.

According to your test log, I think the disconnected->reason == WIFI_REASON_BEACON_TIMEOUT will be better than (disconnected->reason != WIFI_REASON_ASSOC_LEAVE) && (disconnected->reason != WIFI_REASON_NO_AP_FOUND) && (disconnected->reason != WIFI_REASON_AP_TSF_RESET) , because there are too many disconnect reason, and it is unreasonable to switch to self-organized mode as soon as the reason is not equal to 8, 201, or 206

How long should it take to for a MESH_IDLE to find a parent again?  I am sure I waited 10s of seconds and it wasn't even scanning.
I am setting up for another run of tests where I will wait longer.  I am just worried about my back trace overflowing using Putty

In the test2 logs, I cannot analyze the entire network change process as I did with test3 because the log only contains part of the information. It is curious why such a reason would occur.

It seems that in test2_2 and test2_3, there was no opportunity to switch to the self-organized network, and the device kept trying to connect to the originally configured parent, but the parent could not be detected.

Hi
Today I am getting problems where nodes get stuck in a loop logging forever.  I can't keep my trace open long enough as I lose the start, but take my word for it please, once in this state it never comes out no matter how long (minutes). eg
I (00:02:07.516) aWifiMesh: <MESH_EVENT_PARENT_DISCONNECTED>reason:139
W (00:02:07.528) aWifiMesh: WiFi Disconnected
>>>last layer = 4, layer = -1
W (00:02:07.529) aWifiMesh: <MESH_EVENT_ROUTING_TABLE_REMOVE>remove 1, new:3
I (128652) mesh: [wifi]disconnected reason:201(), continuous:1/max:12, non-root, vote(,stopped)<><>
I (00:02:07.644) aWifiMesh: <MESH_EVENT_PARENT_DISCONNECTED>reason:201
W (00:02:07.645) aWifiMesh: WiFi Disconnected
>>>last layer = 4, layer = -1
I (128772) mesh: [wifi]disconnected reason:201(), continuous:2/max:12, non-root, vote(,stopped)<><>
I (00:02:07.769) aWifiMesh: <MESH_EVENT_PARENT_DISCONNECTED>reason:201
W (00:02:07.769) aWifiMesh: WiFi Disconnected
>>>last layer = 4, layer = -1
I (128882) mesh: 1145[xrsp:1]the asked:19, max window:2, force to increase/decrease(up) xseqno:17 for child 48:ca:43:9b:5d:20, xrsp_seqno:14, heap:101160
I (128892) mesh: 1307[recv]cidx[0]48:ca:43:9b:5d:20 xseqno loss, current/new:15/19, in:17, out:17, pending:0
I (128892) mesh: [wifi]disconnected reason:201(), continuous:3/max:12, non-root, vote(,stopped)<><>
I (00:02:07.893) aWifiMesh: <MESH_EVENT_PARENT_DISCONNECTED>reason:201
W (00:02:07.894) aWifiMesh: WiFi Disconnected
>>>last layer = 4, layer = -1
I (129022) mesh: [wifi]disconnected reason:201(), continuous:4/max:12, non-root, vote(,stopped)<><>
I (00:02:08.018) aWifiMesh: <MESH_EVENT_PARENT_DISCONNECTED>reason:201
W (00:02:08.019) aWifiMesh: WiFi Disconnected
>>>last layer = 4, layer = -1
I (129142) mesh: [wifi]disconnected reason:201(), continuous:5/max:12, non-root, vote(,stopped)<><>
I (00:02:08.143) aWifiMesh: <MESH_EVENT_PARENT_DISCONNECTED>reason:201
W (00:02:08.144) aWifiMesh: WiFi Disconnected
>>>last layer = 4, layer = -1
I (129272) mesh: [wifi]disconnected reason:201(), continuous:6/max:12, non-root, vote(,stopped)<><>
I (00:02:08.268) aWifiMesh: <MESH_EVENT_PARENT_DISCONNECTED>reason:201
W (00:02:08.269) aWifiMesh: WiFi Disconnected
>>>last layer = 4, layer = -1

Nov 25.zip
I found I am the author of one way that this can happen, if I scan and switch parents manually.  See test1 logs attached where
node A MAC 48:27:e2:18:39:80 switches no node B 48:ca:43:9b:5d:20
Bode B MAC:48:ca:43:9b:5d:20 switches no node A 48:27:e2:18:39:80
This is one cause of the above.  I think I can fix this by checking that I am not switching the nodes parent to one of its children.
I think it probably also makes sense to not swap to a parent node which has a higher layer than this node too.
But while not ideal that I am doing this, it shouldn't result in the node getting stuck in a  disconnect loop?
But Test 2 looks the same problem but is not triggered by the above.  MESH_ROOT is powered off.  A panic crash on a MESH_NODE 48:ca:43:9b:5d:20 which still happens from time to time, but my bigger concern is that after this, MESH_NODE MAC: 48:27:e2:18:39:80 gets stuck in the disconnect loop
Are you able to reproduce these problems with the ip_internal_network you modified a week or so back?  I get lost in all of this and feel we would make better progress if you were able to test, analyze and debug directly.

Hi again
Regarding your analysis of test 3 specifically node A where you said.
node A found root leave, beacon timeout, parent disconnect, enable self-organized. However, at that moment, it was sending data, and it is trying to reconnecting, when you queried, the device was shown as in the MESH_IDLE state. I believe that if the device remains in an idle state and cannot recover, then this is an issue. However, if there are no subsequent logs, I don't consider it a problem. You cannot expect the device to always be in a non-idle state whenever the application layer checks the mesh status.
The disconnect came at 00:01:18
I (00:01:18.371) aWifiMesh: <MESH_EVENT_PARENT_DISCONNECTED>reason:200
I stopped the log after 00:01:58, 40 seconds later and nothing was happening, no visible signs of scanning for a parent.
I am sure that when this occurs, no matter how long I leave it, it does not recover.
Also when it gets stuck in the disconnect loop, logs the repetitive sequence (above) indefinitely.

Two posts back I wrote
_"I found I am the author of one way that this can happen, if I scan and switch parents manually. See test1 logs attached where
node A MAC 48:27:e2:18:39:80 switches no node B 48:ca:43:9b:5d:20
Bode B MAC:48:ca:43:9b:5d:20 switches no node A 48:27:e2:18:39:80
This is one cause of the above. I think I can fix this by checking that I am not switching the nodes parent to one of its children.
I think it probably also makes sense to not swap to a parent node which has a higher layer than this node too.
But while not ideal that I am doing this, it shouldn't result in the node getting stuck in a disconnect loop?"_
When I looked at the code, I am finding it difficult to decipher the variables
parent_record is the best parent candidate found so far
assoc I think is each node found in the scan esp_mesh_scan_get_ap_record(&record, &assoc); is this correct?
I am not clear on what parent_assoc is or how it gets set
It is initialized to:
the original source (taken from the example project "manual_networking") seems to be already checking for this in the line:
               if (assoc.layer < parent_assoc.layer || assoc.layer2_cap < parent_assoc.layer2_cap) {
Do you agree?  Could you take a look at the test 1 logs as they do appear to be setting parent to each other.
Am I correct in thinking this alos looks after checking against switching to a child as a child would be on a higher layer number

Hi
By the way all yesterdays test and logs and today were made with your recommendation of only using
disconnected->reason == WIFI_REASON_BEACON_TIMEOUT
I have been testing getting the 4 nodes stacked up across 4 layers and powering off the NODE on layer 2 rather than the MESH_ROOT as this provide a cleaner set of logs.
See test 1
Layer 1 MESH_ROOT 48:ca:43:9b:53:d8
NODE A Layer 2 48:27:e2:18:39:80
NODE B Layer 3 48:ca:43:9b:54:c0
NODE C Layer 4 48:ca:43:9b:5d:20
Then power down NODE A on layer 2
NODE B switched from layer 3 to layer 2 and parent from NODE A to MESH_ROOT - perfect
NODE C stayed on layer 4 with parent 48:ca:43:9b:54:c1 which is now on layer 2, and does not show in the
Is this valid?
Node B moved from layer 3 to 2 when its parent dropped. Why did Node C not move to layer 3 ?
It stayed like this for minutes while I wrote this up

Regarding the issue of the log infinite loop (aWifiMesh: <MESH_EVENT_PARENT_DISCONNECTED>reason:201), I have already explained it in my previous comment.

It seems that in test2_2 and test2_3, there was no opportunity to switch to the self-organized network, and the device kept trying to connect to the originally configured parent, but the parent could not be detected.

Maybe we should first investigate why the specified parent node cannot be found at this point. Is it due to a power failure, has it become idle, or is there another underlying reason?

Are you able to reproduce these problems with the ip_internal_network you modified a week or so back? I get lost in all of this and feel we would make better progress if you were able to test, analyze and debug directly.

Sorry, I can't reproduce your issue on my side.

I have already discussed this with you in my previous comment: when selecting a better parent node, what criteria do you prioritize? I believe you can completely disregard the conditions in the example and instead design your own criteria based on your specific needs. First, you can move the definitions of parent_assoc and parent_record outside,

then update the parent_assoc->layer when connecting to the parent.

Before scanning, retrieve the current parent information.

Finally, within the findClosestParent() function, design the criteria for selecting a better parent based on your requirements and the issues encountered during testing.

My thoughts were, I am not wanting to build the mesh network from scratch as I start with a self configured network. I am only planning to make changes to nodes with poor RSSIs. So far my tests have been successful network architecture wise (when I have a fixed ROOT so I don't get the broken mesh problem).

You mentioned that you don't want to rebuild the network from scratch, but instead, you want to adjust the initial network formed by the self-organizing process. However, during the actual testing, I've observed that you often call scan at the application layer while the initial network is still being formed, which forcibly interrupts the self-organizing process.

So, when you call scan at the application layer, is it completely random? Would it make sense to first check whether the initial network has been fully formed before manually triggering the scan?
I believe we must first resolve the issues mentioned in points 3 and 4 before proceeding with further problem analysis. If the initial logic framework isn't properly established, it could lead to a range of unforeseen issues down the line, which would be quite painful for me to handle.

The problem I am trying to solve all the way through this is triggered by the MESH_ROOT or the parent of a node being powered off or rebooted. So yes definitely I have been doing both powering off and rebooting the parent and both trigger this issue.  So this is why parent can not be found at this point, the question is how to fix it.  This can and will happen in the field.
The latest catch of one of these today, I powered off the parent on layer 2 and the MESH_NODE on layer 3 continued to report this see attached.  This was made after the changes documented in point 3 below.
MESH_EVENT_PARENT_DISCONNECTED.txt
So the symptoms are continuous <MESH_EVENT_PARENT_DISCONNECTED> events OR nothing.  Node just sits in MESH_IDLE state and does nothing.

I have made the changes you suggested, but as you commented this would stop my nodes finding a better parent.  So I modified the event <MESH_EVENT_PARENT_CONNECTED> to:
parent_assoc.layer = mesh_layer; // MBS consider can switch to parent on same level or better

and function:
void findClosestParent(int num) { // after a WiFi scan
...
                if (assoc.layer <= parent_assoc.layer || assoc.layer2_cap < parent_assoc.layer2_cap) { // parents on same layer or better qualify
...

I understand this could drop a node up (numerically) one layer, but I think this is acceptable as it is much better to a further layer back with a stronger RSSI from my testing.  Do you understand and accept this strategy, at least as far as not causing the broken network issue?

My background task only scans for a better parent, if (esp_mesh_get_type() != MESH_IDLE), as I assumed this meant the network was established.
As mentioned earlier, I have been manually triggering the scan from my console (my main task) when I consider the network to be established based on the event ip_event_handler triggering which calls my printMeshInfo(); function.
Two questions:

1 You said Would it make sense to first check whether the initial network has been fully formed before manually triggering the scan?  What is a suitable test to determine the network is fully formed?
2 Am I potentially doing too much in the event_handler functions?  Is there any restriction on how long they can run (within reason of course)?
3 How long should it take to for a node which is disconnected or MESH_IDLE to start to search for a parent again or scan for MESH_ROOT if the MESH_ROOT disappears?  Just roughly eg 10 seconds, 30 seconds, longer?  I have waited 10s of minutes while writing up logs etc...
Also once the network is broken and not able to fix itself, I sometimes execute the following commands from my console to try and fix it.
wifi scan - triggers my wifiMeshScan() function to look for a new parent. But only if the node is not MESH_IDLE.
wifi root - triggers my wifiMeshRoot() function which calls ESP_ERROR_CHECK(esp_mesh_set_self_organized(true, true));
But neither of these has ever fixed it.

For the <MESH_EVENT_PARENT_DISCONNECTED>reason:201 loop issue, is it possible to add a timer or counter, so that once the time or count exceeds a certain threshold, the function esp_mesh_set_self_organized(true, true) is called to reselect the parent?
How about directly comparing with mesh_layer ?

void findClosestParent(int num) { // after a WiFi scan
...
                if (assoc.layer <= mesh_layer || assoc.layer2_cap < parent_assoc.layer2_cap) { // parents on same layer or better qualify
...

My background task only scans for a better parent, if (esp_mesh_get_type() != MESH_IDLE), as I assumed this meant the network was established.

Why does this phenomenon occur if you only call scanning when it is not idle? It's obvious that the device hasn't been connected to WiFi yet.

4.

How long should it take to for a node which is disconnected or MESH_IDLE to start to search for a parent again or scan for MESH_ROOT if the MESH_ROOT disappears?

If the device is in self-organized network, it will attempt to reconnect the original parent for at least 6 seconds before selecting a new parent.
5.

wifi root - triggers my wifiMeshRoot() function which calls ESP_ERROR_CHECK(esp_mesh_set_self_organized(true, true));

Have you ever called this command during the testing process? I don't think I saw this command in the log.

wifi scan - triggers my wifiMeshScan() function to look for a new parent. But only if the node is not MESH_IDLE.

The function of this scan cannot cover all situations because the judgment criteria are relatively single. Perhaps we can add more judgments？

Yes I am happy to add this if it worked.  But as I said for now I can invoke esp_mesh_set_self_organized(true, true) manually from my console command "mesh root".  See logs attached, where I rebooted the MESH_ROOT to start the problems:
Nov 29.zip

test 1 mesh root
line 378 Reboot the mesh root
test 1 became 2nd root
Line 524 the disconnect loop started.
Line 946 invoke wifi root command
Node becomes MESH_ROOT but there already is a MESH_ROOT
test 1 became node on layer 5
Line 126 starts logging I (65677) wifi:>>>intv = 102400, max = 0.  I don't know what this means
Line 437 disconnect loop starts
line 1621 run wifi root command
Node becomes MESH_NODE on layer 5 to parent  48:ca:43:9b:53:d9 which is on layer 2
test 1 node 48ca439b53d8
for your reference.  Ends up mesh node on layer 2 to the second MESH_ROOT

You said "How about directly comparing with mesh_layer ?"
I don't understand what you mean here. Please elaborate

You said "Why does this phenomenon occur if you only call scanning when it is not idle? It's obvious that the device hasn't been connected to WiFi yet."