OTA Updates Issues ESP32 with code examples: 0009 and 0011

I am trying to implement OTA updates on ESP32 devices. I am using Thingsborad as the platform to store my firmware versions.
My goal is to upload different firmware versions (of the same Title) from the "OTA updates" tab from ThingsBoard.
I am using the code provided over: https://github.com/thingsboard/thingsboard-arduino-sdk/tree/master/examples/0009-esp8266_esp32_process_OTA_MQTT and also https://github.com/thingsboard/thingsboard-arduino-sdk/tree/master/examples/0011-esp8266_esp32_subscribe_OTA_MQTT.
The device is succesfully conecting to Thingsboard, and set the Tittle and Version of the Firmware, in the Dashboard that is gotten from the code "esp8266_esp32_subscribe_OTA_MQTT" or "esp8266_esp32_process_OTA_MQTT"

Then it will compare the "Target version" assigned to the device to this hardcoded Version in the code.
1- If the Tittle of the Target Firmware doesn't match the Current Firmware Tittle: It will fail to update the Target firmware to the device - This is expected behavior.
2- If the tittles match, BUT the Target version is the same: It will fail to upload the Target firmware, evethough the device is not really ruuning the Current firmware version. - As per my understanding this is not expected behavior.
3- If the tittles match, and the Target version is the different to the hardcoded Current version, the device will download and update its firmware to the Target Firmware. However after this comes the issue:
a. The device doen't update the Current Firmware version with the corresponding version that is running after it was sucesfully downloaded and configured from Thingsboard.
b. The device "losses" connection to Thingboard making it impossible to update anymore the Firmware that it is currently ruunning.
I would like to understand if the capabilities of the examples provided in this Repository, as well as if there is any further documentation to understand how to set "real-time" firmware updates to a device over the air, while using Thingsboard platform.
Thanks in advanced.

For point number 2 it is expected behaviour, the version you flash over OTA needs to have a different version name, because if it is the same version name it is expected to be the exact same code as is currently running on the device.
Meaning it assumes it does not have to update, if the version is different but the title is the same it knows that the code is different and that is should update to that version.
Because the hard-coded version of the device reflects the current version that is on the device.
To fix your example simple call the firmware binary you want to flash 1.1.0 instead of 1.0.0.

a. The device doen't update the Current Firmware version with the corresponding version that is running after it was sucesfully downloaded and configured from Thingsboard.

What firmware binary are you flashing onto the device, because the portion that should upload the current version is this code portion in the main loop.
if (!currentFWSent) {
  currentFWSent = tb.Firmware_Send_Info(CURRENT_FIRMWARE_TITLE, CURRENT_FIRMWARE_VERSION)
                  && tb.Firmware_Send_State(FW_STATE_UPDATED);
}

If the flashes firmware does not have these constant variables set correctly.
constexpr char CURRENT_FIRMWARE_TITLE[] = "TEST";
constexpr char CURRENT_FIRMWARE_VERSION[] = "1.0.0";

It will send a version which it doesn't even have.

b. The device "losses" connection to Thingboard making it impossible to update anymore the Firmware that it is currently ruunning.

If this is the case and not the problem mentioned above with point a), then it would be nice if you could attach some log messages of the disconnection and the failure to send the firmware status.

@MathewHDYT thanks for the quick response.
Basically what I am seeing right now, with the base code from the Repository, with both example 009 and 011 is that when this code is uploaded to the ESP32, it will not run FW1. Wich will be define in Thingsboard as:
constexpr char CURRENT_FIRMWARE_TITLE[] = "BLINK";
constexpr char CURRENT_FIRMWARE_VERSION[] = "1.0.0";
Blinkv10.ino.txt
I changed .bin to .bin just to attach the file
This version of the FW will be updated in Thingsboard Dashboard as the Current FW: Tittle: BLINK and Version: 1.0.0. At this point in the execution the ESP32 is not executing the FW: BLINK v1.0.0.
After this, when setting the Target FW as BLINK version 1.1.0; the Dashboard shows how this new FW version is now downloaded and sucesfully loaded in the ESP32. At this point of the execution, the ESP32, starts "blinking", this means that it starts running the Target FW v1.1.0. Up to this point everything is working "fine". Bellow I am ataching the code used for BLINK v1.1.0 - It is basically the same code as BLINK v1.0.0 but the freq of the LED is changed.
Blinkv05.ino.txt
Now, comes my issue. After this new FW : BLINK v1.1.0 is running in the ESP32, and I can tell is running because of the Arduino console and the flashing LED in the ESP32, my device has no longer communication to update its current FW version, or recieve new FW updated from THingsboard. This is the main issue I am having. My question is if the example code (009 or 011) should support this functionality? Or if there is any missing configuration step that is required in the ESP32 in order to have both the FW BLINK v1.1.0 running - enabling my LED to blink at any freq, AND parallel to this keep my connection active with Thingsboard in order to tell my device when there is a new FW update that needs to be downloaded.
I hope this message is clear, thanks again for the support.

This version of the FW will be updated in Thingsboard Dashboard as the Current FW: Tittle: BLINK and Version: 1.0.0. At this point in the execution the ESP32 is not executing the FW: BLINK v1.0.0.

As mentioned before this will not work because the Dashboard Firmware has the version 1.0.0 and the current Firmware Version constant on your device has the same version.
Updates are disallowed for same version, because we assume it is the same code. Therefore updating has no tangible benefit nor reason, if after the update the device still has the exact same code and functionalities.
I hope this portion is more understandable now.

For the second issue are you really sure the device has not connection again, a simple way to test is to see if the version is resent after the device has updated.
Specifically on your device that is updated, check the shared attributes for fw title / version and check if they changed very recently after the device has rebooted.

If they did than your device could establish a connection and has restarted and resend its firmware title / version successfully.
Furthermore this would mean your problem is simply that the binary you created (Blinkv05.txt) still has the internal constant  set to 1.0.0. Instead of what it should be 1.1.0. See below for the code that should be in your Blinkv05.ino
constexpr char CURRENT_FIRMWARE_TITLE[] = "BLINK";
constexpr char CURRENT_FIRMWARE_VERSION[] = "1.1.0";

I hope this is your issue if not there might be some WiFi connection issues which we would have to debug further.

Please correct me if I am wrong, but the ".bin" file that I am generating is based on this code:
#define INTERNAL_LED 33
void setup() {
pinMode(INTERNAL_LED,OUTPUT);
Serial.begin(115200);
}
void loop() {
digitalWrite(INTERNAL_LED,HIGH);
Serial.println("LED ON");
delay(500);
digitalWrite(INTERNAL_LED,LOW);
Serial.println("LED OFF");
delay(500);
}
And this is the only code being uploaded to the ESP32 with the OTA. Is this enough?
Or this simple code should be integrated with the example 0011-esp8266_esp32_subscribe_OTA_MQTT? If so, in what specific sections of the code the "functional code" should be added? in this simple case it is just a digital write to turn ON and OFF an LED.

Okay of course that makes sense I understand now where your problem is coming  from.
You are flashing a complete new firmware binary onto your device, so you need to think of your device only doing what you told it to in the script you flashed on to it.
Meaning it has no functionality to connect with ThingsBoard or send its version it simply can toggle the LED.
To implement that functionality you have to merge the code above with the example and flash that onto the device instead.
You'll find the merged examples below I hope those help :D, this would be merged with example-009, as you can see you simply need to add the code you have in setup and loop to the methods in the already existing loop and setup of the example.
#if defined(ESP8266)
#include <ESP8266WiFi.h>
// Disable PROGMEM because the ESP8266WiFi library,
// does not support flash strings.
#define THINGSBOARD_ENABLE_PROGMEM 0
#elif defined(ESP32)
#include <WiFi.h>
#include <WiFiClientSecure.h>
#endif

#include <ThingsBoard.h>

// Additional include added to toggle internal LED
#define INTERNAL_LED 33

// Whether the given script is using encryption or not,
// generally recommended as it increases security (communication with the server is not in clear text anymore),
// it does come with an overhead tough as having an encrypted session requires a lot of memory,
// which might not be avaialable on lower end devices.
#define ENCRYPTED false

// Firmware title and version used to compare with remote version, to check if an update is needed.
// Title needs to be the same and version needs to be different --> downgrading is possible
#if THINGSBOARD_ENABLE_PROGMEM
constexpr char CURRENT_FIRMWARE_TITLE[] PROGMEM = "TEST";
constexpr char CURRENT_FIRMWARE_VERSION[] PROGMEM = "1.0.0";
#else
constexpr char CURRENT_FIRMWARE_TITLE[] = "TEST";
constexpr char CURRENT_FIRMWARE_VERSION[] = "1.0.0";
#endif

// Firmware state send at the start of the firmware, to inform the cloud about the current firmware and that it was installed correctly,
// especially important when using OTA update, because the OTA update sends the last firmware state as UPDATING, meaning the device is restarting
// if the device restarted correctly and has the new given firmware title and version it should then send thoose to the cloud with the state UPDATED,
// to inform any end user that the device has successfully restarted and does actually contain the version it was flashed too
#if THINGSBOARD_ENABLE_PROGMEM
constexpr char FW_STATE_UPDATED[] PROGMEM = "UPDATED";
#else
constexpr char FW_STATE_UPDATED[] = "UPDATED";
#endif

// Maximum amount of retries we attempt to download each firmware chunck over MQTT
#if THINGSBOARD_ENABLE_PROGMEM
constexpr uint8_t FIRMWARE_FAILURE_RETRIES PROGMEM = 5U;
#else
constexpr uint8_t FIRMWARE_FAILURE_RETRIES = 5U;
#endif
// Size of each firmware chunck downloaded over MQTT,
// increased packet size, might increase download speed
#if THINGSBOARD_ENABLE_PROGMEM
constexpr uint16_t FIRMWARE_PACKET_SIZE PROGMEM = 4096U;
#else
constexpr uint16_t FIRMWARE_PACKET_SIZE = 4096U;
#endif

// PROGMEM can only be added when using the ESP32 WiFiClient,
// will cause a crash if using the ESP8266WiFiSTAClass instead.
#if THINGSBOARD_ENABLE_PROGMEM
constexpr char WIFI_SSID[] PROGMEM = "YOUR_WIFI_SSID";
constexpr char WIFI_PASSWORD[] PROGMEM = "YOUR_WIFI_PASSWORD";
#else
constexpr char WIFI_SSID[] = "YOUR_WIFI_SSID";
constexpr char WIFI_PASSWORD[] = "YOUR_WIFI_PASSWORD";
#endif

// See https://thingsboard.io/docs/getting-started-guides/helloworld/
// to understand how to obtain an access token
#if THINGSBOARD_ENABLE_PROGMEM
constexpr char TOKEN[] PROGMEM = "YOUR_DEVICE_ACCESS_TOKEN";
#else
constexpr char TOKEN[] = "YOUR_DEVICE_ACCESS_TOKEN";
#endif

// Thingsboard we want to establish a connection too
#if THINGSBOARD_ENABLE_PROGMEM
constexpr char THINGSBOARD_SERVER[] PROGMEM = "demo.thingsboard.io";
#else
constexpr char THINGSBOARD_SERVER[] = "demo.thingsboard.io";
#endif

// MQTT port used to communicate with the server, 1883 is the default unencrypted MQTT port,
// whereas 8883 would be the default encrypted SSL MQTT port
#if ENCRYPTED
#if THINGSBOARD_ENABLE_PROGMEM
constexpr uint16_t THINGSBOARD_PORT PROGMEM = 8883U;
#else
constexpr uint16_t THINGSBOARD_PORT = 8883U;
#endif
#else
#if THINGSBOARD_ENABLE_PROGMEM
constexpr uint16_t THINGSBOARD_PORT PROGMEM = 1883U;
#else
constexpr uint16_t THINGSBOARD_PORT = 1883U;
#endif
#endif

// Maximum size packets will ever be sent or received by the underlying MQTT client,
// if the size is to small messages might not be sent or received messages will be discarded
#if THINGSBOARD_ENABLE_PROGMEM
constexpr uint32_t MAX_MESSAGE_SIZE PROGMEM = 512U;
#else
constexpr uint32_t MAX_MESSAGE_SIZE = 512U;
#endif

// Baud rate for the debugging serial connection
// If the Serial output is mangled, ensure to change the monitor speed accordingly to this variable
#if THINGSBOARD_ENABLE_PROGMEM
constexpr uint32_t SERIAL_DEBUG_BAUD PROGMEM = 115200U;
#else
constexpr uint32_t SERIAL_DEBUG_BAUD = 115200U;
#endif

#if ENCRYPTED
// See https://comodosslstore.com/resources/what-is-a-root-ca-certificate-and-how-do-i-download-it/
// on how to get the root certificate of the server we want to communicate with,
// this is needed to establish a secure connection and changes depending on the website.
#if THINGSBOARD_ENABLE_PROGMEM
constexpr char ROOT_CERT[] PROGMEM = R"(-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----
)";
#else
constexpr char ROOT_CERT[] = R"(-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----
)";
#endif
#endif

// Initialize underlying client, used to establish a connection
#if ENCRYPTED
WiFiClientSecure espClient;
#else
WiFiClient espClient;
#endif
// Initialize ThingsBoard instance with the maximum needed buffer size
ThingsBoard tb(espClient, MAX_MESSAGE_SIZE);

// Statuses for updating
bool currentFWSent = false;
bool updateRequestSent = false;

/// @brief Initalizes WiFi connection,
// will endlessly delay until a connection has been successfully established
void InitWiFi() {
#if THINGSBOARD_ENABLE_PROGMEM
  Serial.println(F("Connecting to AP ..."));
#else
  Serial.println("Connecting to AP ...");
#endif
  // Attempting to establish a connection to the given WiFi network
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  while (WiFi.status() != WL_CONNECTED) {
    // Delay 500ms until a connection has been succesfully established
    delay(500);
#if THINGSBOARD_ENABLE_PROGMEM
    Serial.print(F("."));
#else
    Serial.print(".");
#endif
  }
#if THINGSBOARD_ENABLE_PROGMEM
  Serial.println(F("Connected to AP"));
#else
  Serial.println("Connected to AP");
#endif
#if ENCRYPTED
  espClient.setCACert(ROOT_CERT);
#endif
}

/// @brief Reconnects the WiFi uses InitWiFi if the connection has been removed
/// @return Returns true as soon as a connection has been established again
bool reconnect() {
  // Check to ensure we aren't connected yet
  const wl_status_t status = WiFi.status();
  if (status == WL_CONNECTED) {
    return true;
  }

  // If we aren't establish a new connection to the given WiFi network
  InitWiFi();
  return true;
}

/// @brief Updated callback that will be called as soon as the firmware update finishes
/// @param success Either true (update succesfull) or false (update failed)
void updatedCallback(const bool& success) {
  if (success) {
#if THINGSBOARD_ENABLE_PROGMEM
    Serial.println(F("Done, Reboot now"));
#else
    Serial.println("Done, Reboot now");
#endif
#if defined(ESP8266)
    ESP.restart();
#elif defined(ESP32)
    esp_restart();
#endif
    return;
  }
#if THINGSBOARD_ENABLE_PROGMEM
  Serial.println(F("Downloading firmware failed"));
#else
  Serial.println("Downloading firmware failed");
#endif
}

/// @brief Progress callback that will be called every time we downloaded a new chunk successfully
/// @param currentChunk
/// @param totalChuncks
void progressCallback(const uint32_t& currentChunk, const uint32_t& totalChuncks) {
  Serial.printf("Progress %.2f%%\n", static_cast<float>(currentChunk * 100U) / totalChuncks);
}

const OTA_Update_Callback callback(&progressCallback, &updatedCallback, CURRENT_FIRMWARE_TITLE, CURRENT_FIRMWARE_VERSION, FIRMWARE_FAILURE_RETRIES, FIRMWARE_PACKET_SIZE);

void setup() {
  // Initalize serial connection for debugging
  Serial.begin(SERIAL_DEBUG_BAUD);
  delay(1000);
  InitWiFi();
// ----------- Added code -----------------------------------------------------------------------
  // Added Pinmode initalization to signal that the given pin is an output pin that will be toggled high or low
  pinMode(INTERNAL_LED, OUTPUT);
// ----------- Added code -----------------------------------------------------------------------
}

void loop() {
// ----------- Added code -----------------------------------------------------------------------
  // Added block to toggle LED, has a total delay of 500 + 500 = 1000, so the actual delay is the same
  // so there is not effect on how fast a device reconnects if a connection has been lost.
  digitalWrite(INTERNAL_LED,HIGH);
  Serial.println("LED ON");
  delay(500);
  digitalWrite(INTERNAL_LED,LOW);
  Serial.println("LED OFF");
  delay(500);
// ----------- Added code -----------------------------------------------------------------------

  if (!reconnect()) {
    return;
  }

  if (!tb.connected()) {
    // Reconnect to the ThingsBoard server,
    // if a connection was disrupted or has not yet been established
    Serial.printf("Connecting to: (%s) with token (%s)\n", THINGSBOARD_SERVER, TOKEN);
    if (!tb.connect(THINGSBOARD_SERVER, TOKEN, THINGSBOARD_PORT)) {
#if THINGSBOARD_ENABLE_PROGMEM
      Serial.println(F("Failed to connect"));
#else
      Serial.println("Failed to connect");
#endif
      return;
    }
  }

  if (!currentFWSent) {
    currentFWSent = tb.Firmware_Send_Info(CURRENT_FIRMWARE_TITLE, CURRENT_FIRMWARE_VERSION) && tb.Firmware_Send_State(FW_STATE_UPDATED);
  }

  if (!updateRequestSent) {
#if THINGSBOARD_ENABLE_PROGMEM
    Serial.println(F("Firwmare Update..."));
#else
    Serial.println("Firwmare Update...");
#endif
    // See https://thingsboard.io/docs/user-guide/ota-updates/
    // to understand how to create a new OTA pacakge and assign it to a device so it can download it.
    updateRequestSent = tb.Start_Firmware_Update(callback);
  }

  tb.loop();
}

Thanks for the detailed response.
I am now tryign with this FW, by changing the delay time between ON and OFF as the different FW versions.
However in the Serial Console I am only seeing the "LED ON" and "LED OFF", running at the initial freq. This is properly uploaded in Thingsboard Dashboard. On the other, when I select a new Target FW, it doesn't download the new FW to the device.

FW1:
...
#if THINGSBOARD_ENABLE_PROGMEM
constexpr char CURRENT_FIRMWARE_TITLE[] PROGMEM = "BlinkCAM";
constexpr char CURRENT_FIRMWARE_VERSION[] PROGMEM = "1.2.0";
#else
constexpr char CURRENT_FIRMWARE_TITLE[] = "BlinkCAM";
constexpr char CURRENT_FIRMWARE_VERSION[] = "1.2.0";
#endif
...
Serial.println("LED ON");
delay(2000);
digitalWrite(INTERNAL_LED,LOW);
Serial.println("LED OFF");
delay(2000);
...
FW2:
...
#if THINGSBOARD_ENABLE_PROGMEM
constexpr char CURRENT_FIRMWARE_TITLE[] PROGMEM = "BlinkCAM";
constexpr char CURRENT_FIRMWARE_VERSION[] PROGMEM = "1.0.5";
#else
constexpr char CURRENT_FIRMWARE_TITLE[] = "BlinkCAM";
constexpr char CURRENT_FIRMWARE_VERSION[] = "1.0.5";
#endif
...
Serial.println("LED ON");
delay(500);
digitalWrite(INTERNAL_LED,LOW);
Serial.println("LED OFF");
delay(500);
...
Any advice or idea of what might be the issue?

The other question that I have is: Since we are setting this parameter in the FW, ".bin" :
#if THINGSBOARD_ENABLE_PROGMEM
constexpr char TOKEN[] PROGMEM = "YOUR_DEVICE_ACCESS_TOKEN";
#else
constexpr char TOKEN[] = "YOUR_DEVICE_ACCESS_TOKEN";
#endif
How can I manage to have different devices associated with 1 common program?
For example, I have 5 devices working under the same FW1, and now I want to move all 5 devices to FW2. How can I manage to change the program for all 5 devices? Do I need to upload in Thingsboard 1 .bin per device? That would be too inefficient.

However in the Serial Console I am only seeing the "LED ON" and "LED OFF", running at the initial freq. This is properly uploaded in Thingsboard Dashboard. On the other, when I select a new Target FW, it doesn't download the new FW to the device

As mentioned above the combined example I provided is for example-009. Which initially sends a request to update but does not react to later setting the firmware to make the above code into example-011 simply change these lines.
  if (!updateRequestSent) {
#if THINGSBOARD_ENABLE_PROGMEM
    Serial.println(F("Firwmare Update..."));
#else
    Serial.println("Firwmare Update...");
#endif
    // See https://thingsboard.io/docs/user-guide/ota-updates/
    // to understand how to create a new OTA pacakge and assign it to a device so it can download it.
    updateRequestSent = tb.Start_Firmware_Update(callback);
  }

To this
if (!updateRequestSent) {
#if THINGSBOARD_ENABLE_PROGMEM
    Serial.println(F("Firwmare Update Subscription..."));
#else
    Serial.println("Firwmare Update Subscription...");
#endif
    // See https://thingsboard.io/docs/user-guide/ota-updates/
    // to understand how to create a new OTA pacakge and assign it to a device so it can download it.
    updateRequestSent = tb.Subscribe_Firmware_Update(callback);
  }

How can I manage to have different devices associated with 1 common program?
For example, I have 5 devices working under the same FW1, and now I want to move all 5 devices to FW2. How can I manage to change the program for all 5 devices? Do I need to upload in Thingsboard 1 .bin per device? That would be too inefficient.

If the devices have the same hardware (device profile) you can use the same device profile for the OTA binary and simply assign them, that is exactly what OTA is meant to do. See ThingsBoard OTA Documentation for more information.
Assigning multiple devices at once would be possible but for that you would need to write or find a Widget that does that, because the Default Firmware Dashboard you are currently using only allows setting the firmware of the device one by one.
If you need help perhaps the Main Repository can help with creating that widget, because that is not connected to this specific library, but the ThingsBoard site itself.

"If the devices have the same hardware (device profile) you can use the same device profile for the OTA binary and simply assign them, that is exactly what OTA is meant to do. See ThingsBoard OTA Documentation for more information."
Related to this statement, I wonder if I have an ESP32 device (D1) already set up and running with FW1, which is set with Token A. And I want to upload the same FW functionality to my new ESP32 device (D2), do I need to create a new FW1 (FW2) that points to the corresponding Token B, that will be associated with my device? Or can both devices, D1 and D2, be uploaded with the same FW1, which has the TokenA hardcoded?

You can upload the same FW to both devices, that is possible because the OTA packages does not directly reference a device but a Device Profile instead.
You can think of a Device Profile as specific Hardware that uploads, and combines multiple actual device that all serve the same purpose (for example measuring temperature).

This allows you to assign the OTA update to any device that shares the same Device Profile, because if it has the same hardware it is expected to work with the same Firmware.
The token that is used to conenct with a registered device on ThingsBoard is seperate and has nothing to do with the OTA update directly or the possiblity to assign firmware.

OK, got it. But in that case, when I see the Dashboard, how can I diferentiate D1 from D2?
For example if for any reason the update fails in D1, but sucesfully finishes in D2. I thought this TOKEN ID allowed me to get information from the device.

It still does because D1 and D2 are two separate devices (they have a separate TOKEN ID), they have a separate progress bar and on the OTA Update Dashboard all devices are separated from each other.
What they share is only the Device Profile, see ThingsBoard Documentation for more information.
Meaning if D1 fails you can see that on the Dashboard and if D2 doesn't fail you can also see that. D1s success or failure has not impact on D2s update progress.

OK, I understand that D1 and D2 have different Tokens. But that is exactly my concern.. Because in the Example 011. there is a parameter that explicitly ask for this Token:

When I work with a common FW to upload to D1 and D2; whichone of the 2 available Tokens should be used? Or which value should be used then in my common FW to allow Thingsboard to identify D1 and D2, when they are both running the same FW, that has set a single Token by default.
Both D1 nd D2 are defined under the same Device Profile, and I see that my Profile has a specific ID, however I am not sure if this can be used.

Ah sorry I think I misunderstood.
If you want to use the exact same firmware with hardcoded Access tokens that of course won't work as you mentioned already.
This is not directly done in the examples because they are meant as starting points and not full feature completes scripts.
To fix that that issue you would need to save the access token and read it from storage that isn't overwritten on OTA update instead.
That would on the ESP32 be either Non-volatile Storage (NVS), SPIFFS or a SD card.
Both have their disadvantages and advantages if you want to read more I directly attached links to the ESP documentation. But in short form NVS is meant for key-value storage of small values the maximum you can store is rather small per key-value pair. Where as SPIFFS is meant for complete files and can store big files.
Meaning NVS is meant for small key-value pairs exactly like your Access Token and SPIFFS is more meant for log or configuration files.

Therefore I would recommend writing the access token once into NVS of each of your device and read it into RAM on Startup and use that to establish a connection with the cloud.
This part of the NVS documentation should help you getting started with writing and reading NVS especially for strings.

Thanks a lot for the explanaition. I'll look at this documentation.
One last question in this same line of paring and identifying devices in Thingsboard. I see that for each device one can create a Certificate in PEM format:

For security and privacy purposes, I'd like to know how this can be integrated with the current example 011?
Also, I'd like to understand how is this "X.509" credential related to the boolean "ENCRYPTED" variable from the example code?
When working with more than 1 device, is this Certificate (in the FW) unique? Or it can be unique to eachone of my devices?

Also I am trying to enable for the ESP826 the same Blink example, for OTA, based on the example 011 as well.
For the ESP32 it is currently working smoth for 1 device.
However for the ESP8266, I am getting this error while compiling:

Is this error somehow expected for this board?
As a next step I'd like also to try to enable OTA in Tingsboard for Raspberry Pi, is there any documentation available for this? Or any feedback from you to take intoaccount for this?

The private key as far as I know should be unique for each device and stored in storage that isn't wiped on OTA update as well.

For the ESP8266, that is only a warning and expected, because there is a check to ensure the array isn't bigger than the stack size if it is it is created on the heap instead.

For the Raspberry Pi, currently OTA is not supported for non ESP32 boards, that is mainly because the firmware has to be written to the binary and in the current implementation Update and Updater are used which only exist on the EP32 and ESP8266.
If you can find an alternative that works for non ESP boards it might be possible to implement it, but it would require some effort.


Docs：Spelling bug

Describe the bug
Feedback on a question from the bioos documentation center.
One of the references is from this scriptCramToBam.wdl · joy/seq-format-conversion01 - 码云 - 开源中国 (gitee.com)
Spelling：‘tuntime’ should be ‘runtime’
To Reproduce
None.
Expected behavior
One or more issue templates filed.
Screenshots

it has been fixed


development/tools/devpi-server: No module named 'py'

Describe the bug
Devpi server seems to be missing its py>=1.4.23 dependency
Steps To Reproduce
Steps to reproduce the behavior:

Run devpi-server --help from c002c6aa977ad22c60398daaa9be52f2203d0006

Screenshots
$ nix-shell -p devpi-server --run "devpi-server --help"
Traceback (most recent call last):
  File "/nix/store/wxx4bj7cxb9a2znqrxq0dljx48c77m0q-devpi-server-6.9.2/bin/.devpi-server-wrapped", line 6, in <module>
    from devpi_server.main import main
  File "/nix/store/wxx4bj7cxb9a2znqrxq0dljx48c77m0q-devpi-server-6.9.2/lib/python3.11/site-packages/devpi_server/main.py", line 11, in <module>
    import py
ModuleNotFoundError: No module named 'py'

Additional context
Add any other context about the problem here.
Notify maintainers
@makefu

Add a :+1: reaction to issues you find important.

Thanks for the report, it seems like py is now a runtime dependency of devpi-server. i've created a PR, you can use the changes there as a hotfix.


CharString variable declaration in GDNative code causes crash when that method is called from GDScript

Godot version
3.5.1.stable
System information
WIndows 10
Issue description
When a CharString variable is declared in a C++ method called by GDScript, Godot will crash shortly after, possibly during, the return from the called C++ method.
I think this was crashing under linux as well, but most of my testing and debugging of this crash was performed under windoze.
I ran into the problem on another project where I was initially using a CharString to convert a Godot::String to a C/C++ string before I found String::alloc_c_string().  However, I left the CharString declaration in that code and was getting random crashes on the first call, unless I had a print() statement immediately before it in the calling GDScript code.  Once I removed the declaration of the unused CharString variable, the crashes stopped.
To verify that removing the CharString variable declaration was really a fix, I added one to the GDNative C++ Example code.  That code now crashes as well.
Steps to reproduce
Start with the GDExample class in the GDNative C++ Example tutorial.  Declare a CharString variable in the _process() method:
void GDExample::_process(float delta) {
  CharString cs;                          // <------- Added this line
  time_passed += speed * delta;

  Vector2 new_position = Vector2(
				 (real_t) (amplitude + (amplitude * sin(time_passed * 2.0))),
				 (real_t) (amplitude + (amplitude * cos(time_passed * 1.5)))
				 );

  set_position(new_position);
}

Build the code.
Run the main.tscn scene.
It should crash.
Minimal reproduction project
GDNative C++ Example

Note: https://github.com/godotengine/godot-cpp/issues/842


disk: sdhc: Remove logically dead code from spi sdhc driver

We already returned out of the function if err is nonzero, therefore it
is impossible to reach this return statement.
Coverity-CID: 205612
Signed-off-by: Maureen Helm<EMAIL_ADDRESS>Fixes #20511

This code was probably meant to look like this:
if ((ocr & SDHC_BUSY) == 0U) {
	/* Card never finished power-up */
	return -ETIMEDOUT;
}

I feel like we need to revisit the code, cause maybe there are some conditions under which we should return TIMEOUT


clang-tidy: Suppress misc-include-cleaner

The
misc-include-cleaner
warning points out any places where types, functions etc. are being used without the
relevant headers being directly included in the source file. While this might sound
like a good idea in principle, in practice it means that there are an absurd number of
warnings being generated for the code base and every PR from now on will likely have a
few more. The benefit seems dubious so let's just disable it.

Might be worth eventually opening a PR to do the required steps though. I'm often confused by unused imports or where something is imported from and how indirectly. For now we can disable it, but let's make an issue to go through it properly later.

We can do, but the only thing is it's likely to flag up things on new PRs and we'll only notice once the bot has reviewed it.  If we do want to enable it, we might want to use it in combination with this pre-commit hook which runs include-what-you-use


[Feature] Wiki in other lanquage

Hello,
nice work you've done on firmware development and this Wiki.
I read you point to specific translated manuals, but ONLINE the Wiki is only in English (+French) lanquage
With this line you can provide Online Wiki in other lanquage with Online translation with in Github.
https://github-com.translate.goog/armel/uv-k5-firmware-custom/wiki?_x_tr_sl=en&_x_tr_tl=nl
Suggest to add some lines on Home of Wiki
Here is the format (EDIT this issue to see it)
Translations
The Wiki can be translated into other languages using Google translate. Four languages are predefined below, but if you open one of them, you will be able to select other languages at the top of the translated page.

Tłumaczenie polskie / Polish translation
Nederlandse vertaling / Dutch translation
Traduzione italiana / Italian translation
Traduction française / French translation

If you need help, just give a reply.
Greetings Erik

Hello Erik,
Good idea.
Done !
Armel F4HWN.

Top.


node_modules\q\q.js:155 unexpected token /

Expected behaviour
run mb --configfile xxx.ejs --allowInjection
inject ejs file correctly
...
Actual behaviour
node_modules\q\q.js:155 unexpected token /
...
Steps to reproduce
...
Software versions used
OS         : windows
mountebank :1.6.0
node.js    : 6.8.1
  (only if installed via npm)
Installation method : nam
  (npm, zip, tar, pkg, deb, rpm)

Log contents in mb.log when running mb --loglevel debug

Log contents here

maybe something error written in the ejs file

we using "//" as comment token, that make the error.


Implement constant equality as primitive equality

Cf. https://github.com/dart-lang/language/issues/1811.
'Having a primitive operator ==' has now been generalized to 'having primitive equality', cf. https://github.com/dart-lang/sdk/issues/51045. The implementation of this feature is ongoing as of Feb 2023 (and I think it has been completed in the analyzer).
As a further regularization of the treatment of equality in constant expressions, this issue is concerned with the implementation of constant equality as primitive equality.
In particular here is the current specification of constant expressions of the form e1 == e2:
An expression of the form \code{$e_1$\,==\,$e_2$} is potentially constant
if $e_1$ and $e_2$ are both potentially constant expressions.
It is further constant if both $e_1$ and $e_2$ are constant and
either $e_1$ evaluates to an object that is an instance of
\code{int}, \code{double}, \code{String}, \code{bool} or \code{Null},
or if $e_2$ evaluates to the null object (\ref{null}).

It will be modified to rely on primitive equality instead, approximately:
An expression of the form \code{$e_1$\,==\,$e_2$} is potentially constant
if $e_1$ and $e_2$ are both potentially constant expressions.
It is further constant if both $e_1$ and $e_2$ are constant, and
$e_1$ has a primitive equality
(\ref{theOperatorEqualsEquals}),
or $e_2$ evaluates to the null object (\ref{null}).

Note that this change implies that enum values can be used as operands to == in a constant expression, cf. https://github.com/dart-lang/language/issues/312. This is a consequence of the fact that no enum declaration can declare or inherit an implementation of operator == or hashCode that overrides the declaration in Object.

Subtasks:

https://github.com/dart-lang/language/issues/2873
https://github.com/dart-lang/sdk/issues/51567
https://github.com/dart-lang/sdk/issues/51566

double does not have primitive equality, right?
So, now const v = 1.2 == 2.3; is an error?

Right, double does not have primitive equality and 1.2 == 2.3 is not a constant expression.

Oops, that's an unintended breaking change. We'll fix the spec update to avoid the breaking change in dart-lang/language#2874. Thanks for the heads up!

dart-lang/language#2874 has now landed, and the ability to have a double as the left operand of a constant == expression has been preserved as a special exception.

Out of curiosity, would "fixing" it be "that breaking"? For example, would any test fail? It seems small enough that could be fixed.

I'm not entirely sure what that means. ;-)
But if we change the rules such that someConstantVariableOfTypeDouble == 1.5 is suddenly not a constant expression then we might break any number of programs without helping anyone. OK, it's probably only very, very few cases, but it might still break a much larger number of lines of code, because any such expression could occur in a widely used library.
double was never considered to have primitive equality (previously known as "having a primitive operator ==") because equality for double is weird: double.nan != double.nan. So we'd have non-standard behaviors in collections where double.nan is used. E.g., const <double, String>{double.nan: "Not a number"} would return null when we try to look up double.nan.
So we did not want to give double primitive equality, and we did not want to break programs for basically no reason, and that's the reason why we did what we did.
In summary, I'm not 100% sure what 'fixing it' would mean. :smile:


Fix distributor -> ingester load balancing

A few changes

Rename Collector type to IngesterDesc
Make ingester tokens uint32s (so they can be generate by rand)
Make the ingester pick 128 random tokens on startup
Make the ConsulClient watch method take a factor function for making IngesterDescs, to prevent iterations of the loop overwriting previous iterations values (this was causing all ingesters to have the same token).
Add prom metric for % ownership of the ring by each ingester.

Thanks. Just a couple of requests for docs. LGTM otherwise.


Measure/maintain code coverage in CI

Causes CI to fail if our code coverage slips; generates a coverage report with every run.
Currently only checks coverage on ubuntu (the GH action plugin only supports that).  This could certainly be improved by making it work on all OSes and eliminating warnings (and there are other plugin options we can explore) but this will suffice to make sure we don't backslide on testing.  We'll have to update the MINIMUM_COVERAGE as we improve things

Yes, if you look at the run output you'll see that it comes in at ~74.3.  The plugin reports fractional percentage but only accepts integers.


command gosu not found when using elasticsearch 6.0

Hello,
I upgraded elasticsearch from version 5.5.0 to 6.0 using docker image https://hub.docker.com/r/khezen/elasticsearch/tags/ and I get this error in the log:
/run/entrypoint.sh: line 16: gosu: command not found
/run/entrypoint.sh: line 17: gosu: command not found
/run/entrypoint.sh: line 22: gosu: command not found
Stalling for Elasticsearch...

gosu command is used
https://github.com/khezen/docker-elasticsearch/blob/master/src/entrypoint.sh#L16
but seems that it has not installed yet. Please help me to confirm.
thanks a lot

Because of this: https://github.com/khezen/docker-elasticsearch/commit/6198ea5178c339e4f623695329f40391063accc7#diff-3254677a7917c6c01f55212f86c57fbf

thanks for helping!


Install-Package AngleSharp results in Depency Loop Detected error

Bug Report
Thanks for doing so much work on this package, probably something with my environment so detail on that is included below.
Description
Installing on Windows 10 as Administrator, with both PowerShell 7.2 Preview (1 & 2) and Windows PowerShell 5.1 results in an error showing Depency Loop after long delay:
Install-Package: Dependency loop detected for package 'AngleSharp'.
Steps to Reproduce
> Install-Package AngleSharp -Version 0.14.0
Install-Package: Dependency loop detected for package 'AngleSharp'.

# Also tried other combinations, e.g., ...
Install-Package AngleSharp -RequiredVersion 0.14.0 -scope AllUsers -force -Source Nuget.org

Expected behavior: [What you expected to happen]
Successful install of the package.

FAQ has no mention of "Dependency Loop" and references to "install" did not discuss issue.
Change log here stops with version 12.0: https://anglesharp.github.io/log.html

This is the changelog: https://github.com/AngleSharp/AngleSharp/blob/master/CHANGELOG.md
I actually never heard of (or experienced) such an error.
 Maybe its due to a beta of .NET 5?
Posting more details on the install request (e.g., what kind of project was used) would be desired. Can you provide a MWE that I can run to reproduce the issue?

This is the changelog: https://github.com/AngleSharp/AngleSharp/blob/master/CHANGELOG.md
I actually never heard of (or experienced) such an error.
 Maybe its due to a beta of .NET 5?
Posting more details on the install request (e.g., what kind of project was used) would be desired. Can you provide a MWE that I can run to reproduce the issue?

@FlorianRappl
Thanks for responding...
I'll attach the trace.
This is a minimal example since it is the install itself that fails & that is using the command given on the Nuget page for AngleSharp, never got as far as an actual project.
Initially I left out the Verbose/Debug trace since it went on seemingly forever and I didn't know at what point (if any) it started repeating.
There are repeated lines but not a great many.
AngleSharpTrace.txt

@FlorianRappl
Thanks for responding...
I'll attach the trace.
This is a minimal example since it is the install itself that fails & that is using the command given on the Nuget page for AngleSharp, never got as far as an actual project.
Initially I left out the Verbose/Debug trace since it went on seemingly forever and I didn't know at what point (if any) it started repeating.
There are repeated lines but not a great many.
AngleSharpTrace.txt

Not sure what to make of this.

This is a minimal example since it is the install itself that fails & that is using the command given on the Nuget page for AngleSharp, never got as far as an actual project.

I cannot follow here. A minimal example would involve a csproj file to see to what project you want to add the dependency. Not sure where else you'd install it.
The trace looks rather that your system has a lot of issues resolving NuGet dependencies (such as AngleSharp itself, then System.Text.Encoding.CodePages, then ..., finally NETStandard.Library). The last one ultimately fails after exhausting retries.
Right now this does not look like an issue with AngleSharp but rather with the specific setup / .NET. A MWE project to reproduce this would be good.

Not sure what to make of this.

This is a minimal example since it is the install itself that fails & that is using the command given on the Nuget page for AngleSharp, never got as far as an actual project.

I cannot follow here. A minimal example would involve a csproj file to see to what project you want to add the dependency. Not sure where else you'd install it.
The trace looks rather that your system has a lot of issues resolving NuGet dependencies (such as AngleSharp itself, then System.Text.Encoding.CodePages, then ..., finally NETStandard.Library). The last one ultimately fails after exhausting retries.
Right now this does not look like an issue with AngleSharp but rather with the specific setup / .NET. A MWE project to reproduce this would be good.

@FlorianRappl
Seems completely reasonable that this is something local, though I have no idea or experience with fixing Nuget installs.
I had no intention of creating a  CSharp project or similar.
My goal was to install the package and then create .NET objects from it using PowerShell.
Maybe that is not possible, but the first first step would be simply installing the project to the system to make it available.
For now, all I want to do is install the package.  Is this not a reasonable thing to do?

@FlorianRappl
Seems completely reasonable that this is something local, though I have no idea or experience with fixing Nuget installs.
I had no intention of creating a  CSharp project or similar.
My goal was to install the package and then create .NET objects from it using PowerShell.
Maybe that is not possible, but the first first step would be simply installing the project to the system to make it available.
For now, all I want to do is install the package.  Is this not a reasonable thing to do?

For now, all I want to do is install the package. Is this not a reasonable thing to do?

Sure, but I fear that AngleSharp does not officially support PowerShell yet. See #870 for another issue with PowerShell. I would love to support this, but that will only work by external contributions. Any help on the topic appreciated.

For now, all I want to do is install the package. Is this not a reasonable thing to do?

Sure, but I fear that AngleSharp does not officially support PowerShell yet. See #870 for another issue with PowerShell. I would love to support this, but that will only work by external contributions. Any help on the topic appreciated.

@FlorianRappl
Whether I can get it to work from PowerShell is my next problem AFTER it cleanly installs.
There is nothing I am reporting her that is related to PowerShell except the use of the Install-Package command given on the Nuget page.
However, do you have any idea how I can go about verifying or cleaning up my .NET to make the provided command work?
Perhaps you (or someone) could try the command given on the Nuget page with WIndows 10 where AngleSharp has not yet been installed?
Install-Package AngleSharp -Version 0.14.0
I am going to arrange to try it on some other machines, and to do a direct download and see if that can be make the install work.

@FlorianRappl
Whether I can get it to work from PowerShell is my next problem AFTER it cleanly installs.
There is nothing I am reporting her that is related to PowerShell except the use of the Install-Package command given on the Nuget page.
However, do you have any idea how I can go about verifying or cleaning up my .NET to make the provided command work?
Perhaps you (or someone) could try the command given on the Nuget page with WIndows 10 where AngleSharp has not yet been installed?
Install-Package AngleSharp -Version 0.14.0
I am going to arrange to try it on some other machines, and to do a direct download and see if that can be make the install work.

The error is not from AngleSharp, but from System.Text.Encoding.CodePages.
So there is nothing we can do in here - the bug needs to be reported there.
(if you try installing System.Text.Encoding.CodePages you'll hit the dependency loop, which you'll therefore hit on any package depending on System.Text.Encoding.CodePages - this is an official MS dependency)

The error is not from AngleSharp, but from System.Text.Encoding.CodePages.
So there is nothing we can do in here - the bug needs to be reported there.
(if you try installing System.Text.Encoding.CodePages you'll hit the dependency loop, which you'll therefore hit on any package depending on System.Text.Encoding.CodePages - this is an official MS dependency)

@FlorianRappl
Thank you so much for that help.
It's been failing on other machines for other people also, and I've been given another hint that it is some sort of obsolete link in the Nuget URI (perhaps).
I'll work on that and report it at System.Text.Encoding.CodePages if I can't work out the answer.
Thanks again.

@FlorianRappl
Thank you so much for that help.
It's been failing on other machines for other people also, and I've been given another hint that it is some sort of obsolete link in the Nuget URI (perhaps).
I'll work on that and report it at System.Text.Encoding.CodePages if I can't work out the answer.
Thanks again.

As written @HerbM it will fail on all machines. System.Text.Encoding.CodePages is one of the dependencies of AngleSharp, and this one has a problem.
So as long as this is not fixed in System.Text.Encoding.CodePages all dependent packages such as AngleSharp will keep failing, too.

As written @HerbM it will fail on all machines. System.Text.Encoding.CodePages is one of the dependencies of AngleSharp, and this one has a problem.
So as long as this is not fixed in System.Text.Encoding.CodePages all dependent packages such as AngleSharp will keep failing, too.

Found a "workaround" ?
Installed AngleSharp from nuget.exe, then install-package but specifying -SkipDependencies
`

nuget install AngleSharp
Install-Package 'AngleSharp' -Scope 'CurrentUser' -Source 'Nuget.org' -SkipDependencies
`

PSVersion                      7.1.4
PSEdition                      Core
If this helps others.

What areas of the library are affected if the dependencies aren't installed?


Blog post save/update is down, when using multilingual

Laravel Version: 5.4.19
Voyager Version: 0.11.9
PHP Version: 5.6-x64
Database Driver & Version: MySQL 5.7-x64

Description:
Got exception when trying to save blog post:

Argument 2 passed to TCG\Voyager\Models\Post::setAttributeTranslations() must be of the type array, null given, called in E:\Projects\syteg.com\vendor\tcg\voyager\src\Traits\Translatable.php on line 308 and defined

Steps To Reproduce:

Turn on multilingual support in config/voyager.php on line 101
'bread' => false, => 'bread' => true,

Edit/Create any blog post:

http://voyager.dev/admin/posts/2

Press SAVE

Hi @d3n4, can you try the latest version? I've test it and it's working.
Also publish vendor assets, views and the config that was also updated.
I hope this works for you. Good luck!

after update, works fine!

Great! :)


Show correct export # in new subquestion form

Previously, exports in the new subquestion form weren't added to the "available pointers", and hence not properly numbered. This PR addresses this issue.

yay, that's so much better. Nice work.


Added SVE implementation to improve the performance on ARM architecture

Motivation:  This pull request aims to improve the performance of training algorithm of XGBoost on ARM architecture by leveraging SVE intrinsics.
Brief description:

This change of including SVE intrinsics improves the performance by 55% as compared to the ARM default.
The modified function iterates over a row of data and updates a histogram based on the given indices and offsets.
The accuracy has been verified after the modifications.

Thank you for the PR! I'm not an expert in SIMDs, is it guaranteed to have aligned pointers and padded memory allocation for the intrinsics?

Hi @trivialfis
The code has been thoroughly validated to ensure alignment and padding issues are addressed. The datatypes have not been altered from the scalar code; instead, the original scalar operations have been translated into SIMD using equivalent SVE intrinsics and there are no compile-time errors.
All potential accuracy issues have been resolved and verified on widely used datasets like Iris, Airlines delay and breast cancer detection.

Specialized allocators like aligned_alloc() doesn't help with SVE intrinsics because:

ARM's SVE SIMD architecture handles data processing in parallel, which inherently considers data alignment. For example for a 256 bit vector length system, we load 8 float elements (8*32) through VLA(vector length agnostic) instructions into a SVE register.
Most of the instructions including widening and narrowing instructions helps take care of the data alignment.

Hi @trivialfis,
Additionally, SVE also provides predicate registers enabling key features such as:
a) Per-lane predication that allows SIMD instructions to be executed conditionally on specific lanes of a SIMD register
b) Predicate-driven loop control and management that helps to manage data that does not align perfectly with the vector length.

Thank you for the detailed info. Could you please help explain why it works without the use of specialized allocators like https://en.cppreference.com/w/c/memory/aligned_alloc ? It's important for us to know the logic for future maintenance.

Hi @trivialfis,
As @divya2108 mentioned, SVE has predication support.
These lines create masks which limit the load/stores from going out of bounds:
https://github.com/dmlc/xgboost/blob/5194c1726bfa7f5d1b10856d87889f4fc1871638/src/common/hist_util.cc#L265-L266
SVE is also happy to do unaligned loads and stores.

Thank you for the explanation! I will take a deeper look.

Started looking into this PR today. Thank you for working on using the arm intrinsic, but could you please add detailed code comments and extract the code into an independent section (like a function that can be inlined)? Most people here (me included) have a background closer to data science instead of low-level programming.

Hi @trivialfis, Thank you for suggesting the appropriate changes. I have made the modifications as recommended. Could you please review the updated changes?

The CMake logic looks right. It only compiles SVE code when the compiler supports it and during the runtime it triggers the SVE code only when the hardware supports SVE (I see there's a runtime HW check for SVE ISA). Changes LGTM.

The CMake logic looks right. It only compiles SVE code when the compiler supports it and during the runtime it triggers the SVE code only when the hardware supports SVE (I see there's a runtime HW check for SVE ISA). Changes LGTM.

Can you point me to where the runtime check happens? As far as I can tell, this only works if the build environment supports compiling and running SVE.
The new path is conditionally compiled with #ifdef XGBOOST_SVE_SUPPORT_DETECTED with no fallback at runtime here:
https://github.com/dmlc/xgboost/pull/10680/files#diff-def34075edb2b3bdb6dc7b5ebcffd518793520fd4fffd70870b12f076a3cb481R305-R308
This makes me think this is only for users compiling from sources on a specific piece of hardware. If we wanted this to work in the generically distributed wheel, we'd have to do the SVE runtime check instead of the #ifdef.
Correct me if I'm wrong 😸

The CMake logic looks right. It only compiles SVE code when the compiler supports it and during the runtime it triggers the SVE code only when the hardware supports SVE (I see there's a runtime HW check for SVE ISA). Changes LGTM.

Can you point me to where the runtime check happens? As far as I can tell, this only works if the build environment supports compiling and running SVE.
The new path is conditionally compiled with #ifdef XGBOOST_SVE_SUPPORT_DETECTED with no fallback at runtime here: https://github.com/dmlc/xgboost/pull/10680/files#diff-def34075edb2b3bdb6dc7b5ebcffd518793520fd4fffd70870b12f076a3cb481R305-R308
This makes me think this is only for users compiling from sources on a specific piece of hardware. If we wanted this to work in the generically distributed wheel, we'd have to do the SVE runtime check instead of the #ifdef.
Correct me if I'm wrong 😸

I agree with you. I found the HW detection logic here: https://github.com/dmlc/xgboost/pull/10680/files#diff-5650b69c609ef22dea88915eb256a6838341248d3ddfd17430388f7f7e58c4feR24
But this is just for compile time. I think similar logic need to be used during runtime and a runtime check is required.
Since there's already a working SVE HW detection logic, should be easy to reintroduce it in the source code file.
CC @divya2108

Is SVE guaranteed to be available for ARM implementation?

Is SVE guaranteed to be available for ARM implementation?

No, SVE is not guaranteed to be available on all ARM implementations. While ARMv8-A architecture, which includes SVE support, is present in newer processors like Graviton3, Graviton4, Grace, it is not mandatory for all ARM CPUs to implement SVE. The code in hist_util.cc checks for SVE support at runtime to ensure that the target hardware supports it & runs the default code otherwise.

The CMake logic looks right. It only compiles SVE code when the compiler supports it and during the runtime it triggers the SVE code only when the hardware supports SVE (I see there's a runtime HW check for SVE ISA). Changes LGTM.

Can you point me to where the runtime check happens? As far as I can tell, this only works if the build environment supports compiling and running SVE.
The new path is conditionally compiled with #ifdef XGBOOST_SVE_SUPPORT_DETECTED with no fallback at runtime here: https://github.com/dmlc/xgboost/pull/10680/files#diff-def34075edb2b3bdb6dc7b5ebcffd518793520fd4fffd70870b12f076a3cb481R305-R308
This makes me think this is only for users compiling from sources on a specific piece of hardware. If we wanted this to work in the generically distributed wheel, we'd have to do the SVE runtime check instead of the #ifdef.
Correct me if I'm wrong 😸

I agree with you. I found the HW detection logic here: https://github.com/dmlc/xgboost/pull/10680/files#diff-5650b69c609ef22dea88915eb256a6838341248d3ddfd17430388f7f7e58c4feR24
But this is just for compile time. I think similar logic need to be used during runtime and a runtime check is required.
Since there's already a working SVE HW detection logic, should be easy to reintroduce it in the source code file.
CC @divya2108

Yes, I agree. Thank you for bringing this to notice.
I have added a SVE hardware check at runtime. Now it is generically compiled and falls back on the default code if SVE hardware support is not detected.
I have verified this by building the code on different architectures. Here is a summary for more clarity:

@trivialfis Thanks for the initial review and your comments. Can you please suggest any additional feedback which might need further evaluation or clarifications from our side. Thanks. cc: @divya2108

Sorry for the slow reply, got stuck at some other work lately. One question, is it possible to reduce the call frequency of check_sve_hw_support to maybe once per training session?

Sorry for the slow reply, got stuck at some other work lately. One question, is it possible to reduce the call frequency of check_sve_hw_support to maybe once per training session?

Yes, it's possible to reduce the frequency of calls to check_sve_hw_support by implementing a caching mechanism that checks the SVE hardware support status only once at the beginning of a training session. I have stored the result and it is being reused throughout the session.

Hi @trivialfis, just wanted to follow up on the code review. Let me know if you need any additional details or clarifications.

Hi @trivialfis, just wanted to follow up on the code review. Let me know if you need any additional details or clarifications.

Hi @trivialfis , we have pushed all the necessary changes. Kindly review and let us know for any additional details or modifications required. Thanks.

Apologies for the slow response, will look into this. Thank you very much for your patience!

@hcho3 I see you have assigned yourself to the PR. Thank you for volunteering! I recently got access to a Grace machine and might be able to do some tests there.

Sorry it must have been by mistake. I will try to look at the PR for the next few days however.

We should ensure that the CI pipeline tests XGBoost with SVE intrinsic.

Is it enabled on the CI?

Thanks @trivialfis for your review and contributions to this implementation! Please let us if you would like us to contribute any additional fixes based on review comments. We would like to confirm with you before proceeding to avoid any duplicate work. Thanks again for your time in review of this PR. We appreciate! cc: @divya2108

@hcho3 Do you think it's possible to have this in the pip wheel?

Could you please share the CPU you were using for the benchmarks? I ran a benchmark on a Grace machine  (I work for NVIDIA) with synthetic data, and the performance is actually lower.
My synthetic data:

n_samples: 67108864
n_features: 256

Training parameters:

64 iterations
6 max depth

* SVE disabled

[63]    Train-rmse:18.88612
Qdm train (sec) ended in:<PHONE_NUMBER>477539 seconds.
Trained for 64 iterations.
{'load-batches': {'load (sec)': 7.225183725357056}, 'load-all': {'concat (sec)': 1.3589859008789062e-05}, 'Qdm': {'Train-DMatrix (sec)': 29.529566526412964, 'train (sec)':<PHONE_NUMBER>477539}}

* SVE enabled

[63]    Train-rmse:18.88612
Qdm train (sec) ended in:<PHONE_NUMBER>7993164 seconds.
Trained for 64 iterations.
{'load-batches': {'load (sec)': 7.193156003952026}, 'load-all': {'concat (sec)': 1.430511474609375e-05}, 'Qdm': {'Train-DMatrix (sec)': 29.482257604599, 'train (sec)':<PHONE_NUMBER>7993164}}

Do you think it's possible to have this in the pip wheel?

Yes, it should be possible.

Could you please share the CPU you were using for the benchmarks? I ran a benchmark on a Grace machine (I work for NVIDIA) with synthetic data, and the performance is actually lower. I have verified that the row-wise kernel is being used.
My synthetic data:

n_samples: 67108864
n_features: 256

Training parameters:

64 iterations
6 max depth

Compilers:

g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

* SVE disabled

[63]    Train-rmse:18.88612
Qdm train (sec) ended in:<PHONE_NUMBER>477539 seconds.
Trained for 64 iterations.
{'load-batches': {'load (sec)': 7.225183725357056}, 'load-all': {'concat (sec)': 1.3589859008789062e-05}, 'Qdm': {'Train-DMatrix (sec)': 29.529566526412964, 'train (sec)':<PHONE_NUMBER>477539}}

* SVE enabled

[63]    Train-rmse:18.88612
Qdm train (sec) ended in:<PHONE_NUMBER>7993164 seconds.
Trained for 64 iterations.
{'load-batches': {'load (sec)': 7.193156003952026}, 'load-all': {'concat (sec)': 1.430511474609375e-05}, 'Qdm': {'Train-DMatrix (sec)': 29.482257604599, 'train (sec)':<PHONE_NUMBER>7993164}}

It's okay to be slower on certain platforms, we can look for a way to disable it. But I would like to get some understanding of how the performance works for your platform as well.

These are the machine and dataset details which I used:
AWS Graviton3, ARM-based CPU
Dataset details: kaggle higgs boson dataset (250000 samples, 32 features)
Training parameters: 120 iterations, 6 max_depth
compiler: g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

Hi @trivialfis, just wanted to follow up on the PR review. Let me know if there’s anything I can do to help or clarify. Thanks!

apologies for the slow response here. I need to test it on aws, and find a way to disable it for grace.

can you please help to review and try it! Let us know your feedback. Thanks

Could you please elaborate on what I should try?

apologies for the slow response here. I need to test it on aws, and find a way to disable it for grace.

Hi @trivialfis, I have added a change to disable SVE for SVE128 supported hardware (grace) and run the default Neon flow in that.  case. Now the SVE implementation works for ARM CPU's having vector length >=256.
And also seems like the changes you pushed in the past seem to be missing now somehow? I can't locate it here in remote and even locally. Can you help me point to the source of these commits (Can't even find them in your xgboost forked repo), I can cherry-pick it back on top of my branch.

Let me take a look tomorrow. Seems there's a force push that overrode the previous changes.

Will try to redo some of the changes and run test.

Tested again on grace today, small overhead for dispatching, I think I can work around it:
master:

[63]	Train-rmse:18.88612
Qdm train (sec) ended in:  97.99142909049988 seconds.
Trained for 64 iterations.
{'load-batches': {'load (sec)': 7.013079643249512}, 'load-all': {'concat (sec)': 1.7404556274414062e-05}, 'Qdm': {'Train-DMatrix (sec)': 30.664090633392334, 'train (sec)': 97.99142909049988}}

sve:

[63]	Train-rmse:18.88612
Qdm train (sec) ended in:<PHONE_NUMBER>9347534 seconds.
Trained for 64 iterations.
{'load-batches': {'load (sec)': 7.116881370544434}, 'load-all': {'concat (sec)': 1.6450881958007812e-05}, 'Qdm': {'Train-DMatrix (sec)': 30.612082481384277, 'train (sec)':<PHONE_NUMBER>9347534}}

Will test again on AWS with graviton.

I just tested the default build on aws c8g.x16large, the vector_length is 128?

I just tested the default build on aws c8g.x16large, the vector_length is 128 and the extension is not used there?

Hi @trivialfis. As discussed above in my comment, the current logic, checks for SVE VL of the hardware
and if the VL < 256, then it falls back to the default flow (Neon).
So, in the machine you tested earlier (Grace) and the c8g.x16large (Graviton4) machine you tested today,
both have SVE128 in the hardware and as per our current logic, it will fall back to default flow (Neon).
Curious to know, if overhead remained the same even in c8g.x16large or it came down? Did you try any workarounds
around that in the code with the dispatcher?
I believe, if you now run the code on machines with SVE>=256, you would observe the boost in performance because
of SVE ACLE code. One good machine to test SVE goodness would be m7g.8xlarge (Graviton 3) which has SVE256 in the hardware.

@divya2108 Do you expect the extension would provide speedup for graviton 4 if we remove the 256-bit restriction? If so I can run the test again. Otherwise, I don't think we will merge the extension only for graviton3.

@divya2108 Do you expect the extension would provide speedup for graviton 4 if we remove the 256-bit restriction? If so I can run the test again. Otherwise, I don't think we will merge the extension only for graviton3 unless there are other latest and accessible platforms that can benefit from this.

It wouldn't be only Graviton 3, A64FX has 512-bit SVE, and we can incrementally improve this to detect the streaming SVE vector length on devices where that's >128-bit.


Automating Infusing Factory is not working with SFM v9

It keeps putting items in the Extra slot, it does not accept them in the input slot. It will not put redstone, etc. in to the extra slot at all. I have it configured to put the relavant items on the UP and North side in this case.

Note: A Metallurgic Infuser works just fine in it's place.

With me happened the same i upgraded one of my metallurgic infuser and when i put in the place the "infuse slot" changed the name to "extra" and stop enter compressed coal automatic

Mine is the opposite but with Ae2 unable to input into Purple Extra Slot.  Items have to be manually inputted.  But regular Metallurgic Infuser in its spot is fine.

To elaborate, I'm also getting the same issue with Ender IO item conduits, item transfer nodes from OpenBlocks, and even the Logistical Transporters from this mod

I believe I've fixed this, I was not using the same algorithms for item checking as I was in the metallurgic infuser.  v9.0.1 will have the change.

Now after launch version 9.0.1 the infuser are accepting itens


Change subsample's low value from 0 to 0.1 in the catboost example.

The catboost example sometimes fails due to the lack of training samples when it uses subsampling (see https://circleci.com/gh/pfnet/optuna/21659). The actual error message is as follows:
_catboost.CatBoostError: catboost/private/libs/algo/tensor_search_helpers.cpp:441: Too few sampling units (subsample=0.00524573, bootstrap_type=Bernoulli): please increase sampling rate or disable sampling
Exited with code 1

Currently, the minimum size of subsamples is 0 since the low value is 0. This PR updates it to 0.1.  I think it is reasonable because the range of the search space in catboost official benchmark is [0.25, 1]. It is a bit lower but the same order of magnitude.

LGTM.


Fix compilation with DMD 2.072.0

As mentioned in https://github.com/stonemaster/dlang-tour/pull/471#issuecomment-258311882,  D:YAML doesn't build with DMD 2.072.0. I fixed the compilation problems, and opened a PR, but in the meantime, I wanted to fix our CI, so I forked the project and registered it at code.dlang.org.
Finally, this PR makes dlang-tour use the forked package which should fix our CI.

It looks like I need to fix a couple of other issues as well:

[ ] Fix mustache-d compilation with DMD 2.072.0
[ ] Upgrade to vibe.d 0.7.30 to enable building dlang-tour with 2.072.0

Omg - the 2.072 release really broke a lot :/
Thanks a lot for your hard efforts!!

I decided to speed things up and did the fork-and-fix thing with mustache-d as well. I'm not a fan of forking libraries, but for libraries that are not actively developed it may be the only way to ensure quick fixes and overall project stability.

I also fixed a couple deprecation messages caused by DMD 2.071. With all that out of the way, I think this PR is ready.

Thanks a lot again :)


Context menu on the sidebar

What's the Problem?
It is possible to hide sections from the sidebar using the section's contextual menu.
It is not possible to view a section without opening the settings menu.
Solution/Idea
We can add a context menu on the sidebar (there is none yet) to show/hide all sections.
Alternatives
None
Priorities

Capability
Priority

This proposal will allow developers to accomplish W
Must

This proposal will allow end users to accomplish X
Should

This proposal will allow developers to accomplish Y
Could

This proposal will allow end users to accomplish Z
Won't

Files Version
No response
Windows Version
No response
Comments
No response

This issue should be resolved with #8317


Fix authentication bug and progress callback

Sometimes authentication didn't work (WebController said "go back" and didn't display the authentication window), because URL can be not only https://oauth.vk.com/authorize?..., but also https://oauth.vk.com/oauth/authorize?...

Hmm, your request limiter doesn't work properly, I didn't change it, but tests fail with too many requests per second

Also fix progress of uploads: call onProgress callback only when actually uploading file, and not getting upload server or saving the uploaded attachment

@usr-sse2 thanx!


Change persistence API for signs

Sign persistence removed in commit 17c30fce156e40994a513854f733adf56dc3082f
Need to redo sign persistence. Should rely on changing updates rather than resaving all data.

Closed. Persistence not needed.
See commit bc1d68a6549b580cad5af16cdf344d6d9b906c14