Getting invalid repository error.

Try going to https://github.com/settings/applications
and then revoking and  re-adding github classrooms and then retry the assignment link in a new tab.

@hifilorau do you mean erase this one?

Hi @Hvierma20. No, that's your repository. You need to go into your user >Settings> Applications> Search for Github Classroom

Ok, I revoked it,  where can I find it to re add it?

Hi, just start your assignment again
El jue., 4 de junio de 2020 7:27 a. m., Hvierma20<EMAIL_ADDRESS>escribió:

Ok, I revoked it, where can I find it to re add it?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Code-the-Dream-School/CtD-Week-1-Git-Assignment/issues/2#issuecomment-638815679,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOEHLQOXBVCQAI2P5A7W3KDRU6HJVANCNFSM4NRDZFRA
.


First commit

This PR intends to solve the following issues:

Memory leak in DonutProcessor when using large images in numpy format
Use the right settings for beam search size > 1
Solve a bug that in very rare cases made the last element predicted by Chipper to have a bbox = None

Example test snipper. Without the proposed changes it runs out of memory and if there is enough memory, it breaks because an element has no bbox. You need to use the proposed document from sample-docs, please change the location accordingly.
Probably on a Mac with a lot of memory the memory problem might not be a problem. I did not see the memory leak in a mac computer. On linux, with 16gb, it runs out of memory pretty quickly, e.g. on a c6i.2xlarge instance.
from unstructured_inference.inference.layout import DocumentLayout
from unstructured_inference.models.base import get_model

model = get_model("chipper")
doc = DocumentLayout.from_file("sample-docs/patent.pdf", model, pdf_image_dpi=300)


Add uptime to crawl data (RIPD-997)

Reviewers: @HowardHinnant @scottschurr

Scott has more reviews outstanding than anyone else - I could take it over?

@rec, that would be fine with me.

:+1:

LGTM :+1:

:+1:


FileSystemClean Task should be replaced with a job based on what StreamDelete actually deleted.

This should leave FileSystemCleanTask to be a less frequent integrity check type job.

Delivered in 7.0


Update sed command to work with MacOS and Linux

The current sed command only works with the GNU/Linux version of sed. To make it cross platform we make two changes:

Change the in-place argument to use cross platform syntax
Change the extended regex flag to the POSIX compatible flag

The backup file is created at .git/rebase-merge/git-rebase-todo.bak and will automatically be cleaned up by git when that directory is deleted at the end of the rebase. No need for us to handle it specially.


fix(onboarding): fix JSON in IAM Policy instructions

fixes #540
//cc @lcouzens

Codecov Report

Merging #543 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #543   +/-   ##
=======================================
  Coverage   83.48%   83.48%
=======================================
  Files         156      156
  Lines        2355     2355
  Branches      381      381
=======================================
  Hits         1966     1966
  Misses        345      345
  Partials       44       44

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 26910f7...32c01ed. Read the comment docs.

Thanks for the review @dlabrecq


Another question, follow-up to #23

github is not allowing me to add a comment to #23, and I don't see how to label this
as a question, maybe after it's submitted ...
Specifying a version of a package generates key:values in maps/versions.map.jinja, very
nice
How about a %mappings section populated by key=value pairs which generalizes
the notion of defining jinja vars?

@ktenney Seems like it could be very useful. Should be easy to implement too. I am open to it. PR?


Show response body when the response code is unexpected.

This makes it much easier to debug test failures.
Instead of just:
Expected response code '400' to match '201'

It will show the response body as well, so you can see any errors. (I always have to manually add code that prints out the response body so that I can see what happened.)

Any plans to merge this soon? I think this is a super helpful feature, and I would say even essential.

I've been using a fork that has these changes rolled in and they're invaluable for debugging when you have an error.


Maybe add a linter, such as ESLint?

Probably a good idea to pass to JS standard + ES Linter

I'd started with eslint:recommended, @Ivshti want me to do a PR?


ESP32 OTA Example with a DHT11 data sent - fail to finish download

I am planning to write code using Arduino framework to use a ESP32 to send DHT11 sensor data to Thingsboard while checking if there's a new version of OTA.
My thingsboard rule chain has been set and have tested with the 0009-esp8266_esp32_process_OTA_MQTT sample.
I suspect it's either the way I code the dht sensor sending to thingsboard that causes the problem.
Here's the main.cpp
https://gist.github.com/csvke/b649ae904155312580a9d9b9c6ddaf58
#include <Arduino.h>
#include <Update.h>
#include <WiFi.h>
#include <WiFiClientSecure.h>
#include <Arduino_MQTT_Client.h>
#include <ThingsBoard.h>
#include <Espressif_Updater.h>
#include <DHT.h>

constexpr char CURRENT_FIRMWARE_TITLE[] PROGMEM = "TEST";
constexpr char CURRENT_FIRMWARE_VERSION[] PROGMEM = "1.0.3";

// Maximum amount of retries we attempt to download each firmware chunck over MQTT
constexpr uint8_t FIRMWARE_FAILURE_RETRIES PROGMEM = 12U;
constexpr uint16_t FIRMWARE_PACKET_SIZE PROGMEM = 4096U;

constexpr char WIFI_SSID[] PROGMEM = "ssid";
constexpr char WIFI_PASSWORD[] PROGMEM = "password";

constexpr char TOKEN[] PROGMEM = "xa0xh7tej8jk05br4f8i";

constexpr char THINGSBOARD_SERVER[] PROGMEM = "xxx.xxx.xxx.xxx";

constexpr uint16_t THINGSBOARD_PORT PROGMEM = 1883U;

// Maximum size packets will ever be sent or received by the underlying MQTT client,
// if the size is to small messages might not be sent or received messages will be discarded
constexpr uint16_t MAX_MESSAGE_SIZE PROGMEM = 512U;

constexpr uint32_t SERIAL_DEBUG_BAUD PROGMEM = 115200U;

constexpr char FW_STATE_UPDATED[] PROGMEM = "UPDATED";

// Initialize underlying client, used to establish a connection
WiFiClient espClient;
// Initalize the Mqtt client instance
Arduino_MQTT_Client mqttClient(espClient);
// Initialize ThingsBoard instance with the maximum needed buffer size
ThingsBoard tb(mqttClient, MAX_MESSAGE_SIZE);
// Initalize the Updater client instance used to flash binary to flash memory
Espressif_Updater updater;

// Statuses for updating
bool currentFWSent = false;
bool updateRequestSent = false;

// Sensor definitions
#define DHTPIN 13
#define DHTTYPE DHT11

DHT dht(DHTPIN, DHTTYPE);

/// @brief Initalizes WiFi connection,
// will endlessly delay until a connection has been successfully established
void InitWiFi() {
  Serial.println(F("Connecting to AP ..."));
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  while (WiFi.status() != WL_CONNECTED) {
    // Delay 500ms until a connection has been successfully established
    delay(500);
    Serial.print(F("."));
  }
  Serial.println(F("Connected to AP"));
}

/// @brief Reconnects the WiFi uses InitWiFi if the connection has been removed
/// @return Returns true as soon as a connection has been established again
bool reconnect() {
  // Check to ensure we aren't connected yet
  const wl_status_t status = WiFi.status();
  if (status == WL_CONNECTED) {
    return true;
  }

  // If we aren't establish a new connection to the given WiFi network
  InitWiFi();
  return true;
}

/// @brief Updated callback that will be called as soon as the firmware update finishes
/// @param success Either true (update successful) or false (update failed)
void updatedCallback(const bool& success) {
  if (success) {
    Serial.println(F("Done, Reboot now"));
    esp_restart();
    return;
  }
}

/// @brief Progress callback that will be called every time we downloaded a new chunk successfully
/// @param currentChunk
/// @param totalChuncks
void progressCallback(const size_t& currentChunk, const size_t& totalChuncks) {
  Serial.printf("Progress %.2f%%\n", static_cast<float>(currentChunk * 100U) / totalChuncks);
}

void setup() {
  Serial.begin(SERIAL_DEBUG_BAUD);
  dht.begin();
  InitWiFi();
  Serial.print("MAC Address: ");
  Serial.println(WiFi.macAddress());

  // Connect to ThingsBoard
  if (!tb.connect(THINGSBOARD_SERVER, TOKEN)) {
    Serial.println("Failed to connect to ThingsBoard");
  }
}

void loop() {
  delay(1000);

  if (!reconnect()) {
    return;
  }
  if (!tb.connected()) {
    // Reconnect to the ThingsBoard server,
    // if a connection was disrupted or has not yet been established
    Serial.printf("Current firmware version is: (%s)\n", CURRENT_FIRMWARE_VERSION); // q: how to print the CURRENT_FIRMWARE_VERSION
    Serial.printf("Connecting to: (%s) with token (%s)\n", THINGSBOARD_SERVER, TOKEN);
    if (!tb.connect(THINGSBOARD_SERVER, TOKEN, THINGSBOARD_PORT)) {
      Serial.println(F("Failed to connect"));
      return;
    }
  }

  if (!currentFWSent) {
    // Firmware state send at the start of the firmware, to inform the cloud about the current firmware and that it was installed correctly,
    // especially important when using OTA update, because the OTA update sends the last firmware state as UPDATING, meaning the device is restarting
    // if the device restarted correctly and has the new given firmware title and version it should then send thoose to the cloud with the state UPDATED,
    // to inform any end user that the device has successfully restarted and does actually contain the version it was flashed too
    currentFWSent = tb.Firmware_Send_Info(CURRENT_FIRMWARE_TITLE, CURRENT_FIRMWARE_VERSION) && tb.Firmware_Send_State(FW_STATE_UPDATED);
  }

  if (!updateRequestSent) {
    Serial.println(F("Checking for updates..."));
    Serial.println(F("Firwmare Update..."));
    const OTA_Update_Callback callback(&progressCallback, &updatedCallback, CURRENT_FIRMWARE_TITLE, CURRENT_FIRMWARE_VERSION, &updater, FIRMWARE_FAILURE_RETRIES, FIRMWARE_PACKET_SIZE);
    // See https://thingsboard.io/docs/user-guide/ota-updates/
    // to understand how to create a new OTA pacakge and assign it to a device so it can download it.
    updateRequestSent = tb.Start_Firmware_Update(callback);
  }

  // Read DHT11 sensor data
  float humidity = dht.readHumidity();
  float temperature = dht.readTemperature();

  // Check if any reads failed and exit early (to try again).
  if (isnan(humidity) || isnan(temperature)) {
    Serial.println("Failed to read from DHT sensor!");
    return;
  }

// Print data to Serial Monitor
Serial.print("Humidity: ");
Serial.print(humidity);
Serial.print(" %\t");
Serial.print("Temperature: ");
Serial.print(temperature);
Serial.println(" *C");

// Print timestamp
Serial.print("Timestamp: ");
Serial.println(millis());

// Get MAC address
String macAddress = WiFi.macAddress();
Serial.print("MAC Address: ");
Serial.println(macAddress);

// Prepare data for ThingsBoard
tb.sendTelemetryData("temperature", temperature);
tb.sendTelemetryData("humidity", humidity);
tb.sendTelemetryData("macAddress", macAddress.c_str());

tb.loop();
}

Here's the ESP32 serial monitor output
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost

First of all I'm pretty sure this issue occurs with the last chunk that we attempt to download, because the size is smaller than the chunk size of 4096. Meaning it is the last chunk which contains the remaining data.
It is weird tough that the calculation seems to be off, because if the received firmware size shared attribute is correct, then it should be 189 * 4096 and 1 * 2816 bytes big. However in your case it seems like the received firmware size was 189 * 4096 and 1 * 2832.
How do you set the firmware size shared attribute. Are you simply using the Firmware Dashboard and assigning a new firmware or are you chaging the shared attributes by hand?

First of all I'm pretty sure this issue occurs with the last chunk that we attempt to download, because the size is smaller than the chunk size of 4096. Meaning it is the last chunk which contains the remaining data.
It is weird tough that the calculation seems to be off, because if the received firmware size shared attribute is correct, then it should be 189 * 4096 and 1 * 2816 bytes big. However in your case it seems like the received firmware size was 189 * 4096 and 1 * 2832.
How do you set the firmware size shared attribute. Are you simply using the Firmware Dashboard and assigning a new firmware or are you chaging the shared attributes by hand?

I'm new to thingsboard.
I simply used the Firmware Dashboard to assign a new firmware.

Okay weird, if you assign the firmware from the dashboard it should work. The only thing I can currently imagine is that it is sending while receiving the data.
For now at the top of your main.cpp file can you add #define THINGSBOARD_ENABLE_DEBUG 1.
Additionally can you switch the PubSubClient version you are currently using to my fork. Currently both sending and receiving write into the same buffer my fork has seperated that behaviour into 2 buffers instead, which means you should be able to send while receiving data. Perhaps that was the issue.

I have enabled #define THINGSBOARD_ENABLE_DEBUG 1 SO HANDY
And also replaced the PubSubClient to your fork.
Still the same. Let me chop out all the sendTelemetryData and see what I will have.

You should see more log messages if you retry the udpate with the #define THINGSBOARD_ENABLE_DEBUG 1, mode enabled. Can you send thoose additional log messages, because currently I'm not really sure what the issue could be. I've tried the update mechanism a lot especially on ESP32 and have never seen such behaviour before.

More weird is that despite I deleted v1.0.3 of the firmware and explicitly set the assigned firmware in dashboard for the device to use v1.0.2, the device has still detected a new firmware of v1.0.3. Is there some sort of cache behind the scene for firmware versions binaries in thingsboard?
As I have been getting myself familiar with thingsboard I might have created a version of 1.0.3, deleted it, and re-uploaded a different v1.0.3 with a different binary size.

have

[TB] Received data from server over topic (v2/fw/response/0/chunk/188)
[TB] Receive chunk (188), with size (4096) bytes
Progress 98.95%
[TB] Received data from server over topic (v2/fw/response/0/chunk/189)
[TB] Receive chunk (189), with size (4096) bytes
Progress 99.48%
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_error":"Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost","fw_state":"FAILED"})

More weird is that despite I deleted v1.0.3 of the firmware and explicitly set the assigned firmware in dashboard for the device to use v1.0.2, the device has still detected a new firmware of v1.0.3. Is there some sort of cache behind the scene for firmware versions binaries in thingsboard?
As I have been getting myself familiar with thingsboard I might have created a version of 1.0.3, deleted it, and re-uploaded a different v1.0.3 with a different binary size.

There is not really a cache per se, however the fw_size, fw_title, fw_version, ... of the assigned firmware binary are saved into shared attributes. Those shared attributes are then what the device receives and checks. If of course the firmware size of v1.0.2 is received but it attempts to update to v1.0.3 it will fail.

I checked the device's shared attributes and there's no entry. However, there are current_fw_title, current_fe_version, fw_error and fw_state sent from the device to thingsboard.
Is there a detailed up-to-date (thingsboard v3.7) version of an OTA tutorial I can follow?

Okay if the shared attributes do not look like this, that would be very weird. Because the update shouldn't even be able to start if theese keys do not exist on the device.
Can you make sure that in the Shared Attributes Tab of the Device you want to update, if those keys exist and if yes what there values are.

No, it doesn't have any Shared attributes (admittedly there was a device that had shared attributes but after a few attempts of deleting devices and re-doing the Rule chains etc., nothing shows up in shared attributes and latest telemetry anymore, only server attributes)
I have trimmed it down to only running the example 0009 as-is with #define THINGSBOARD_ENABLE_DEBUG 1 on, this is what serial monitor comes out:
Connecting to AP ...
....Connected to AP
Connecting to: (rpi4-workbench-lab-hk.local) with token (p2qknqhdmc7jeg85f7ad)
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"current_fw_title":"ESP32-DHT11","current_fw_version":"1.0.0"})
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_state":"UPDATED"})
Firwmare Update...
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"current_fw_title":"ESP32-DHT11","current_fw_version":"1.0.0"})
[TB] Sending data to server over topic (v1/devices/me/attributes/request/1) with data ({"sharedKeys":"fw_checksum,fw_checksum_algorithm,fw_size,fw_title,fw_version,"})
[TB] Received data from server over topic (v1/devices/me/attributes/response/1)
[TB] Calling subscribed callback for request with response id (1)
[TB] No new firmware assigned on the given device
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_error":"No new firmware assigned on the given device","fw_state":"FAILED"})

Yeah that error atleast makes sense you have no shared attributes assigned that could inform the device that it has a new firmware version to update to so it doesn't.
What exactly have you changed on the rule chain. Because the OTA update should run perfectly without any changes to it, perhaps those are somehow causing problems.
Because normally if you assign the Firmware over the Firmware Dashboard it simply sets the Shared Attribute values as shown above, if that is not the case then something goes wrong in the rule chain probably.

Please confirm the steps for the OTA process:

Setup all the device and get the access token
Setup device profile
Add 'Check is ESP32 firmware synced' rule chains, add 'Check is ESP32 firmware synced' rule chains to Root Rule Chain with the links Attributes Updated and Success
Edit Device Profile's Default rule chain to Check is ESP32 firmware synced
Load the firmware with access token via serial UART to ESP32
Let ESP32 connect to Thingsboard
(Wait until ESP32 sends shared attributes to Thingsboard?)
(Do I need to load v1.0.0 firmware with the exact checksum to Thingsboard?)
Upload firmware 1.0.1 to Thingsboard
Change device profile to assigned firmware v1.0.1
Wait for OTA firmware update??

I'm highly confused by at least half this step, the OTA update works perfectly with the default rule chain, the checking if the firmware is synced is done by this library you don't need the rule chain to do that. (Step 3, 4)
Furthermore I don't know why the ESP32 should send shared attributes that isn't even possible (without workarounds), the device can only ever subscribe or request shared attributes but not create them. You don't need step 8.
Step 9 to 11 are correct, I'm pretty sure these rule chains you added are somehow borking the expected update process of this library. Can you retry simply with the default Rule chain. So just skip step 3, 4, 7 and 8

May I ask where should I put the tb.sendTelemetryData(); inside the void loop ();?

After lots of tweaking, I can only make the OTA process partially work.
What's working is that the firmware is actually pushed to the device, but only if I specifically choose the version of firmware from the Firmware dashboard, assign the firmware in device or device profile will not work.
Also after the update is completed, the device reboot and is with the desired updated version of firmware via OTA (I added a  Serial.printf("Current firmware version is: (%s)\n", CURRENT_FIRMWARE_VERSION); for debugging), however, the device somehow send Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_error":"Firmware version (1.0.2) already up to date","fw_state":"FAILED"}) after the status UPDATED. So the dashboard is displaying the device stuck in Initiated status.
I am so sorry that it's taken so long to implement this feature. But I'm really struggling between reading the documentation, putting the settings into my thingsboard instance and guided (or mis-guided) by a very limited video in Youtube together with a 5-year old official video of the OTA feature.
Progress 99.48%
Current firmware version is: (1.0.1)
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Receive chunk (190), with size (4032) bytes
Progress 100.00%
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_state":"DOWNLOADED"})
[TB] Checksum is the same as expected
[TB] Update success
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_state":"UPDATING"})
Done, Reboot now
ets Jul 29 2019 12:21:46

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:1184
load:0x40078000,len:13232
load:0x40080400,len:3028
entry 0x400805e4
Connecting to AP ...
......Connected to AP
MAC Address: E8:31:CD:32:05:E0
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"current_fw_title":"ESP32-DHT11-SENSOR","current_fw_version":"1.0.2"})
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_state":"UPDATED"})
Checking for updates...
Firwmare Update...
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"current_fw_title":"ESP32-DHT11-SENSOR","current_fw_version":"1.0.2"})
[TB] Sending data to server over topic (v1/devices/me/attributes/request/1) with data ({"sharedKeys":"fw_checksum,fw_checksum_algorithm,fw_size,fw_title,fw_version,"})
Current firmware version is: (1.0.2)
[TB] Received data from server over topic (v1/devices/me/attributes/response/1)
[TB] Calling subscribed callback for request with response id (1)
[TB] Firmware version (1.0.2) already up to date
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_error":"Firmware version (1.0.2) already up to date","fw_state":"FAILED"})

I am so sorry that it's taken so long to implement this feature. But I'm really struggling between reading the documentation, putting the settings into my thingsboard instance and guided (or mis-guided) by a very limited video in Youtube together with a 5-year old official video of the OTA feature.

No problem I hope it hasn't been too much of a hassle.

For the OTA process it is not partially working, OTA is working exactly as expected now. Because if you assign the firmware over the Device Profile or directly to the Device it will not work, because it does not change the Shared Attribute values and therefore does not inform the device correctly that it can update to a new version.
If you assign it over the Dashboard however it does change thoose Shared Attributes so it does not cause any issues.

Somehow send Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_error":"Firmware version (1.0.2) already up to date","fw_state":"FAILED"}) after the status UPDATED

Will be fixed it the newest version, but currently if you call Start_Firmware_Update it checks the received shared attributes, if we even need to update and if we don't it simply sends the failed status with the given error message. With my newest fork this will be resolved as well and send the UPDATED state automatically.

I am so sorry that it's taken so long to implement this feature. But I'm really struggling between reading the documentation, putting the settings into my thingsboard instance and guided (or mis-guided) by a very limited video in Youtube together with a 5-year old official video of the OTA feature.

No problem I hope it hasn't been too much of a hassle.

For the OTA process it is not partially working, OTA is working exactly as expected now. Because if you assign the firmware over the Device Profile or directly to the Device it will not work, because it does not change the Shared Attribute values and therefore does not inform the device correctly that it can update to a new version.
If you assign it over the Dashboard however it does change thoose Shared Attributes so it does not cause any issues.

Somehow send Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_error":"Firmware version (1.0.2) already up to date","fw_state":"FAILED"}) after the status UPDATED

Will be fixed it the newest version, but currently if you call Start_Firmware_Update it checks the received shared attributes, if we even need to update and if we don't it simply sends the failed status with the given error message. With my newest fork this will be resolved as well and send the UPDATED state automatically.

I finally got it working for what I want with 'examples/0011-esp8266_esp32_subscribe_OTA_MQTT/'
My expected behaviour is to have a device loaded with v1.0.0 version of firmware, i upload the firmware v1.0.1 in ThingsBoard OTA, change the device profile assigned firmware, the device detect version increment, the device download and flash the v1.0.1 firmware, the device sent updated state when completed, ThingsBoard OTA dashboard shows updated.
On the side note, what's the approach if I have a bunch of ESP32 grouped in the same device profile that I want to deploy an update to all of them? Because the sample code has the access token hard coded in, do I have to create partitions in the ESP32 and store the credentials in a partition that will not be updated (so I can write firmware code without the access token in it)?
Many many thanks for the help along the way. I feel the whole OTA topic deserves a lot more TLC from an overview write up in the documentation to an updated guide either in video or writing. I can only understand half of what it does after investing 20 hours in it :( but worth the effort

Yeah sorry to hear I hope the OTA process atleast makes more sense now.

On the side note, what's the approach if I have a bunch of ESP32 grouped in the same device profile that I want to deploy an update to all of them? Because the sample code has the access token hard coded in, do I have to create partitions in the ESP32 and store the credentials in a partition that will not be updated (so I can write firmware code without the access token in it)?

The easiest way which I would recommend is to use NVS, it is exactly meant for this use case, being storing small key value pairs in a partition that is safe from updating.
The official documentation from ESP32 on NVS might help.

I feel the whole OTA topic deserves a lot more TLC from an overview write up in the documentation to an updated guide either in video or writing

If you want to understand how it works under the hood on the ESP32 with the partitions and what it writes this documentation from ESP32 might be useful. And for the general process on the Cloud and how it get's notified this page does exaplain it pretty well and is up to date.
I think the main reason for confusion was simply that you used this guide which is pretty old and is not compatible with this library.

@MathewHDYT once again thank you for your help, I have written up a firmware that uses NVS based on the examples/0011-esp8266_esp32_subscribe_OTA_MQTT.
However, I wonder: if assigning a firmware to a particular device profile cannot trigger a batch firmware update (that's what I thought  Device profile > Assigned Firmware is for) and I have to individually choose what device to assign to what firmware in the Firmware Dashboard, what is the best practice to batch update firmware?
Once I iron this issue out, I thought it'd be useful if I release the source code of that, together with another example of self-provisioning, in case someone wants to jump-start and build a close-to-production workflow for a ESP32 sensor data logger with self-provisioning to Thingsboard.
I have also written up a build script and a upload script for the PlatformIO IDE and the code base is under Arduino framework.

What is the best practice to batch update firmware?

There is no premade widget available that fufills that functionality and I think there are 2 major ways to solve it both including custom JavaScript code on the ThingsBoard cloud instance.
Either you detect the change of the assigned firmware in teh device profile on the rule chain and then write the necessary shared attribute values to every device of that type from the assigned firmware.
Or alternatively you make a completly custom widget that allows to select multiple devices or a device profile in a dropdown and select an OTA package and when a submit button is clicked all selected device will have the necessary shared attribute values written,


Update autocomplete settings strings and mark them as untranslated

Task/Issue URL:
Description
Steps to test this PR
Feature 1

[ ]
[ ]

UI changes

Before
After

!(Upload before screenshot)
(Upload after screenshot)

#4684  👈
develop

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @0nko and the rest of your teammates on  Graphite


move to TimeProvider

I haven't yet checked if the produced package is impacted but I am reluctant moving this forward. We want to ensure that the core platform is not having any dependency because this is one of the main issues of VSTest.

@Evangelink i can respect that. will close for now.
although note that Microsoft.Bcl.TimeProvider is only a polyfill package for older runtimes. it is not required in net8 and higher.


Number

r? @jimblandy

Didn't end up using is_empty but I figure we will use it eventually.

Didn't end up using is_empty but I figure we will use it eventually.

Nevermind, I noticed a hole in the tests and fixing it ended up using that method \o/

Coverage increased (+8.5%) to 59.055% when pulling 0b0137269b6dd9972b779ea885fa9b8a479ce12f on number into 4ce77915f100dc82488078dd4d291075c4a4d305 on master.

Coverage increased (+11.8%) to 62.308% when pulling 1af9d48cdc0b12e9e6b58504420c777eadf5bcae on number into 4ce77915f100dc82488078dd4d291075c4a4d305 on master.

Coverage increased (+11.8%) to 62.308% when pulling dad2769858fd36cd910f213fa415c04aaaca6f1d on number into 4ce77915f100dc82488078dd4d291075c4a4d305 on master.


Add caught error to core.setFailed() message

Hi.
With this PR, I added the caught error to the setFailed() message.
This will help user diagnose their problems.
Best regards and thank you in advance.

This would be very helpful, as long as it doesn't contain any sensitive information.

My team and I have waisted two days just because we can't figure out what is wrong when logging in.
The error message says Az CLI Login failed but that is misleading the user into thinking they are using invalid credentials. We definitely need the actual error in the logs, or at least in the debug logs.
Please approve this PR ...

Hi @eiffel-fl , AFAIK we are logging the CLI, Pwsh errors in the logs for the latest versions. Did you come across any errors that are not being logged as part of the current version. Please let me know.
https://github.com/Azure/login/blob/master/src/main.ts#L15-L32

The error message displayed is very generic and is misleading. This PR logs
the real error which is helpful to understand whether the credentials are
incorrect or there is something else going on.
On Wed, Oct 26, 2022 at 7:41 AM eiffel-fl @.***> wrote:

Hi @eiffel-fl https://github.com/eiffel-fl ,
Hi!
AFAIK we are logging the CLI, Pwsh errors in the logs for the latest
versions. Did you come across any errors that are not being logged as part
of the current version? Please let me know.
https://github.com/Azure/login/blob/master/src/main.ts#L15-L32
I do not think the problem this PR solved it about logging but rather what
is logged.
Indeed, in case of caught error, a generic message is printed which gives
no details over the underlying error.
With this PR, I just added the caught error to the logged message, then
giving more hints over the problem.
Best regards.
—
Reply to this email directly, view it on GitHub
https://github.com/Azure/login/pull/243#issuecomment-1291904008, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ACCHL5KBOEEWIWF2JQ5W3SDWFEKHBANCNFSM6AAAAAAQASGWCU
.
You are receiving this because you commented.Message ID:
@.***>

--
Besmir ALIA

Can you also add some screenshots for the changes in logs, something like before after along with the case you are trying. Thanks.

Omg, this feels so bureaucratic. So many hoops to jump for such a simple PR
On Wed, Dec 7, 2022 at 11:24 AM eiffel-fl @.***> wrote:

No, this PR is not inactive.
@BALAGA-GAYATRI https://github.com/BALAGA-GAYATRI I added the
screenshot you asked to see the differences this contribution brings, so is
it possible to get some feedback? This would be appreciated.
—
Reply to this email directly, view it on GitHub
https://github.com/Azure/login/pull/243#issuecomment-1341231775, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ACCHL5KIJOFHYAYAX54FB5DWMC23TANCNFSM6AAAAAAQASGWCU
.
You are receiving this because you commented.Message ID:
@.***>

--
Besmir ALIA

Hi, merged the changes to the master. Please raise a PR with these changes against releases/v1 branch from which the latest release will be created and changes would go live. Thanks!


-nan CHARR contamination values

When running sce-vcf for one sample I get:
[12:06:24-884] - INFO: SCE-VCF v0.1.2
[12:06:24-884] - ARG: Input VCF: @["sample.sortdup.bqsr.hc.vcf.gz"]
[12:06:24-884] - ARG: Output: stdout
[12:06:24-884] - ARG: AD field: AD
[12:06:24-884] - ARG: AF field: AF
[12:06:24-884] - ARG: Is refAF: false
[12:06:24-884] - ARG: REF AF limits: 0.1,0.9
[12:06:24-884] - ARG: Het AB limit: 0.25,0.75
[12:06:24-884] - ARG: Min GQ: 20
[12:06:24-884] - ARG: DP limit: 20,100
#SAMPLE HQ_HOM  HQ_HOM_RATE     HQ_HET  HQ_HET_RATE     CHARR   MEAN_REF_AB_HOM_ALT     HETEROZYGOSITY_RATE     INCONSISTENT_AB_HET_RATE
[12:06:24-907] - INFO: Variant processing started
...
[12:07:00-100] - INFO: 6160954 variants processed, 1960156 vars ignored: 87866 multiallelic, 989349 indels, 882941 outside ref AF limits, 0 missing AF tag
[12:07:00-101] - INFO: Computing contamination values
PGP0622076      0       -nan    3712593 0.99375 -nan    -nan    1.00000 0.30501
[12:07:00-101] - INFO: Computed contamination values for 1 sample(s) in 33 seconds and 974 milliseconds
I tried normalizing it with "bcftools view -f PASS,. -m2 -M2 -v snps" without much success:
[12:14:15-051] - INFO: 5033767 variants processed, 882941 vars ignored: 0 multiallelic, 0 indels, 882941 outside ref AF limits, 0 missing AF tag
[12:14:15-052] - INFO: Computing contamination values
PGP0622076      0       -nan    3712593 0.99375 -nan    -nan    1.00000 0.30501
[12:14:15-052] - INFO: Computed contamination values for 1 sample(s) in 22 seconds and 540 milliseconds
Anyway to debug this error? There should be a --debug function. :D
What else can I do to help get this fixed?
Best Regards,

Hi!
Apologies for being unresponsive. I was swamped with new stuff, but I have some time now trying to fix these bugs.
If I understand correctly, you get a result file like this at the end of the run.
#SAMPLE HQ_HOM HQ_HOM_RATE HQ_HET HQ_HET_RATE CHARR MEAN_REF_AB_HOM_ALT HETEROZYGOSITY_RATE INCONSISTENT_AB_HET_RATE
PGP0622076 0 -nan 3712593 0.99375 -nan -nan 1.00000 0.30501

In this case, it seems that the tool cannot find any high-quality ALT homozygous site in your VCF. To determine HQ homozygous genotypes, the tool first skips indels and variants outside configured AF limits and then considers the configured GQ and DP threshold for variants within the.
Here, you have HQ_HOM sites == 0. Thus, it may be that in your VCF file, there are no ALT homozygous genotypes satisfying all the criteria.
Can you check if there is any genotype passing the HQ criteria?
You can get examples of HQ variants using bcftools.
bcftools query -i 'GT="AA" & FMT/DP >= 20 & FMT/DP <= 100 & GQ >= 20 & TYPE="snp" & AF <= 0.9 & AF >= 0.1' -f '[%CHROM %POS %REF %ALT %SAMPLE %GT %GQ %DP\n]' input.vcf.gz

While I perform a final test, you can try the new executable I posted in #1.
If no HQ data is available after filtering, your sample will still show' nan' values, but you should see a better log message warning you about this.
Let me know how it works for you.

I assume this is solved.


badge description with test case specific library path  variable

extend the badge description to allow the combination of test cases from different test suites, by extending the requirement list by an additional test library variable.

This has been made redundant by using interoperability requirements in badges instead of testcases.


New Socks

@czechboy0 can we get this bumped to Socks 0.8.0?

I've updated it here. Had to bump the version number a couple times from playing around with it, but I can submit a PR later.

Thanks!


FAQCs Error with unequal R1 and R2 files

Describe the bug
If the number of R1 and R2 files are not equal in terms of number, FAQC will not generate the output files. This has been observed with SRA datasets downloaded from NCBI.
If the bug is verified, a possible solution would be to filter the input files to remove reads with no corresponding pair.

A step has been added to the bwa-pre-process workflow to filter unpaired reads from FASTQ files.


Support Unit Tests for TOS

What changes are proposed in this pull request?
I have created some unit tests for TOS.
Why are the changes needed?
1、Because it can help us check our code.
Does this PR introduce any user facing changes?
No.

alluxio-bot, merge this please

alluxio-bot, merge this please


R19: Home Page: The Software Home Page banner shall display average recalls per day over year-to-date along with the caption 'Average Recalls per day over calendar year'

Derived from FDA metadata. Tools: JQuery

metadata:  recall_initiation_date,status, completed,pending, on-going, country

Jquery ajax request to openFDA, visualized in d3 used to satisfy this requirement

Completed.
SQA


SameSite for System.Web

Fixes https://github.com/aspnet/AspNetCore.Docs/issues/16026
Internal review URL

@StephenMolloy


Filter out monospaced font in font selection view

It is hard to see all monospaced font at a glance. It would be great to toggle between all fonts and monospaced fonts.

I guess some people would like a toggle button, especially for those who edit non-code contents frequently.


"data:," in address bar and tests don´t run - Chrome

I just started trying Protractor and can't get it running.
I followed the tutorial on protractortest.org
First I launched the selenium server with: webdriver-manager start.
Then I used "protractor protractor.conf.js", which results in a new Chrome being opened with "data:," in the address bar. Nothing else happens after that.
In the selenium terminal window I see stuff like:
java.util.concurrent.ExecutionException: org.openqa.selenium.WebDriverException: java.lang.reflect.InvocationTargetException
All help appreciated.

Please read the contributing guidelines; these types of questions are ill-suited for GitHub. Thanks!


Use msgraphgocore http client to download onedrive file

We're currently using the default http client which should be configured with the correct timeout/retry handlers. The msgraphgocore default client seems suitable here.

Am I missing something or is https://github.com/alcionai/corso/pull/1484 the only change needed for this(apart from maybe generating the url when we start the download)?

On a related note, we are already getting the URL for downloading at the time of actual fetch. Ref: https://github.com/alcionai/corso/blob/274fba8708085f4ef29c9db4f7144b7e9d2204e8/src/internal/connector/onedrive/item.go#L37

On a related note...

Graph Download URL expiry - should/can we defer creation of the upload/download URL to when we start the operation

We are already creating the download URL at the time of fetch. https://github.com/alcionai/corso/blob/274fba8708085f4ef29c9db4f7144b7e9d2204e8/src/internal/connector/onedrive/item.go#L37

We are already creating the download URL at the time of fetch.

So the code in item.go initializes the reader (and the download URL) but it is only returned and used via the data collection to kopia here.
My concern is that for a large hierarchy - there can be a delay. I'm not sure yet how long these URLs are valid.

We can change it to return just the URL and change ToReader to create a ReadCloser from the URL. But that will require a project wide change to the type itemReaderFunc.
https://github.com/alcionai/corso/blob/990e5514d22d590aa8a70e6d484730a7fac0b18f/src/internal/connector/onedrive/collection.go#L44-L49
We can have it be a URL, but that would break the nice abstraction that we have as a ReadCloser. We will also have to update what Item will look like.
https://github.com/alcionai/corso/blob/990e5514d22d590aa8a70e6d484730a7fac0b18f/src/internal/connector/onedrive/collection.go#L88-L92
As for the expiration, it looks like the default url is valid for ~1hour as per SO and docs. One solution mentioned in SO was to generate a share url, but I don't think that is a good solution, nor can we generalize it.
If it is OK, I can make the change to return a URL from driveItemReader, but just wanted to double check on the implementation as we will be changing widely used types.
/cc @ryanfkeepers


Ability to only restart one matrix job

When viewing a specific matrix build job and hitting REBUILD, the build is completely restarted. It would be nice if the restart button on a specific matrix job view would only restart this particular matrix job, not all of them.

unfortunately we are moving in the opposite direction. Builds will be immutable and restarting a build will result in a new build number. It will therefore not be possible to restart a single matrix item for an existing pipeline.
Reference issue https://github.com/drone/drone/issues/2035


Upgrade to Rouge 2.0

This PR upgrades to Rouge 2.0 which was released on June 14th.
See https://github.com/jneen/rouge#formatters for more information.

Neat. How's the compatibility of this vs. the old one?

I really need these upgrades

javascript - lack of ES6 string support
javascript - switch case statements marked as errors: https://github.com/jneen/rouge/issues/497

@bhollis I believe the only difference is that due to https://github.com/jneen/rouge/commit/f287925ad4ab0ca161d46d6df2ec3c0954772c3e, the CSS class names gutter, table and code are now prefixed with rouge- by default.
Plus, lots of bugfixes.

Released middleman-syntax 3.0.0


⚠ Bofrost.at has degraded performance

In 87bccb0, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1786 ms

Resolved: Bofrost.at performance has improved in 9050c80 after 6 minutes.


Critical error with ObjectDB

The last commit b3a6cc097d5bc60ee6e7ba30b177bed3a8aadc71 compiles properly on Xubuntu 14.10 64 but the binary crashes immediately with the following error:

ERROR: cleanup: ObjectDB Instances still exist!
At: core/object.cpp:1715.

That's weird, that commint has nothing to do with the Linux build, what was the last commit that did work ok?
Sent from my iPad

On Feb 12, 2015, at 2:50 AM, gokudomatic<EMAIL_ADDRESS>wrote:
The last commit b3a6cc0 compiles properly on Xubuntu 14.10 64 but the binary crashes immediately with the following error:
ERROR: cleanup: ObjectDB Instances still exist!
At: core/object.cpp:1715.
—
Reply to this email directly or view it on GitHub.

that error needs to be fixed, but if it appears, it's simply that the
application quit
On Thu, Feb 12, 2015 at 7:16 AM, rraallvv<EMAIL_ADDRESS>wrote:

That's weird, that commint has nothing to do with the Linux build, what
was the last commit that did work ok?
Sent from my iPad

On Feb 12, 2015, at 2:50 AM, gokudomatic<EMAIL_ADDRESS>wrote:
The last commit b3a6cc0 compiles properly on Xubuntu 14.10 64 but the
binary crashes immediately with the following error:
ERROR: cleanup: ObjectDB Instances still exist!
At: core/object.cpp:1715.
—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
https://github.com/okamstudio/godot/issues/1360#issuecomment-74047271.

I cleaned, removed all files in bin and recompiled. Now it's ok. I'm not sure if I did a mistake, but the error doesn't come anymore.


Update metadata and required fields for EC7

Update metadata (row 6 - List Values) for columns:

S (Project Demographic Distribution - Primary Populations Served)
U (Project Demographic Distribution - Secondary Populations Served)
W (Project Demographic Distribution - Tertiary Populations Served)

The following fields are no longer needed for EC 7.1 - 7.2:

S (Project Demographic Distribution - Primary Populations Served)
T (Primary Project Demographic Explanation)
U (Project Demographic Distribution - Secondary Populations Served)
V (Secondary Project Demographic Explanation)
W (Project Demographic Distribution - Tertiary Populations Served)
X (Tertiary Project Demographic Explanation)

The following updates were made to v20220815
EC 7
- Updated metadata for demographics (cols S, U, W)
- These fields are no longer required for cols S – X
Logic
- Visibility for 7.1-7.2 have had the visibility for columns S – X removed


feat(style): component popover

Closing because tokens need a re-specification and the styles need a re-work.


TASK: Node v10 compatibility

Use correct read method depending of used node version.

Fixed! Thank you!


:bug: Update jira instance connection column to use popover

When a jira instance on the main jira configuration table has a connectivity issue, the connection column will show "Not connected". Clicking the text will now open a popover to display additional details about the state of the instance.  The popover replaces a standard modal box with similar information.
Fixes: #855
What it looked like before:

What it looks like now:

@sjd78, that sounds good, although during a previous conversation with @rromannissen the idea is to display errors from the Jira in code block (same for the tickets errors).
Be aware the message can't be several lines long too.
So this needs to be validated by UXD.
Thansk

Looks good, please use I18n for text as in initial code ?

I swapped the "Connection" text back to t("terms.instance") for the popover title.  I'll leave the description in the body above the code block since the popover design guidelines always include a header and descriptive text.  Having just the header with the code block text looks a bit awkward.
(rebased also)


How to get observations from states?

According to my testing thus far, it seems that not all games have the same method of accessing player observations, and there are four separate ways to get the current game state:
state.observation_state_string(Optional[player])
state.observation_state_tensor(Optional[player])
state.information_state_string(Optional[player])
state.information_state_tensor(Optional[player])

Without having to do a bunch of try/excepts, is there a place that documents which games have which methods implemented, or is there a possibility that I'm not seeing a section of documentation that pertains to this?
Just some code example of what I'm doing to find out what's implemented:
import pyspiel
import numpy as np

game = pyspiel.load_game("tarok")
state = game.new_initial_state()
while not state.is_terminal():

    while state.is_chance_node():
        outcomes_with_probs = state.chance_outcomes()
        action_list, prob_list = zip(*outcomes_with_probs)
        action = np.random.choice(action_list, p=prob_list)
        state.apply_action(action)

    legal_actions = state.legal_actions()
    action = legal_actions[0]
    state.apply_action(action)

    try:
        try:
            try:
                print(state.observation_string())
            except:
                print(state.observation_string(state.current_player()))
        except:
            try:
                print(state.observation_tensor())
            except:
                print(state.observation_tensor(state.current_player()))
    except:
        try:
            try:
                print(state.information_state_tensor())
            except:
                print(state.information_state_tensor(state.current_player()))
        except:
            try:
                raise Exception
                print(state.information_state_string())
            except:
                print(state.information_state_string(state.current_player()))

Hi @jjshoots,
Easiest thing is to just check the game type. Each game type object has two fields provides_information_state_tensor and provides_observation_tensor.
Here's an example of how the RL environment does it:
https://github.com/deepmind/open_spiel/blob/97a199a3276ef35b932f39dc01786855aa36545f/open_spiel/python/rl_environment.py#L198
This is not a general rule, but roughly true: all perfect information games will have observation defined, all imperfect info games will have information state defined, but some have both defined (some have none defined, but they are rare).

Ah, gotcha, thank you!

Reopening this to hitchhike with an adjacent question, what about string-based observation/information states, is there a flag to check for that? AFAICT, not all games have tensor based observations and only have string-based, e.g. battleship.

Yes there are similar properties for the string versions, see the GameType struct here:  https://github.com/deepmind/open_spiel/blob/97a199a3276ef35b932f39dc01786855aa36545f/open_spiel/spiel.h#L120
The strings are used mostly for tabular methods, e.g. as unique identifiers for keys that index into a table, or for optimization methods (e.g. CFR, sequence-form linear programming).
Which ones are included per game depends on who implemented them and how they've being used. Battleship was a contribution and the contributor didn't implement the tensor functions because they did not use the game for RL. We recently added the infoset tensor function to it (because it's an imperfect information game), but not the observation one.


[Feature][core] Implement more type conversions for IcebergConversions

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation
In #3731 we've introduced IcebergCommitCallback to create Iceberg metadata after commit. We convert Paimon objects to Iceberg byte buffers in IcebergConversions, however it currently only supports a few types and does not support types like timestamp and time.
Solution
Support timestamp and time conversions in IcebergConversions.
Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

I'm willing to submit a PR!

Do this according to the following specifications?
time | Stores microseconds from midnight in an 8-byte little-endian long
timestamp | Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long
then,like this:
case TIME:
TIMESTAMP:
return ByteBuffer.allocate(8)
.order(ByteOrder.LITTLE_ENDIAN)
.putLong(0, (long) value);

I'm willing to submit a PR!

Thanks for your interest. Please note that there are two types of timestamp in Paimon: TimestampType and LocalZonedTimestampType. TimestampType is timestamptz in Iceberg, while LocalZonedTimestampType is timestamp in Iceberg. You also need to be careful with the precesion of timestamp, because Iceberg has timestamp and timestamp_ns.
All in all you need to write tests carefully, especially for timestamp.

I want to be simple
Whether there are similar code references？
@tsreaper

I don't know that Object value  is  string ? or Timestamp  ?
mabye  Timestamp  ?  then  change to  buffer?

The Paimon object for TimestampType and LocalZonedTimestampType is org.apache.paimon.data.Timestamp. It stores millisecond and nanoOfMillisecond in the object.
For TimestampType, this millisecond is the time starting from 1970-01-01 00:00:00 UTC, so it is timestamptz in Iceberg. For LocalZonedTimestampType, this millisecond is the time starting from 1970-01-01 00:00:00 local timezone, so it is timestamp in Iceberg.
Also you need to check if precision <= 6, if yes it is timestamp or timestamptz; Otherwise if precision > 6 its timestamp_ns or timestamptz_ns.
Don't forget to add complete tests for each timestamp type and different precisions.

It seems a bit complicated
icerberg does not have local-timestamp-micros ,but paimon has local-timestamp-micros,
Maybe I need to learn more about paimon

"local-timestamp-micros" is not a data type in Paimon. It is a data type in avro. Iceberg's avro reader has to deal with it. However for this issue, you only need to convert between Paimon types and Iceberg types, so you don't need to consider about avro types.

I am using the test case IcebergCompatibilityTest, whose underlying file format is avro.
Do I need to write my own test cases?
Which underlying file storage format should I use?

I found this problem. Only the partition information is used.
The test case only needs to ensure the partition ？

I have completed it, but nanoseconds cannot be tested, because the avro format in the test case cannot support precisions greater than 6
I don't know if my code idea is correct. Please review the code

@tsreaper Can I submit PR?

@tsreaper Can I submit PR?

Sure, feel free to submit.

@tsreaper    Please help to review my pr, thank you very much

@tsreaper
next step?
With that done, I'm ready to continue contributing


Added blueprints, minor changes in README.md

The blueprints are work in progress. I want to unify them.
In README.md there were still two occurrences of sensor.smart_irrigation_daily_adjusted_run_time

ehhmmm it looks like you're not up to date. you need to first catch up as we already merged this pr and then create a new pr with your minor changes and blueprints.


Using FASTA_GXF_BUSCO_PLOT  sub workflow

Hello Team!
I built a reusable sub workflow for performing BUSCO on genome alone, or on genome and annotation. I have used it in two other pipelines. It is currently sitting in my own modules repo:  https://github.com/GallVp/nxf-components/blob/7188d37139f8dbdfd73cbfb5b0e3f811d1993392/subworkflows/gallvp/fasta_gxf_busco_plot/main.nf#L7
I think we should be sharing this across pipelines. I can submit it to nf-core/modules and add it here. Checking if there is interest here and if there is need to modify it to suite the needs of this pipeline?

Mentions, @FernandoDuarteF , @chriswyatt1, @stephenturner
nf-core issue: https://github.com/nf-core/modules/issues/5584
Other pipelines using it:

https://github.com/Plant-Food-Research-Open/assemblyqc/blob/cec2728d0785d97e0e9493558c83d83d0b48240e/workflows/assemblyqc.nf#L442
https://github.com/Plant-Food-Research-Open/genepal/blob/24a8f50e241105b98c88035fd9c2a0cad8ff728f/workflows/genepal.nf#L206

That looks really useful, will check it out next week.

I think we should definitively incorporate this to our pipeline.
I was also thinking we should make genome.nf and genome_and_annotation.nf into workflows, as right now they are subworkflows.

We should also have a look at the BUSCO subworkflow from the MAG pipeline. See #85.

We should also have a look at the BUSCO subworkflow from the MAG pipeline. See #85.

My implementation does have a val_busco_lineages_path input. If this is provided, busco will download the data there once and will reuse it. If all the data is already downloaded, then busco will use that instead of redownloading it.
For cloud environments, this parameter needs to be optimised so that a tar ball can be passed similar to what mag is doing.

Hi @GallVp, I would like to incorporate your BUSCO subworkflow into our pipeline, at least for now. We can update it later once it's made into an nf-core subworkflow.
I'm not sure what's the etiquette to follow here, especially when it comes to giving you proper credit.

Thank you @FernandoDuarteF
It will be good to share the workflows across pipelines and maintain them jointly. I am happy to create a PR and that will put me as a contributor on the repo. That's enough credit for me. How about we do the following,

I submit the sub-workflow to nf-core/modules and invite your review. Hopefully, it will be accepted now that there are two pipelines using it. If not, we will have to maintain it where it is, i.e. in my modules repo.
I create a PR to genomeqc for adding the sub-workflow and again invite your review.

Thank you @GallVp! Awesome, let's do that then.


Ensure binary() for kinit input args

@ElMaxo merge ?

Ready

I have done the tests yesterday.
All good (except that it's not quite easy to get kerberos working).
awesome work @ElMaxo

@zmstone Thanks for kind words. To be honest, i don't like kerberos, all dirty work with kerberos setup at cluster was made by our dev ops. Also, they use Apache Ambari, as far as i know it can do it process simplier

Apache Ambrai.
I'll check that out.
thanks for the info.


[Bug]: Intermittent test failure

Description:
The following test case fails intermittently in the build actions (Noticed this intermittent failure when building locally as well)  :
[fail] testWellknownCertBackendWithinFunction:
      Something wrong with the connection

Duplicate of https://github.com/ballerina-platform/ballerina-standard-library/issues/4222


Create Sudoku class.

the program loops 3 times and solves behaves perfectly by finding unique solutions and replacing the solutions.  then just stops... tested by iterating 4 , 5 times no changes...


Use scalafmt to test formatting after running tests

remove scala reform, add scalafmt not automatic, but as part of the CI.
The first PR should fail before I run fmt, then we will see if it succeeds.

I'm just going to merge this if it is green. It is only formatting changes.

Codecov Report

:exclamation: No coverage uploaded for pull request base (develop@dc17788). Click here to learn what that means.
The diff coverage is 75.07%.

@@            Coverage Diff             @@
##             develop     #656   +/-   ##
==========================================
  Coverage           ?   89.48%
==========================================
  Files              ?      113
  Lines              ?     8960
  Branches           ?      487
==========================================
  Hits               ?     8018
  Misses             ?      942
  Partials           ?        0

Impacted Files
Coverage Δ

...scala/com/twitter/algebird/BufferedOperation.scala
93.75% <ø> (ø)

...la/com/twitter/algebird/scalacheck/Arbitrary.scala
100% <ø> (ø)

.../scala/com/twitter/algebird/MetricProperties.scala
0% <ø> (ø)

...src/main/scala/com/twitter/algebird/Identity.scala
66.66% <ø> (ø)

...scala/com/twitter/algebird/scalacheck/PosNum.scala
100% <ø> (ø)

...om/twitter/algebird/GeneratedAbstractAlgebra.scala
98.29% <ø> (ø)

.../src/main/scala/com/twitter/algebird/package.scala
0% <ø> (ø)

...com/twitter/algebird/GeneratedProductAlgebra.scala
97.43% <ø> (ø)

...main/scala/com/twitter/algebird/SummingQueue.scala
100% <ø> (ø)

...twitter/algebird/scalacheck/ExpHistInstances.scala
100% <ø> (ø)

... and 101 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dc17788...05855ce. Read the comment docs.


Question about the training of MAGVIT-v2

Hi, authors! Thanks for your great work!
Here I would like to ask for some details of the pretraining of MAGVIT-v2. I see the codes include DinoV2(https://github.com/showlab/Show-o/blob/53cbe82bcda44f40c9e77c01c7f25fe75c055b66/models/common_modules.py#L360). I wonder if you choose it for the discriminator. Could you provide more details on the tokenizer training process?
Thank you in advance!

Hi, Robert, we didn't use the DINO features. We do not have a plan to release the training code of MAGVIT-v2 due to internal issues. Alternatively, you can take a look at this repo. All of us follow the training details provided in the MAGVIT-v2 paper. :)

Thanks for your quick reply. Could I know how you justify whether to finish the training or not?

Hi, reconstruction FID on ImageNet can help. Or empirically train it for 1000k/2000k iterations.

Thanks for your reply!


Use spf13/cobra for docker update

This fix is part of the effort to convert commands to spf13/cobra #23211.
Thif fix coverted command docker update to use spf13/cobra
Signed-off-by: Yong Tang<EMAIL_ADDRESS>

LGTM

sorry @yongtang, this one needs a rebase as well :cry:

Thanks @thaJeztah. The PR has been rebased.

What the?? On gccgo:
06:25:33 FAIL: docker_cli_help_test.go:15: TestHelpTextVerify.pN52_github_com_docker_docker_integration_cli.DockerSuite
06:25:33
06:25:33 docker_cli_help_test.go:143:
06:25:33     c.Fatal(err)
06:25:33 ... Error: Error on "logout" help. non-empty stderr:"Aborted\n\npanic during panic\n\nruntime stack:\nstack trace unavailable\n"

Thanks @icecrime for the review. I am hoping the failure is a one-time failure but I will keep an eye on it in case it is repeatable.

ping @vdemeester PTAL

Sorry @yongtang, this needs a rebase now :cry:

Thanks @thaJeztah the pull request has been rebased.

Ping @vdemeester @thaJeztah.

@yongtang Sorry for delay! Let me know if you want us to carry PR.

@LK4D4 Thanks for the review. I updated the pull request to change the blkioWeight in opts. Please let me know if there are any issues.

Thanks @LK4D4 The pull request has been updated. Please let me know if there are any other issues.

@yongtang perfect! I'll merge if ci pass.

LGTM 🐮 assuming it's green 😇


Can this library support flash attention?

As flash attention shows, I guess this library cannot support flash attention usage, since mx_mapping.inject_pyt_ops mainly affect the pytorch ops or module. Please correct me if I am wrong.

You're correct. The library won't quantize the flash-attention ops.


Bug: Help Command Doesn't Recognize Input Commands with Different Letter Case

#585 the chain cog shares this issue since the code was copy-pasted from the help command code.

This is an easy fix overall, it does need to possibly take into account the ignore_case option on the command but thats not the most pressing issue

would def accept a pr for this

Would it be better to just implement that or just force everything into lowercase for it to be read?


Adds instructions for Roboto

Resolves #132

Hey @iRoachie, I've just found out you give there the same link to both where's the font located and how add font to project - there is this one https://fonts.google.com/specimen/Roboto. Could you make a new pull request with correct second link?

Oops Lol sorry about that


Created and updated docs for 1.20-17

Created and updated docs for 1.20-17
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve


jsonpb: mapped enum cannot marshal/unmarshal string

enum Numeral {
  UNKNOWN = 0;
  ARABIC = 1;
  ROMAN = 2;
}
message Mappy {
  map<string, Numeral> enumy = 6;
}

The enumy field in Mappy message is always marshaled as int, and cannot be unmarshaled from string  enum like {"enumy": {"foo": "ARABIC", "bar": "ROMAN"}}
This is probably because generated message field does not have enum property and not handled as Enum in jsonpb.marshalValue.
The test case is skipped for now.
https://github.com/golang/protobuf/blob/master/jsonpb/jsonpb_test.go#L388-L389

Same issue here.


Issue with OpsWorks, Chef Zero, Jenkins Cookbook

Hey guys!
First off, great cookbook!
We're running into a funny error when attempting to push this out to OpsWorks stacks:
Errno::EEXIST
File exists - (../../jenkins_credentials_create/serverspec/assert_created_spec.rb, /opt/aws/opsworks/current/merged-cookbooks/./jenkins/test/integration/smoke_war_stable/serverspec/assert_credentials_created_spec.rb)
This affects both Amazon linux and Ubuntu instances. We're only calling the "master" and "java" recipes via "jenkins::master" and "jenkins::java", and have pointed directly to this repo.

Console log:
[2015-06-09T21:57:53+00:00] INFO: Running queued delayed notifications before re-raising exception
[2015-06-09T21:57:53+00:00] ERROR: Running exception handlers
[2015-06-09T21:57:53+00:00] ERROR: Exception handlers complete
[2015-06-09T21:57:53+00:00] FATAL: Stacktrace dumped to /var/lib/aws/opsworks/cache.stage1/chef-stacktrace.out
[2015-06-09T21:57:53+00:00] ERROR: ruby_block[merge all cookbooks sources] (opsworks_custom_cookbooks::load line 12) had an error: Errno::EEXIST: File exists - (../../jenkins_credentials_create/serverspec/assert_created_spec.rb, /opt/aws/opsworks/current/merged-cookbooks/./jenkins/test/integration/smoke_war_stable/serverspec/assert_credentials_created_spec.rb)
[2015-06-09T21:57:53+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)


error downloading the template

Describe the bug
There was an error downloading the template from URI 'https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/Playbooks/Get-SOCActions/azuredeploy.json'. Ensure that the template is publicly accessible and that the publisher has enabled CORS policy on the endpoint. To deploy this template, download the template manually and paste the contents in the 'Build your own template in the editor' option below.
To Reproduce
Steps to reproduce the behavior:

Go to https://github.com/rinure-msft/Azure-Sentinel/tree/master/Playbooks/Get-SOCActions
click deploy to azure

Expected behavior
the template gets added

Hi, this is a version created by Rinure-msft, I can see that there isn't a deployment file yet, unfortunately you'll have to wait until it's created and pushed to master or alternatively follow this guide: https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/quickstart-create-templates-use-the-portal#edit-and-deploy-the-template
and paste the contents from https://github.com/rinure-msft/Azure-Sentinel/blob/master/Playbooks/Get-SOCActions/Get-SOCActions.json by step 4 instead of making the changes

@DeanGross - Thanks, This issue has been fixed. Please let us know, if you see any other issues


should nproc-per-node equals to sequence_parallel_size?

Great work. I have a question, should nproc-per-node equals to sequence_parallel_size when do open-sora inference using DSP?
Thanks in advance.

sp size should be as small as possible

@oahzxl why sp size should be as small as possible?

because the essense of sequence parallel is to accomodate long sequence at the cost of extra communication cost. so if memory is enough, keep it low


Cleanup silly warnings.

Description
This PR cleans up some annoying compiler warnings during build.

Checks and build are passing: https://github.com/ko4life-net/ko/actions/runs/3222884524/jobs/5272404398
Merging.


Websocket connection terminates after ~40 seconds

I have a spring cloud gateway on top of a python-based service. The service is not registered to any registration discovery. Following is the configuration of my spring cloud gateway.
spring:
  cloud:
    gateway:
      routes:
      - id: service_websocket
        uri: wss://serviceed.test.domain.com:443
        predicates:
        - Path=/service/**
        filters:
        - StripPrefix=1

I'm able to create a WebSocket connection. However, these connections are being terminated abruptly after ~40s.  Following are the logs of spring gateway.
2021-07-15 20:35:58.215 DEBUG 1363 --- [ctor-http-nio-4] o.s.w.r.s.a.ReactorNettyWebSocketSession : [19be1ed2] Session id "19be1ed2" for wss://serviceed.test.domain.com:443/endpoint
2021-07-15 20:35:58.215 DEBUG 1363 --- [ctor-http-nio-4] o.s.w.r.s.c.ReactorNettyWebSocketClient  : Started session '19be1ed2' for wss://serviceed.test.domain.com:443/endpoint
2021-07-15 20:35:58.215 DEBUG 1363 --- [ctor-http-nio-4] reactor.netty.ReactorNetty               : [id: 0x8bdedad8, L:/<IP_ADDRESS>:8080 - R:/<IP_ADDRESS>:55594] Added decoder [WebSocketFrameAggregator] at the end of the user pipeline, full pipeline: [wsdecoder, wsencoder, WebSocketFrameAggregator, reactor.right.reactiveBridge, DefaultChannelPipeline$TailContext#0]
2021-07-15 20:35:58.215 DEBUG 1363 --- [ctor-http-nio-4] r.n.channel.ChannelOperationsHandler     : [id: 0x5fd37a51, L:/<IP_ADDRESS>:55595 - R:serviceed.test.domain.com/<IP_ADDRESS>:443] New sending options
2021-07-15 20:35:58.215 DEBUG 1363 --- [ctor-http-nio-4] reactor.netty.ReactorNetty               : [id: 0x5fd37a51, L:/<IP_ADDRESS>:55595 - R:serviceed.test.domain.com/<IP_ADDRESS>:443] Added decoder [WebSocketFrameAggregator] at the end of the user pipeline, full pipeline: [reactor.left.sslHandler, reactor.left.httpCodec, ws-decoder, ws-encoder, WebSocketFrameAggregator, reactor.right.reactiveBridge, DefaultChannelPipeline$TailContext#0]
2021-07-15 20:35:58.215 DEBUG 1363 --- [ctor-http-nio-4] r.n.channel.ChannelOperationsHandler     : [id: 0x8bdedad8, L:/<IP_ADDRESS>:8080 - R:/<IP_ADDRESS>:55594] New sending options
2021-07-15 20:35:58.216 DEBUG 1363 --- [ctor-http-nio-4] r.n.channel.ChannelOperationsHandler     : [id: 0x5fd37a51, L:/<IP_ADDRESS>:55595 - R:serviceed.test.domain.com/<IP_ADDRESS>:443] Writing object
2021-07-15 20:35:58.216 DEBUG 1363 --- [ctor-http-nio-4] reactor.netty.channel.FluxReceive        : [id: 0x8bdedad8, L:/<IP_ADDRESS>:8080 - R:/<IP_ADDRESS>:55594] Subscribing inbound receiver [pending: 0, cancelled:false, inboundDone: false]
2021-07-15 20:35:58.216 DEBUG 1363 --- [ctor-http-nio-4] r.n.channel.ChannelOperationsHandler     : [id: 0x8bdedad8, L:/<IP_ADDRESS>:8080 - R:/<IP_ADDRESS>:55594] Writing object
2021-07-15 20:35:58.216 DEBUG 1363 --- [ctor-http-nio-4] reactor.netty.channel.FluxReceive        : [id: 0x5fd37a51, L:/<IP_ADDRESS>:55595 - R:serviceed.test.domain.com/<IP_ADDRESS>:443] Subscribing inbound receiver [pending: 0, cancelled:false, inboundDone: false]
2021-07-15 20:35:58.216 DEBUG 1363 --- [ctor-http-nio-4] i.n.h.c.h.w.WebSocket08FrameDecoder      : Decoding WebSocket Frame opCode=1
2021-07-15 20:35:58.216 DEBUG 1363 --- [ctor-http-nio-4] i.n.h.c.h.w.WebSocket08FrameDecoder      : Decoding WebSocket Frame length=19
2021-07-15 20:35:58.216 DEBUG 1363 --- [ctor-http-nio-4] i.n.h.c.h.w.WebSocket08FrameEncoder      : Encoding WebSocket Frame opCode=1 length=19
2021-07-15 20:35:59.376 DEBUG 1363 --- [ctor-http-nio-4] i.n.h.c.h.w.WebSocket08FrameDecoder      : Decoding WebSocket Frame opCode=1
2021-07-15 20:36:37.853 DEBUG 1363 --- [ctor-http-nio-4] i.n.h.c.h.w.WebSocket08FrameEncoder      : Encoding WebSocket Frame opCode=1 length=10996
2021-07-15 20:36:38.158 DEBUG 1363 --- [ctor-http-nio-4] i.n.h.c.h.w.WebSocket08FrameDecoder      : Decoding WebSocket Frame opCode=1
2021-07-15 20:36:38.158 DEBUG 1363 --- [ctor-http-nio-4] i.n.h.c.h.w.WebSocket08FrameDecoder      : Decoding WebSocket Frame length=145
2021-07-15 20:36:38.158 DEBUG 1363 --- [ctor-http-nio-4] i.n.h.c.h.w.WebSocket08FrameEncoder      : Encoding WebSocket Frame opCode=1 length=145
2021-07-15 20:36:38.219 DEBUG 1363 --- [ctor-http-nio-4] i.n.h.c.h.w.WebSocket08FrameDecoder      : Decoding WebSocket Frame opCode=8
2021-07-15 20:36:38.219 DEBUG 1363 --- [ctor-http-nio-4] i.n.h.c.h.w.WebSocket08FrameDecoder      : Decoding WebSocket Frame length=2
<Skipping few encoding and decoding recurring logs..................................>
2021-07-152021-07-15 20:36:38.219 DEBUG 1363 --- [ctor-http-nio-4] i.n.h.c.h.w.WebSocket08FrameDecoder      :
Decoding WebSocket Frame length=4
2021-07-15 20:36:18.216 DEBUG 1363 --- [ctor-http-nio-4] r.n.channel.ChannelOperationsHandler     : [id: 0x5fd37a51, L:/<IP_ADDRESS>:55595 - R:serviceed.test.domain.com/<IP_ADDRESS>:443] Writing object PongWebSocketFrame(data: PooledUnsafeDirectByteBuf(ridx: 0, widx: 4, cap: 4))
2021-07-15 20:36:38.220 DEBUG 1363 --- [ctor-http-nio-4] r.n.http.client.HttpClientOperations     : [id: 0x5fd37a51, L:/<IP_ADDRESS>:55595 - R:serviceed.test.domain.com/<IP_ADDRESS>:443] CloseWebSocketFrame detected. Closing Websocket
2021-07-15 20:36:38.220 DEBUG 1363 --- [ctor-http-nio-4] r.n.channel.ChannelOperationsHandler     : [id: 0x8bdedad8, L:/<IP_ADDRESS>:8080 - R:/<IP_ADDRESS>:55594] Writing object CloseWebSocketFrame(data: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf(ridx: 0, widx: 0, cap: 0))
2021-07-15 20:36:38.220 DEBUG 1363 --- [ctor-http-nio-4] i.n.h.c.h.w.WebSocket08FrameEncoder      : Encoding WebSocket Frame opCode=8 length=0
2021-07-15 20:36:38.220 DEBUG 1363 --- [ctor-http-nio-4] r.n.channel.ChannelOperationsHandler     : [id: 0x5fd37a51, L:/<IP_ADDRESS>:55595 - R:serviceed.test.domain.com/<IP_ADDRESS>:443] Writing object CloseWebSocketFrame(data: SimpleLeakAwareByteBuf(PooledUnsafeDirectByteBuf(ridx: 0, widx: 2, cap: 2)))
2021-07-15 20:36:38.220 DEBUG 1363 --- [ctor-http-nio-4] i.n.h.c.h.w.WebSocket08FrameEncoder      : Encoding WebSocket Frame opCode=10 length=4
2021-07-15 20:36:38.220 DEBUG 1363 --- [ctor-http-nio-4] i.n.h.c.h.w.WebSocket08FrameEncoder      : Encoding WebSocket Frame opCode=8 length=2
2021-07-15 20:36:38.221 DEBUG 1363 --- [ctor-http-nio-4] r.n.resources.PooledConnectionProvider   : [id: 0x5fd37a51, L:/<IP_ADDRESS>:55595 ! R:serviceed.test.domain.com/<IP_ADDRESS>:443] Channel cleaned, now 1 active connections and 1 inactive connections
2021-07-15 20:36:38.221 DEBUG 1363 --- [ctor-http-nio-4] r.n.resources.PooledConnectionProvider   : [id: 0x5fd37a51, L:/<IP_ADDRESS>:55595 ! R:serviceed.test.domain.com/<IP_ADDRESS>:443] Channel closed, now 1 active connections and 0 inactive connections
2021-07-15 20:36:38.221 DEBUG 1363 --- [ctor-http-nio-4] reactor.netty.ReactorNetty               : [id: 0x5fd37a51, L:/<IP_ADDRESS>:55595 ! R:serviceed.test.domain.com/<IP_ADDRESS>:443] Non Removed handler: reactor.left.httpAggregator, context: null, pipeline: DefaultChannelPipeline{(reactor.left.sslHandler = io.netty.handler.ssl.SslHandler), (ws-decoder = io.netty.handler.codec.http.websocketx.WebSocket13FrameDecoder), (ws-encoder = io.netty.handler.codec.http.websocketx.WebSocket13FrameEncoder), (WebSocketFrameAggregator = io.netty.handler.codec.http.websocketx.WebSocketFrameAggregator), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2021-07-15 20:36:38.222 DEBUG 1363 --- [ctor-http-nio-4] r.n.resources.PooledConnectionProvider   : [id: 0x5fd37a51, L:/<IP_ADDRESS>:55595 ! R:serviceed.test.domain.com/<IP_ADDRESS>:443] onStateChange(ws{uri=/endpoint, connection=PooledConnection{channel=[id: 0x5fd37a51, L:/<IP_ADDRESS>:55595 ! R:serviceed.test.domain.com/<IP_ADDRESS>:443]}}, [disconnecting])
2021-07-15 20:36:38.222 DEBUG 1363 --- [ctor-http-nio-4] r.n.http.client.HttpClientOperations     : [id: 0x5fd37a51, L:/<IP_ADDRESS>:55595 ! R:serviceed.test.domain.com/<IP_ADDRESS>:443] Cancelling Websocket inbound. Closing Websocket

When I connect to the service without a spring gateway, connections are working as expected. Am I missing any configuration ?
Spring Cloud: Greenwich.SR1
Spring boot: 2.1.4.RELEASE
Thanks,
Vijeth

By the way, the gateway version I use is 2.2.0.RELEASE.
I have the spring cloud gateway on top of Jupyter notebook, which is a python-based service. when I connect to jupyter without gateway, it‘s’ working well. That problem took me an afternoon to look into, but I still got nothing.
I guess maybe something prevented the websocket ping frame, but no evidence.

greenwich is no longer supported, can you try with 2020.0.4?


payment-v2 endpoint test hard to PR from my fork - please replace app.test.js with this code to bee-travels/bee-travels-node MASTER br

PR from bee-travels/bee-travels-node to my fork had merge conflicts I just could not resolve at this early hour :)
app.test.js.zip

Adding it to my fork


More data from Rockfish cruise dataset

@bbest Do you want me to help out with the data and graphs for the other pelagic indicators? I'm getting data from http://oceanview.pfeg.noaa.gov/erddap
If it is not what you had in mind we can change it.
Jennifer Brown's list of organisms of interest to the sanctuary includes:
copepods
krill
squid
gelatinous zooplankton
anchovy
sardine
mackerel
salmon
rockfish
hake
cormorants
common murre
auklets
pinnipeds
turtles

Hi @reikopm,
Yes, that'd be great. Here's more background from Jarrod:
Santora et al. Appendix 1.docx
Attached is the taxa appendix table and % occurrence for pulling down and interpreting the rockfish data from ERDDAP:
For YOY groundfish and rockfish, group all "YOY."
For Forage, group all non "YOY"
For ALL, group ALL taxa in the table
See the metadata file for additional taxa identification consistency through time and Region (C= core, etc...).


fix(ComponentGallery): import module defaults

Related Ticket(s)
#1338
related https://github.com/carbon-design-system/carbon-for-ibm-dotcom-website/pull/1226
Description
This PR updates the ComponentGallery image imports so that thumbnails load correctly again. The breakage likely comes from webpack changes related to updating Gatsby, as they are showing up as modules rather than the SVG strings

Thank you @emyarod !


Code appears before plots when using figure layouts

In format: html, the following document will first show the echoed code, then a two column layout with a total of three plots and a data frame. In format: closeread-html, the echoed code shows up after the two column layout.
---
title: "Orphaned code output test"
format: closeread-html
---

:::{.cr-layout}

Test test test

:::{.cr-crossfade cr-to="big-code"}
Test test test
:::

Test test test

```{r}
#| echo: true
#| cr-id: big-code
#| layout-ncol: 2

a <- rnorm(100)
b <- 1 + rexp(100)
c <- a + b

a <- rnorm(100)
b <- 1 + rexp(100)
c <- a + b

a <- rnorm(100)
b <- 1 + rexp(100)
c <- a + b

plot(mtcars$mpg, mtcars$dist)

plot(mtcars$hp, mtcars$dist)

plot(mtcars$hp, mtcars$mpg)

mtcars
\```

:::

Initial diagnosis
When our lua filter encounters the layout div, in the AST the code cell looks like:
Div {.cell cr-id="big-code" layout-ncol="2"}
Blocks:

CodeBlock {.r .cell-code}
Div {.cell-output-display}
Blocks:

Image

CodeBlock {.r .cell-code}
Div {.cell-output-display}
Blocks:

Image

CodeBlock {.r .cell-code}
Div {.cell-output-display}
Blocks:

Image

CodeBlock {.r .cell-code}
Div {.cell-output-display}
Blocks:

CodeBlock

That is, it separates the code into different CodeBlocks based on the lines that produce output, then interleaves those CodeBlocks with output (3 images and a CodeBlock to print the dataframe).
At the end of our lua filter, the whole DivAbove is wrapped into two more Divs:
Div {.sticky-col}
Blocks:

Div {.sticky-col-stack}
Blocks:
DivAbove

That looks about right! The fact that in the flat html format the CodeBlock is not separated and interleaved into the output suggests the filter for layout-ncol runs after our filter does to rearrange those blocks. That rearrangement must be breaking out Div wrapper structure.
Next steps: write a separate lua filter that runs late to see if we can catch the AST post-layout filter.

@jimjam-slam Oy, this was a deep cut:
https://github.com/quarto-dev/quarto-cli/discussions/10174

Dang, nice work on the diagnosis!

Circling back to this and updating the test case following the shift to Quarto cross-ref style trigger references:
---
title: "Orphaned code output test"
format:
  closeread-html:
    theme: cosmo
---

:::{.cr-section}

Test test test

Test test trigger big code @cr-big-code

Test test test

:::{#cr-big-code}
```{r}
#| echo: true
#| layout-ncol: 2

a <- rnorm(100)
b <- 1 + rexp(100)
c <- a + b

a <- rnorm(100)
b <- 1 + rexp(100)
c <- a + b

a <- rnorm(100)
b <- 1 + rexp(100)
c <- a + b

plot(mtcars$mpg, mtcars$dist)
plot(mtcars$hp, mtcars$dist)
plot(mtcars$hp, mtcars$mpg)
mtcars

```
:::

:::

All done!

On Safari:

So there's still a layout problem even though we encourage folks to wrap the entire code cell in a fenced div (rather than adding id: cr-... directly to the code cell.


Implement len and is_empty for EventReaders

Objective
Provide a non-consuming method of checking if there are events in an EventReader.
Fixes #2967
Solution
Implements the len and is_empty functions for EventReader and ManualEventReader, giving users the ability to check for the presence of new events without consuming any.

bors r+

bors r+


Possibile permission error when installing on macOS (14.5) via Homebrew

What system are you running Yazi on?
macOS
What terminal are you running Yazi in?
iTerm2
Did you try the latest code to see if this problem got fixed?
Tried, but the problem still
yazi --debug output

Backtrace omitted. Run with RUST_BACKTRACE=1 to display it.
Run with RUST_BACKTRACE=full to include source snippets.

The application panicked (crashed).
  initializing rolling file appender failed: InitError { context: "failed to create initial log file", source: Os { code: 13, kind: PermissionDenied, message: "Permission denied" } }
in /Users/{myUser}/Library/Caches/Homebrew/cargo_cache/registry/src/index.crates.io-6f17d22bba15001f/tracing-appender-0.2.3/src/rolling.rs, line 154
thread: main

Describe the bug
Just wanted to try out Yazi and installed it via homebrew on macOS Sonoma (14.5) (tried both stable 0.2.5 and HEAD versions, same error), but upon launching it with yazi a permission error is returned. Looks like incorrect/missing folder permissions on where it's going to save logs in this case.
Running with sudo works fine, but I suppose it should do it without it.
Expected Behavior
Yazi starting
To Reproduce

Install via homebrew on macOS Sonoma (14.5)
run yazi in terminal

Configuration
None, didn't add any custom config
Anything else?
No response

Yazi needs to create log files in the ~/.local/state/yazi directory. Does it have write permissions? And what does echo $XDG_STATE_HOME say?

Thanks for you time, that was it actually!
For anyone who might encounter the same issue:
So, $XDG_STATE_HOME wasn't set to anything (I think macOS does not fill these out by default). I set them up in zsh so that now XDG_STATE_HOME is ~/.local/state.
I then fixed permissions for the folder and files and everything is now working.
Closing the issue


Distribute Inspector Application as a separate npm package

Release Jenkins builds are not yet updated
Webkit commit SHA would be updated later

run ci

run ci

run ci

run ci


New <script setup> and ref sugar implementation

Related RFC: https://github.com/vuejs/rfcs/pull/222

@knightly build this

@knightly-bot build this

How Can I use this new syntax in a new vue project?

Build with Vite is not working (Dev OK). Repro: https://github.com/antfu/vite-starter-ref-sugar
<script setup>
import { computed } from 'vue'

ref: count = 0
ref: doubled = computed(() => count * 2)

const inc = () => count++
const dec = () => count -= 1
</script>

<template>
  <div>{{ count }} * 2 = {{ doubled }}</div>
  <button @click="dec">Dec</button>
  <button @click="inc">Inc</button>
</template>

Dev

Build

Error: ReferenceError: count is not defined

After reading https://github.com/vuejs/rfcs/blob/script-setup-2/active-rfcs/0000-script-setup.md,
I have some questions,

Using this


Package import specifier "#internal/nitro/virtual/app-config" is not defined

Environment
npx nuxi info
Working directory: /Users/nico/sources/empty-app                                                                                11:10:13 AM
Nuxt project info: (copied to clipboard)                                                                                11:10:13 AM

Operating System: Darwin
Node Version:     v22.3.0
Nuxt Version:     3.12.3
CLI Version:      3.12.0
Nitro Version:    2.9.7
Package Manager<EMAIL_ADDRESS>Builder:          -
User Config:      compatibilityDate, devtools, modules
Runtime Modules<EMAIL_ADDRESS>Build Modules:    -

Reproduction
I started from an empty app, then add a single nuxt module.
This module has been spotted in another context as being the source of the exact same issue. The link below is a reproduction context, with both dev and production run logs. It also provides the .output I got after using npx nuxi build.
https://github.com/kogratte/nuxt-icons-issue
Describe the bug
Working on another project, I have an issue Package import specifier "#internal/nitro/virtual/app-config" is not defined in package /Users/nico/sources/empty-app/node_modules/nitropack/package.json imported from /Users/nico/sources/empty-app/node_modules/nitropack/dist/runtime/config.mjs.
As we were not aware of the origin, we tried to deep dive to finaly use a "remove and try" method. We finally spotted that the simple presence of this module (nuxt/icons) in our reusable layer what at the origin of the issue. As soon as we remove it, things goes well.
We've been wondering if it could be due to a bad combination, and created the linked repository.
npx nuxi dev => Ends with error.
npx nuxi build => OK.
node .output/server/index.mj => Same error as nuxi dev.
Additional context
No response
Logs
See .log files in the linked repository.

Got the same issue here

Coincidence or not, but when I updated Nuxt to 3.12.4, everything built without errors.

I had the same problem. After deleting .nuxt and node_modules and reinstalling node_modules, it was gone

Hey. I am experiencing this issue. When developing a module, if I add this module to the playground, the playground will not start.
Here is a minimal reproduction link: https://stackblitz.com/edit/nuxt-starter-xqmvk1?file=playground/nuxt.config.ts
If you remove the comment in front of this module in the nuxt.config file within the playground folder, you will get this error. I get this error when adding the content module to playground as well.
Is there any workaround for this at the moment?

Also got into this error just by using npx nuxi@latest init -t ui followed by npm run dev

Locking the nitropack version to 2.9.6 (from 2.9.7) fixed the error for me.
I did this using the override-section in package.json:
  "overrides": {
    "nitropack": "2.9.6"
  }

Similar issues haven been seen at other modules:

https://github.com/nuxt/content/issues/2736
https://github.com/getsentry/sentry-javascript/pull/13183

@Anton-Plagemann thank you for the patch. I tried it in the reproduction i posted above your comment and it worked!
I added the override to the playground's package.json file. Appreciate it
Next is to try it locally and then try deploying to Netlify

@pi0 Seems like the release of nitro 2.9.7 did break something, especially for clean setups (e.g. npx nuxi@latest init -t ui or npx nuxi@latest init docs -t content) or module authors (see my comment). It would be highly appreciated if you could take a look at this! 🙏

@Anton-Plagemann for local dev, i had to lock nitropack at v2.9.5 for things to work

@Anton-Plagemann I've got it working with nitro 2.9.7

hi folks, same issue, I think this is related:
https://github.com/directus-labs/agency-os/issues/109

Hi folks, something that fixed here is stop using pnpm. If I run pnpm install and later on I run the server I got this error, if I just run npm install and then npm run dev it runs smootly. But it needs to be ran in a new repo in which pnpm was never user.

I try to install nuxt/icon in this project rather than only install in the reusable layer,  and then it works.

Same issue - I haven't managed to fix...

Here too.

Here too.

I tried
"overrides": {
    "nitropack": "2.9.6"
  }

and also directly npm install<EMAIL_ADDRESS>but neither worked...

Steps to reproduce:

Clone https://github.com/JhumanJ/opnform
cd client
npm install
npm run build: should be successful
node .output/server/index.mjs

Which gives this:
node .output/server/index.mjs
node:internal/errors:496
    ErrorCaptureStackTrace(err);
    ^

TypeError [ERR_PACKAGE_IMPORT_NOT_DEFINED]: Package import specifier "#internal/nitro/virtual/app-config" is not defined in package /Users/jhumanj/Documents/coding/laravel-projects/notiontools/client/.output/server/node_modules/nitropack/package.json imported from /Users/jhumanj/Documents/coding/laravel-projects/notiontools/client/.output/server/node_modules/nitropack/dist/runtime/config.mjs
    at new NodeError (node:internal/errors:405:5)
    at importNotDefined (node:internal/modules/esm/resolve:353:10)
    at packageImportsResolve (node:internal/modules/esm/resolve:803:9)
    at moduleResolve (node:internal/modules/esm/resolve:967:16)
    at defaultResolve (node:internal/modules/esm/resolve:1193:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:403:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:372:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:249:38)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:76:39)
    at link (node:internal/modules/esm/module_job:75:36) {
  code: 'ERR_PACKAGE_IMPORT_NOT_DEFINED'
}

Node.js v18.19.0

What I tried:

setting nitropack override version
installing nitropack dependency
Updating versions of all depencies (nuxt, vue etc)
Different versions of node

I'm not sure if this is related, but at some point while trying to update all dependencies, i obtained the following error running the built server:
node .output/server/index.mjs
Listening on http://[::]:3000
[nuxt] [request error] [unhandled] [500] Cannot find module '/Users/jhumanj/Documents/coding/laravel-projects/OpnForm/client/.output/server/node_modules/vue/server-renderer/index.mjs' imported from /Users/jhumanj/Documents/coding/laravel-projects/OpnForm/client/.output/server/chunks/routes/renderer.mjs
  at new NodeError (node:internal/errors:405:5)
  at finalizeResolution (node:internal/modules/esm/resolve:327:11)
  at moduleResolve (node:internal/modules/esm/resolve:980:10)
  at defaultResolve (node:internal/modules/esm/resolve:1193:11)
  at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:403:12)
  at ModuleLoader.resolve (node:internal/modules/esm/loader:372:25)
  at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:249:38)
  at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:76:39)
  at link (node:internal/modules/esm/module_job:75:36)

No real fix for this as yet? The overrides trick isnt working anymore

I tried to investigate but haven't had a clue yet. My assumption is that it might be caused by some transitive dependencies updates and introduced breakage that neither Nitro nor Nuxt Icon. But nothing confirmed yet.

Thanks @antfu .
Have you experienced the problem locally?
I can create meaningful docs for even a brand new module. The playground won't start if either the content module/icon module is in it.

If you can make a more minimal reproduction i would be happy to investigate it. It is most likely a module tries to import from #internal/nitro (or worst #internal/nitro/*) but they should instead import from nitropack/runtime or better #imports.

Hey @pi0 , I do have a minimal repo setup here: https://stackblitz.com/edit/nuxt-starter-xqmvk1?file=playground/nuxt.config.ts

Thanks for the reproductions.
~> https://github.com/nuxt/icon/pull/249

Thanks for the fix.
Now to make a report in the content module 🤣

@antfu What is the root cause? In my case, I can't avoid it with #imports .
import { useRequestBody } from "nitropack/runtime/utils";

It's not work:
import { useRequestBody } from "#imports";

useRequestBody  is just an example, in fact, I still need to call other functions.

I just came back to one of my projects after 2 weeks and I now get this out of nowhere.
[17:48:32]  ERROR  [worker reload] [worker init] Package import specifier "#internal/nitro/virtual/app-config" is not defined in package D:\xxx\node_modules\nitropack\package.json imported from D:\xxx\node_modules\nitropack\dist\runtime\config.mjs

  at __node_internal_captureLargerStackTrace (node:internal/errors:496:5)
  at new NodeError (node:internal/errors:405:5)
  at importNotDefined (node:internal/modules/esm/resolve:353:10)
  at packageImportsResolve (node:internal/modules/esm/resolve:803:9)
  at moduleResolve (node:internal/modules/esm/resolve:967:16)
  at defaultResolve (node:internal/modules/esm/resolve:1193:11)
  at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:403:12)
  at ModuleLoader.resolve (node:internal/modules/esm/loader:372:25)
  at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:249:38)
  at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:76:39)
  at link (node:internal/modules/esm/module_job:75:36)


fix(masthead): prevent megamenus from overlaying global banner

Note: This PR currently relies on changes from https://github.com/carbon-design-system/carbon-for-ibm-dotcom/pull/11773, which is why there are extra commits. We'll want to hold off on merging this until that PR has been merged.
Related Ticket(s)
Closes #11775
Description
Currently, <c4d-megamenu> are slotted into a <ul> that is position: fixed;.  This is what allows them to extend the full width of the viewport on desktop, but it also means they cannot align themselves relative to other elements using CSS alone. At this stage in the jenga tower, there's not a great CSS-only fix for our issue.
So we turn to our good friend JavaScript for assistance. <c4d-top-nav-menu> items now report the position of their bottom edge via a --c4d-masthead-top-nav-menu-bottom-edge CSS custom property when toggled open. <c4d-megamenu> elements reference this prop to position themselves in the exact right place when they get loaded into the DOM.
Changelog
New

Makes <c4d-top-nav-menu> items expose the position of their bottom edge via a --c4d-masthead-top-nav-menu-bottom-edge CSS custom property.

Changed

Updates <c4d-megamenu> CSS to reference --c4d-masthead-top-nav-menu-bottom-edge value to position themselves correctly when they load into the DOM.

Closing in favor of #11837


Scenario 2 - Home Page

Change the Home Page according to the Functional Specification with Learn , About QIPR Video, Login,Register,Search Bar and featured Projects. The nav bar is subject to change.

This is outdated, we will create a new issue should it be needed.


https://kit.svelte.dev/docs 500s out with an "Unexpected token < in JSON at position 0"

Describe the bug
Navigating to https://kit.svelte.dev/docs gives a 500 error: "Unexpected token < in JSON at position 0". Yikes indeed.

Also affects old navigation schema, for instance: https://kit.svelte.dev/docs#adapters
Looks like it's a combination of the new docs refactor and the new automatic shadow endpoint system thinking its going to receive a JSON file and getting something else instead...
Reproduction
Go to https://kit.svelte.dev/docs or https://kit.svelte.dev/docs#adapters
Logs
No response
System Info
System:
    OS: Linux 5.13 Linux Mint 20.3 (Una)
    CPU: (8) x64 Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
    Memory: 601.65 MB / 15.38 GB
  Browsers:
    Brave Browser: <IP_ADDRESS>
    Chromium: 98.0.4758.80
    Firefox: 97.0

Severity
annoyance
Additional Information
No response

The underlying cause is being tracked in #3848.

This can be temporarily fixed by using a client-rendered version of the site. Click "FAQ" in the upper right corner, then click "Docs" and voila!

Seems to be a duplicate of https://github.com/sveltejs/kit/issues/3849.

The second half of https://github.com/sveltejs/kit/issues/3848 — https://kit.svelte.dev/docs is failing at the moment because (AFAICT) it's attempting to server-render /docs.json which was (necessarily, because it uses the filesystem) prerendered during build.

Search results of Docs • SvelteKit and Read the docs are broken at the moment as well.
For the mean time, I am removing all https://kit.svelte.dev/docs histories from my synced web-browsers.

Closed by https://github.com/sveltejs/kit/pull/3858.