[SPARK-27521][SQL] Move data source v2 to catalyst module

What changes were proposed in this pull request?
Currently we are in a strange status that, some data source v2 interfaces(catalog related) are in sql/catalyst, some data source v2 interfaces(Table, ScanBuilder, DataReader, etc.) are in sql/core.
I don't see a reason to keep data source v2 API in 2 modules. If we should pick one module, I think sql/catalyst is the one to go.
Catalyst module already has some user-facing stuff like DataType, Row, etc. And we have to update Analyzer and SessionCatalog to support the new catalog plugin, which needs to be in the catalyst module.
This PR can solve the problem we have in https://github.com/apache/spark/pull/24246
How was this patch tested?
existing tests

No matter how we are going to change the package organization, I think there is no objection to move data source v2 to sql/catalyst.
cc @rxin @rdblue @mccheah @gatorsmile @gengliangwang

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10062/
Test PASSed.

Test build #104748 has started for PR 24416 at commit c62aae3.

Test build #104748 has finished for PR 24416 at commit c62aae3.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104748/
Test FAILed.

Hi, @cloud-fan . Could you fix the build error?
[error] /home/jenkins/workspace/SparkPullRequestBuilder/sql/catalyst/src/main/java/org/apache/spark/sql/sources/v2/TableProvider.java:21:  error: cannot find symbol
[error] import org.apache.spark.sql.sources.DataSourceRegister;
[error]

Test build #104796 has started for PR 24416 at commit afc2fec.

Test build #104796 has finished for PR 24416 at commit afc2fec.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait DataSourceRegister

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104796/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10107/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10119/
Test PASSed.

Test build #104808 has started for PR 24416 at commit 0163dec.

Test build #104808 has finished for PR 24416 at commit 0163dec.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait DataSourceRegister

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104808/
Test FAILed.

I remind sql/catalyst module is (implicitly) tend to be treated as non-public so leaving classes as public was OK - if we decide to place public APIs to sql/catalyst and treat sql/catalyst module be public, there might be some spots we may want to explicitly hide.

@HeartSaVioR this has been discussed before - I think while that was previously the case, the current push to Data Source V2 necessitates us to change that opinion. There might be a different Maven module organization that is sensible, for example having an sql-api or catalyst-api module.

I remind sql/catalyst module is (implicitly) tend to be treated as non-public

Actually this is not true, at least the public DataType API is in sql/catalyst.

Looks like this moved the v2 package and the vectorized package. Is it necessary to move ArrowColumnVector? That seems like an implementation class and not an interface.
Also, does it make sense to move DataSourceRegister? That seems like a part of v1 that we've just reused. Maybe we should add a method to TableProvider instead of moving the trait?

Looks like this moved the v2 package and the vectorized package. Is it necessary to move ArrowColumnVector? That seems like an implementation class and not an interface.

I think @cloud-fan move the vectorized package because org.apache.spark.sql.vectorized.ColumnarBatch is used in PartitionReaderFactory .

@gengliangwang, I agree. It looks like the entire package was moved. Is it necessary to move the entire package, or should this move just the interfaces?

If it is in the sql package, we can't use it catalyst package unless we create new abstraction in API level.  Also, ColumnarBatch is a final class.  Though ugly, it seems necessary to me.

Sounds fine to me.

Could you resolve the conflicts please? @cloud-fan

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11073/
Test PASSed.

Test build #105818 has started for PR 24416 at commit 4d56a2b.

Test build #105818 has finished for PR 24416 at commit 4d56a2b.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105818/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11074/
Test PASSed.

Test build #105819 has started for PR 24416 at commit ab4f40c.

Test build #105819 has finished for PR 24416 at commit ab4f40c.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105819/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11075/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11078/
Test PASSed.

Test build #105822 has started for PR 24416 at commit c9e00bc.

Test build #105822 has finished for PR 24416 at commit c9e00bc.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105822/
Test FAILed.

Test build #105825 has started for PR 24416 at commit 7af4ed1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11081/
Test PASSed.

Test build #105825 has finished for PR 24416 at commit 7af4ed1.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105825/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11086/
Test PASSed.

Test build #105830 has started for PR 24416 at commit 21ed731.

Test build #105830 has finished for PR 24416 at commit 21ed731.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105830/
Test FAILed.

Test build #105842 has started for PR 24416 at commit a9eac90.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11099/
Test PASSed.

Test build #105842 has finished for PR 24416 at commit a9eac90.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105842/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11114/
Test PASSed.

Test build #105858 has started for PR 24416 at commit 9567cc7.

Test build #105858 has finished for PR 24416 at commit 9567cc7.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105858/
Test FAILed.

Retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11152/
Test PASSed.

Test build #105896 has started for PR 24416 at commit 9567cc7.

Test build #105896 has finished for PR 24416 at commit 9567cc7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105896/
Test PASSed.

Test build #106018 has started for PR 24416 at commit 4b9c964.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11268/
Test PASSed.

Test build #106018 has finished for PR 24416 at commit 4b9c964.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106018/
Test FAILed.

Test build #106020 has started for PR 24416 at commit c1b5932.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11270/
Test PASSed.

Test build #106020 has finished for PR 24416 at commit c1b5932.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106020/
Test FAILed.

I moved a few public classes from sql/core to sql/catalyst, without changing the package name. I think this is binary compatible as sql/core depends on sql/catalyst.
I'm sure if the mima failures are legitimate,  do you have any ideas? @srowen @JoshRosen

The MiMa warning is legit because those classes are no longer in core. catalyst depends on core, right, not the other way around? so anyone that only depends on core won't see these classes anymore. That could be fine as a breaking change for Spark 3, but it's a legit warning.

sql/core depends on sql/catalyst, so the classes should always be there. When looking at sql/core on its own, this is legitimate. But it should be safe to make this change because the classes are always available if you have all of the required dependencies.

Am I crazy, or is it other way around? I don't see a catalyst dependency from core, and wouldn't actually expect that? But yeah I'm agreeing that this is an OK 'breaking' change to exclude.

sql/catalyst depends on sql/core: https://github.com/apache/spark/blob/master/sql/core/pom.xml#L63
Though both are dependent on spark-core.

Oh, I kept thinking and looking at core not sql/core. Right. Well, same conclusion.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11294/
Test PASSed.

Test build #106044 has started for PR 24416 at commit 9220e78.

Test build #106044 has finished for PR 24416 at commit 9220e78.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106044/
Test PASSed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11443/
Test PASSed.

Test build #106194 has started for PR 24416 at commit 9220e78.

Test build #106194 has finished for PR 24416 at commit 9220e78.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106194/
Test PASSed.

Thanks! Merged to master.

Finally, Nice! Thank you, guys.

Hello,
I am trying to migrate from Spark 2.x to 3.x and replacing those
import org.apache.spark.sql.sources.v2.writer.streaming.StreamWriter
import org.apache.spark.sql.sources.v2.{DataSourceOptions, StreamWriteSupport}

had been challenging. My investigation conducted me here, I read the code change and added sql-catalyst to my dependencies but can't figure out where those files went.
Anyone could point me in the right direction ? Thanks !

DS v2 has evolved a lot from Spark 2 to 3. I'm afraid you may need to stay with Spark 2 or rewrite your v2 source entirely.

DS v2 has evolved a lot from Spark 2 to 3. I'm afraid you may need to stay with Spark 2 or rewrite your v2 source entirely.

I dont mind rewriting it. Would you have a tutorial I can follow ?

You can always find it in tests :P
https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2Suite.scala

You can always find it in tests :P
https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2Suite.scala

Super, I will take a look. Thank you so much for your help ! Have a wonderful rest of your day.


Update web player in docs for custom registerComponent names

In the web player in the docs, allows AppRegistry.registerComponent('name', App) to use anything for 'name'. It is ignored by the web player - last registration wins.

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please sign up at https://code.facebook.com/cla - and if you have received this in error or have any questions, please drop us a line at<EMAIL_ADDRESS>Thanks!

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

@facebook-github-bot shipit

Thanks for importing. If you are an FB employee go to Phabricator to review.


load_file supports upserting

concern:
I think that the first step for this ticket would be to propose what the API change would look like. The append and join APIs are are already quite complicated and I wouldn't want to push that complexity into the load_file API (which is a crucial endpoint).
Please comment below with an API proposal @tatiana

With PR #210 load_file has an ability to append to DB.

On the table level we have this already implemented. We can close this right @dimberman ?


Validate Event Filters

Let’s skip the validation commit for now. It can be done in jsonrpc module, that way we dont need to do it per handler.

@jelilat I added a commit here, feel free to take a look and merge if you are happy with it

@jelilat I added a commit here, feel free to take a look and merge if you are happy with it

That's a much better approach. Thanks


Use items function instead of iteritems

Looks like in Python3 iteritems function was deprecated therefore playbook does no work on 3.x version,.
Replacing iteritems to items.

Handling: https://github.com/ernestas-poskus/ansible-prometheus/issues/26

According to https://www.tutorialspoint.com/python/dictionary_items.htm

items() returns a list of dict's (key, value) tuple pairs

So change shouldn't impact Python 2 users.

@tomaszkrzyzanowski
Issue is solved, I have added additional docker image for testing ansible python3, the problem was with iteritems function as you described. In Python3 iteritems function is deprecated and it is suggested to use items function instead. Final solution was not that simple, since items function in Python 3 (Python2 works as iteritems) every time returns random order which in turn makes playbook not idempotent. I have refactored main config from dict/hash variable to multi dimension array.
Thank you for reporting this :bow:


OTA update downloads but fails at the end

I am trying to perform an OTA update using my ESP8266. When I try to do the update, it starts downloading it and it fails at the end, saying that the firmware update failed.
Most of the chunks fail to download, but after retrying it can manage to download them and continue the download of the firmware. However, when it reaches the end of the download the Serial Monitor output is not successful.
[TB] Receive chunk (121), with size (4096) bytes
[TB] Error during Update.write
Progress 98.39%
[TB] Callback onMQTTMessage from topic: (v2/fw/response/0/chunk/121)
[TB] Receive chunk (121), with size (4096) bytes
[TB] Error during Update.write
[TB] Callback onMQTTMessage from topic: (v2/fw/response/0/chunk/122)
[TB] Receive chunk (122), with size (4096) bytes
[TB] Error during Update.write
Progress 99.19%
[TB] Callback onMQTTMessage from topic: (v2/fw/response/0/chunk/122)
[TB] Receive chunk (122), with size (4096) bytes
[TB] Error during Update.write
[TB] Callback onMQTTMessage from topic: (v2/fw/response/0/chunk/123)
[TB] Receive chunk (123), with size (1728) bytes
[TB] Error during Update.write
Downloading firmware failed

I have seen that at the beginning of the update the serial monitor says that there was an error during Update.begin so I think that the problem is because of that.
[TB] A new Firmware is available:
[TB] (1.1) => (1.2)
[TB] Attempting to download over MQTT...
Progress 0.81%
[TB] Callback onMQTTMessage from topic: (v2/fw/response/0/chunk/1)
[TB] Receive chunk (1), with size (4096) bytes
[TB] Error during Update.write
[TB] Callback onMQTTMessage from topic: (v2/fw/response/0/chunk/0)
[TB] Receive chunk (0), with size (4096) bytes
[TB] Error during Update.begin
[TB] Callback onMQTTMessage from topic: (v2/fw/response/0/chunk/1)
[TB] Receive chunk (1), with size (4096) bytes
[TB] Error during Update.write
Progress 1.61%
[TB] Callback onMQTTMessage from topic: (v2/fw/response/0/chunk/1)
[TB] Receive chunk (1), with size (4096) bytes
[TB] Error during Update.write
[TB] Callback onMQTTMessage from topic: (v2/fw/response/0/chunk/2)

The code I am using right now is the following:
(I have removed credentials like the WiFi SSID, Password, Server IP and Token)
#include <ESP8266WiFi.h>
#define THINGSBOARD_ENABLE_PROGMEM 0
#include <ThingsBoard.h>
#include <Arduino.h>
#include <Adafruit_AHTX0.h>
#include <TBPubSubClient.h>

// Firmware title and version used to compare with remote version, to check if an update is needed.
// Title needs to be the same and version needs to be different --> downgrading is possible
constexpr char CURRENT_FIRMWARE_TITLE[] = "Test";
constexpr char CURRENT_FIRMWARE_VERSION[] = "1.2";

// Firmware state send at the start of the firmware, to inform the cloud about the current firmware and that it was installed correctly,
// especially important when using OTA update, because the OTA update sends the last firmware state as UPDATING, meaning the device is restarting
// if the device restarted correctly and has the new given firmware title and version it should then send thoose to the cloud with the state UPDATED,
// to inform any end user that the device has successfully restarted and does actually contain the version it was flashed too
constexpr char FW_STATE_UPDATED[] = "UPDATED";

// Maximum amount of retries we attempt to download each firmware chunck over MQTT
constexpr uint8_t FIRMWARE_FAILURE_RETRIES = 255U;

// Size of each firmware chunck downloaded over MQTT,
// increased packet size, might increase download speed
constexpr uint16_t FIRMWARE_PACKET_SIZE = 4096U;

// PROGMEM can only be added when using the ESP32 WiFiClient,
// will cause a crash if using the ESP8266WiFiSTAClass instead.
constexpr char WIFI_SSID[] = "";
constexpr char WIFI_PASSWORD[] = "";

constexpr char TOKEN[] = "";

// Thingsboard we want to establish a connection too
constexpr char THINGSBOARD_SERVER[] = "";

// MQTT port used to communicate with the server, 1883 is the default unencrypted MQTT port,
// whereas 8883 would be the default encrypted SSL MQTT port
constexpr uint16_t THINGSBOARD_PORT = 1883U;

// Maximum size packets will ever be sent or received by the underlying MQTT client,
// if the size is to small messages might not be sent or received messages will be discarded
constexpr uint32_t MAX_MESSAGE_SIZE = 512U;

// Baud rate for the debugging serial connection
// If the Serial output is mangled, ensure to change the monitor speed accordingly to this variable
constexpr uint32_t SERIAL_DEBUG_BAUD = 115200U;

// Initialize underlying client, used to establish a connection
WiFiClient espClient;

// Initialize ThingsBoard instance with the maximum needed buffer size
ThingsBoard tb(espClient, MAX_MESSAGE_SIZE);

// Statuses for updating
bool currentFWSent = false;
bool updateRequestSent = false;

void MQTTconnect();
void printTempValues(int sleep);
bool Chrono(int T);
void goToSleep();

/// @brief Initalizes WiFi connection,
// will endlessly delay until a connection has been successfully established
void InitWiFi() {
  Serial.println("Connecting to AP ...");
  // Attempting to establish a connection to the given WiFi network
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  while (WiFi.status() != WL_CONNECTED) {
    // Delay 500ms until a connection has been succesfully established
    delay(500);
    Serial.print(".");
  }
  Serial.println("Connected to AP");
}

/// Reconnects the WiFi uses InitWiFi if the connection has been removed
/// Returns true as soon as a connection has been established again
bool reconnect() {
  // Check to ensure we aren't connected yet
  const wl_status_t status = WiFi.status();
  if (status == WL_CONNECTED) {
    return true;
  }

  // If we aren't establish a new connection to the given WiFi network
  InitWiFi();
  return true;
}

/// Updated callback that will be called as soon as the firmware update finishes
/// success Either true (update successful) or false (update failed)
void updatedCallback(const bool& success) {
  if (success) {
    Serial.println("Done, Reboot now");
    ESP.restart();
    return;
  }
  Serial.println("Downloading firmware failed");
}

/// Progress callback that will be called every time we downloaded a new chunk successfully
void progressCallback(const uint32_t& currentChunk, const uint32_t& totalChuncks) {
  Serial.printf("Progress %.2f%%\n", static_cast<float>(currentChunk * 100U) / totalChuncks);
}

const OTA_Update_Callback callback(&progressCallback, &updatedCallback, CURRENT_FIRMWARE_TITLE, CURRENT_FIRMWARE_VERSION, FIRMWARE_FAILURE_RETRIES, FIRMWARE_PACKET_SIZE, 60000);

Adafruit_AHTX0 aht;

// -- AHT Red LED pin.
//      It is configured to be pulled LOW when BME280 sensor is NOT detected.
#define RED_LED_AHT D3//0//

// -- Power Supply pin.
//      It is configured to be pulled High to feed ADS1115, so when ESP8266 is on DeepSleep mode
//      it will be pulled Low and battery led will be turned off.
#define POWER_SUP_PIN D6//12//

// -- ADS Red LED pin.
//      It is configured to be pulled low when BME280 sensor is NOT detected.
#define RED_LED_ADS D8//15//D7//13//

PubSubClient client(espClient);

// System variables
bool needReset = false;
float SleepTime = 3e7; // 1.8e9 is 30 minutes
int BlinkStatus = 0;
unsigned aht_status;

void setup() {
  // Initalize serial connection for debugging
  Serial.begin(SERIAL_DEBUG_BAUD);

  delay(100);

 // -- AHT20 initialization
  aht_status = aht.begin();
  delay(200);
  Serial.print("AHT20 Status: "); Serial.println(aht_status);

  delay(200);

  InitWiFi();

}

void loop() {
  //client.loop();
    if (needReset){
      Serial.println("Rebooting after 1 second.");
      delay(1000);
      ESP.restart();
    }

  if (!reconnect()) {
    return;
  }

  if (!tb.connected()) {
    // Reconnect to the ThingsBoard server,
    // if a connection was disrupted or has not yet been established
    Serial.printf("Connecting to: (%s) with token (%s)\n", THINGSBOARD_SERVER, TOKEN);
    if (!tb.connect(THINGSBOARD_SERVER, TOKEN, THINGSBOARD_PORT)) {
      Serial.println("Failed to connect");
      return;
    }
  }
  else{
      printTempValues(1); //argument 1 is for sending device to DeepSleep
    }

  }

  if (!currentFWSent) {
    currentFWSent = tb.Firmware_Send_Info(CURRENT_FIRMWARE_TITLE, CURRENT_FIRMWARE_VERSION) && tb.Firmware_Send_State(FW_STATE_UPDATED);
  }

  if (!updateRequestSent) {
    Serial.println("Firwmare Update Subscription...");
    // See https://thingsboard.io/docs/user-guide/ota-updates/
    // to understand how to create a new OTA pacakge and assign it to a device so it can download it.
    updateRequestSent = tb.Subscribe_Firmware_Update(callback);
  }

  client.loop();
  tb.loop();
}

//////////////////// Function for mapping between two ranges ////////////////////
float mapfloat(float x, float in_min, float in_max, float out_min, float out_max)
{
  return (x - in_min) * (out_max - out_min) / (in_max - in_min) + out_min;
}

//////////////////// Print values of AHT20 sensor ////////////////////
void printTempValues(int sleep) {

  // -- Now, get temperature and humidity values.
  sensors_event_t humidity, temp;
  aht.getEvent(&humidity, &temp);
  Serial.print("Temperature: "); Serial.print(temp.temperature); Serial.println(" °C");

  Serial.print("Humidity: "); Serial.print(humidity.relative_humidity); Serial.println("% rH");

  Serial.print("Sent telemetry for 'temperature': ");Serial.println(tb.sendTelemetryFloat("temperature", temp.temperature));
  Serial.print("Sent telemetry for 'humidity': ");Serial.println(tb.sendTelemetryFloat("humidity", humidity.relative_humidity));
  delay(1000);
  if (sleep == 1){
    delay(1000);
    goToSleep();
  }
}

//////////////////// Check sleep time and send ESP to sleep ////////////////////
void goToSleep() {
  Serial.println("Now I will sleep for 30 seconds");
  delay(200);
  ESP.deepSleep(SleepTime); // 3.6e9 is 1 hour =  3.600.000.000 microseconds

}

Can you try if the update works if you do not enter deepSleep on the ESP8266. I'm not sure if flashing the firmware onto the device is supported while the device is in deep sleep mode.

Yes, I forgot to mention I was using v0.10.2.
After upgrading the version of the library I was getting the error as expected:
[TB] No keys that we subscribed too were changed, skipping callback
[TB] Unable to de-serialize received json data with error (InvalidInput)

After implementing the fix in #149 I get:
[TB] Failed to initalize flash updater
[TB] Receive chunk (0), with size (4096) bytes

Regarding the deepSleep:
The goal of this project is to have a remote device that will be sleeping most of the time and only wake up to take a few measurements and go back to sleep right after. The user should have the ability to update the code remotely using OTA updates.

Since the device is subscribed to the firmware update topic it will not update the firmware while it is sleeping because it will not be connected to the Internet so the device is not able to get the data needed for it. So, we were thinking about updating the code on ThingsBoard and start the update once the device wakes up. (We would have some flags to prevent the ESP8266 go to sleep before we make sure the device has correctly updates but we have not implemented it yet.)

We have only tried to update the code of the device while it is not sleeping to check if it is able to download and update correctly and then is when we get this error. (After this we would work on updating the code while it is sleeping so it can update after waking up but, as I said, it is not implemented yet because we are currently working on getting a simpler version working).

Okay bad news, this is relatively surely not a problem with this library, because it is the underlying component that should flash the memory that fails to initalize.
Good news however it is probably not an implementation problem but a problem with how you are using the update.
The initial flash begin can fail because of multiple errors, but one of them and the one I find currently most likely is that the binary size you receive from the server is too big for your OTA data partition or that you have not created a 2nd OTA data partition altogether.
What kind of partitions.csv file are you using, because I am assuming you are using the default one and did not change the configuration for it. Because if you did not that might be the cause for the update initialisation failing, because to do an OTA update the partitions.csv needs two app partitions and the new firmware is then flashed onto the non-active one. Once that has been completed we reboot onto the new flashed firmware.
But the problem this of course will not work if there is no 2nd app partition. An example of how the partition should look like for an OTA update can be found below.
# Name, Type, SubType, Offset, Size, Flags
otadata,            data,       ota,            0xE000,           0x2000,
app0,               app,        ota_0,                ,           0x300000,
app1,               app,        ota_1,                ,           0x300000,
spiffs,             data,       spiffs,               ,           0x3F6000,
nvs,                data,       nvs,                  ,           0x25000,
nvs_key,            data,       nvs_keys,             ,           0x25000,
test,               app,        test,                 ,           0x300000,

Okay good news, that was the problem. I did not fix it exactly as you said because apparently for the ESP8266 module the way that changing the flash size is this. So I changed the board_build.ldscript flag in my platformio.ini and now the OTA update works.

Nice to know that the partitions file fixed your issue.
Would be nice if you could close this issue as resolved.

Yes, thank you!


Revert "Removed dublicates"

Reverts Awesome-Windows/Awesome#338

Pull Request Reverted.

Pull Request Reverted.


Redesign Kedge website

Redesigns the Kedge website to a new friendly front-layout.

Going to merge this as-is in order to get it up-and-running and
go-from-there!

If you have any suggestions / updates, please open an issue and ping me and I can quickly change it, otherwise, if you simply fork the gh-pages repo, you can make the correct changes to the index!


main

MNT: Re-rendered with conda-build 24.7.1, conda-smithy 3.44.9, and conda-forge-pinning 2<IP_ADDRESS>.06.14
Add setuptools to host requirements

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.


datapath/linux,maps/ipcache: consistently use BackedByLPM() helper

This commit introduces no functional changes.

This change is

test-me-please

Coverage decreased (-0.03%) to 44.592% when pulling df1416221c0aa7fbdb40da943bd4ecb640bb0ea4 on pr/rolinh/use_backedbylpm_helper into abf5366a5d4418f01432563d8e402fb957cd12e7 on master.


gluoncv resnet50_v1b load_parameters  attributeError: /mxnet/lib/libmxnet.so: undefined symbol: MXIsNumpyShape

Description
(A clear and concise description of what the bug is.)
when i use gluoncv mask rcnn, resnet50_v1b is pretained_base=true, there is attributeError
Error Message
(Paste the complete error message, including stack trace.)

To Reproduce
(If you developed your own code, please provide a short script that reproduces the error. For existing examples, please provide link.)
Steps to reproduce
(Paste the commands you ran that produced the error.)

What have you tried to solve it?

Environment
We recommend using our script for collecting the diagnositc information. Run the following command and paste the outputs below:
cudn 10
mxnet 1.5
gluoncv 0.6
curl --retry 10 -s https://raw.githubusercontent.com/dmlc/gluon-nlp/master/tools/diagnose.py | python

# paste outputs here

@apeforest assign @reminisce