Trouble Batching Data

I've followed many examples and instructions in this repository, but am still having issues batching data.
My situation: I’m currently using an ESP32 iOT microcontroller to collect some data and am sending the data to a local InfluxDB server for storage. I’m using the Arduino InfluxDB client to connect the microcontroller to the database and am wanting to batch my data to increase my sample rate. I’ve looked at batching examples and instructions on GitHub but it doesn’t seem to work. In my testing, I can either only get the data to come in synchronously or it doesn’t make it into InfluxDB at all.
Here is my code, in case it is helpful:
/**
 * Secure Write Example code for InfluxDBClient library for Arduino
 * Enter WiFi and InfluxDB parameters below
 *
 * Demonstrates connection to any InfluxDB instance accesible via:
 *  - unsecured http://...
 *  - secure https://... (appropriate certificate is required)
 *  - InfluxDB 2 Cloud at https://cloud2.influxdata.com/ (certificate is preconfigured)
 * Measures signal level of all visible WiFi networks including signal level of the actually connected one
 * This example demonstrates time handling, how to write measures with different priorities, batching and retry
 * Data can be immediately seen in a InfluxDB 2 Cloud UI - measurements wifi_status and wifi_networks
 **/
#include <Arduino.h>
#include <math.h>
#if defined(ESP32)
#include <WiFiMulti.h>
WiFiMulti wifiMulti;
#define DEVICE "ESP32"
#elif defined(ESP8266)
#include <ESP8266WiFiMulti.h>
ESP8266WiFiMulti wifiMulti;
#define DEVICE "ESP8266"
#define WIFI_AUTH_OPEN ENC_TYPE_NONE
#endif

#include <InfluxDbClient.h>
#include <InfluxDbCloud.h>

// WiFi AP SSID
#define WIFI_SSID "Dynamiq Guest"
// WiFi password
#define WIFI_PASSWORD "comeflywithme"
// InfluxDB v2 server url, e.g. https://eu-central-1-1.aws.cloud2.influxdata.com (Use: InfluxDB UI -> Load Data -> Client Libraries)
#define INFLUXDB_URL "http://<IP_ADDRESS>:8086"
// InfluxDB v2 server or cloud API authentication token (Use: InfluxDB UI -> Load Data -> Tokens -> <select token>)
#define INFLUXDB_TOKEN "-qRM4acj3ic_Cf7hmeymLhrJzrXQVcRDgQEANPNlplMXddUEFtSjtWJdazJnZ-_gQH7FjMDYQ6yRYp_4ed5mzQ=="
// InfluxDB v2 organization id (Use: InfluxDB UI -> Settings -> Profile -> <name under tile> )
#define INFLUXDB_ORG "Dynamiq"
// InfluxDB v2 bucket name (Use: InfluxDB UI -> Load Data -> Buckets)
#define INFLUXDB_BUCKET "Wifi_Testing"
// Set timezone string according to https://www.gnu.org/software/libc/manual/html_node/TZ-Variable.html
// Examples:
//  Pacific Time:   "PST8PDT"
//  Eastern:        "EST5EDT"
//  Japanesse:      "JST-9"
//  Central Europe: "CET-1CEST,M3.5.0,M10.5.0/3"
#define TZ_INFO "CET-1CEST,M3.5.0,M10.5.0/3"
// NTP servers the for time synchronization.
// For the fastest time sync find NTP servers in your area: https://www.pool.ntp.org/zone/
#define NTP_SERVER1  "pool.ntp.org"
#define NTP_SERVER2  "time.nis.gov"
#define WRITE_PRECISION WritePrecision::MS
#define MAX_BATCH_SIZE 20
#define WRITE_BUFFER_SIZE 40

// InfluxDB client instance with preconfigured InfluxCloud certificate
InfluxDBClient client(INFLUXDB_URL, INFLUXDB_ORG, INFLUXDB_BUCKET, INFLUXDB_TOKEN, InfluxDbCloud2CACert);
// InfluxDB client instance without preconfigured InfluxCloud certificate for insecure connection
//InfluxDBClient client(INFLUXDB_URL, INFLUXDB_ORG, INFLUXDB_BUCKET, INFLUXDB_TOKEN);

// Data point
Point sensorSine("sine_wave");
Point sensorCosine("cosine_wave");

// Number for loops to sync time using NTP
int iterations = 0;
double t = 0;

void setup() {
  Serial.begin(9600);

  // Setup wifi
  WiFi.mode(WIFI_STA);
  wifiMulti.addAP(WIFI_SSID, WIFI_PASSWORD);

  Serial.print("Connecting to wifi");
  while (wifiMulti.run() != WL_CONNECTED) {
    Serial.print(".");
    delay(500);
  }
  Serial.println();

  // Add tags
  sensorSine.addTag("device", DEVICE);
  sensorSine.addTag("SSID", WiFi.SSID());
  sensorCosine.addTag("device", DEVICE);
  sensorCosine.addTag("SSID", WiFi.SSID());

  // Alternatively, set insecure connection to skip server certificate validation
  client.setInsecure();

  // Accurate time is necessary for certificate validation and writing in batches
  // Syncing progress and the time will be printed to Serial.
  timeSync(TZ_INFO, NTP_SERVER1, NTP_SERVER2);

  // Check server connection
  if (client.validateConnection()) {
    Serial.print("Connected to InfluxDB: ");
    Serial.println(client.getServerUrl());
  } else {
    Serial.print("InfluxDB connection failed: ");
    Serial.println(client.getLastErrorMessage());
  }

  // Enable messages batching and retry buffer
  client.setWriteOptions(WriteOptions().writePrecision(WRITE_PRECISION).batchSize(MAX_BATCH_SIZE).bufferSize(WRITE_BUFFER_SIZE));
}

void loop() {
  // Sync time for batching once per hour
  if (iterations++ >= 100) {
    timeSync(TZ_INFO, NTP_SERVER1, NTP_SERVER2);
    iterations = 0;
  }

  for(int i = 0; i < MAX_BATCH_SIZE; i++){
    // Report RSSI of currently connected network
    sensorSine.clearFields();
    sensorCosine.clearFields();
    //sensorSine.setTime(time(nullptr));
    //sensorCosine.setTime(time(nullptr));
    sensorSine.addField("sin", 10*cos(0.5*t)+5);
    sensorCosine.addField("cos", 10*sin(0.5*t)+5);

    // Print what are we exactly writing
    Serial.print("Writing: ");
    Serial.print(client.pointToLineProtocol(sensorSine));
    Serial.print("\t");
    Serial.println(client.pointToLineProtocol(sensorCosine));

    // Write point into buffer - high priority measure
    client.writePoint(sensorSine);
    client.writePoint(sensorCosine);

    // Clear fields for next usage. Tags remain the same.
    sensorSine.clearFields();
    sensorCosine.clearFields();
    t = t + 0.01;
  }
  client.flushBuffer();

    // If no Wifi signal, try to reconnect it
    if (wifiMulti.run() != WL_CONNECTED) {
      Serial.println("Wifi connection lost");
    }

  // End of the iteration - force write of all the values into InfluxDB as single transaction
  Serial.println("Flushing data into InfluxDB");
  if (!client.flushBuffer()) {
    Serial.print("InfluxDB flush failed: ");
    Serial.println(client.getLastErrorMessage());
    Serial.print("Full buffer: ");
    Serial.println(client.isBufferFull() ? "Yes" : "No");
  }

  // Wait 10s
  //Serial.println("Wait 10s");
  //delay(2000);
}

@eric-w-dyn, thanks for using this client.
I don't see a big problem in your code.  All points should be written unless there is an error. How do you check data?
Few points to improve:

You are writing 2 points, but your writing loop ends in MAX_BATCH_SIZE. This means the first batch is full in the half of the loop and it is written immediately. Also, the second batch written is in the end of the loop .
if you want to rely on flushBuffer, make buffer size slightly higher, e.g.#num-points+1
Check the return status of writePoint and flushBuffer, like:

if (!client.flushBuffer()) {
    Serial.print("InfluxDB flush failed: ");
    Serial.println(client.getLastErrorMessage());
}

to know precisely what happens during writing, enable debug by uncommeting line 32 in the library sources src/InfluxDbClient.cpp:

// Uncomment bellow in case of a problem and rebuild sketch
#define INFLUXDB_CLIENT_DEBUG_ENABLE
#include "util/debug.h"

@vlastahajek Thanks for the tips. I think I got batching working properly. However, I'm still not happy for the results that I'm getting. Do you have any configuration tips to minimize the time it take to flush the batch? I'm collecting data from an analog sensor at 80Hz, thus I would like to write a continuous stream to InfluxDB at 80Hz. I'm getting about 80Hz within each batch, but there are relatively long gaps between each batch where I lose information. I have attached a screenshot of my Grafana dashboard that shows these gaps.

@eric-w-dyn, when writing data less more then 2 per minute, you should set HTTP reuse:
client.setHTTPOptions(HTTPOptions().connectionReuse(true);

if you gather data at a different time than adding points, keep time for it.
If you will still encounter a lag, please enable debug and post the debug output here.

@vlastahajek I already had HTTP reuse enabled. And yes, I am syncing with a time server to ensure the points have the correct time.
 And this is the gab shown in the chart when you are not reading data.
Try sending smaller batches more frequently. Byt you will still get gaps.
As you want to read 80Hz data and you are using ESP32, I would suggest reading a sensor on core 0 (using xTaskCreatePinnedToCore) and writing data to a queue that would be read on in the loop (core 1) and written to the database.

Yeah, I've tried splitting tasks into different cores a little already, but haven't yet reached a working program. My understanding is that ESP32 should already split networking tasks into a different core by default. Perhaps I am misled in that understanding...

My understanding is that core 0 is used for radio (WiFi, BLE) management, not networking. I have a custom 433Mhz ASK sensors network. ESP32 acts as a gateway that writes data from sensors to a database. As writing takes time and had to put the radio receiving task to core 0 to don't miss 433MHz radio packets.
Where do have the server? I tried your use case and when I have a server on rpi on the same network, it's written almost instantly. Such long times are usually on remote SSL server (e.g. AWS)
If nothing changes in your setup to speed up writing, as sending 80 lines takes 4s (total roundtrip), you won't be able to write it at such speed. Even you will create data on core 0, writing takes so long that writing task will be a lot slower than reading task.

If you have a server on the local computer, writing shouldn't be a problem. No more than 100ms, which should be sufficient for you use case. If writing takes seconds, there must be a problem somewhere else.
When you turn off debug (writing of debug info adds some delay) and measure flushing, what is the time?  E.g. if batch-size is 100:
 uint32_t d = millis();
 if (!client.writePoint(sensorCosine)) {
    Serial.print("InfluxDB write failed: ");
    Serial.println(client.getLastErrorMessage());
  }
  d = millis()-d;
  if(i==99) {
      Serial.printf("Sending took %dms\n", d);
  }

Ok, I just tried that. For some reason, it's working a lot faster now... I'm getting much faster write speeds (up to 1000 Hz), but I'm still getting little breaks in the data stream. See screenshot below:

In debug mode, I notice that it says "Flushing buffer: is oversized false [...]" after every write record (see output below). This doesn't seem write. The whole point of batching is to avoid flushing data after every point.
5.415 [D] Flushing buffer: is oversized false, is timeout false, is buffer full false
5.504 [D] writeRecord: bufferPointer: 0, batchPointer: 0, _bufferCeiling: 0
5.583 [D] Flushing buffer: is oversized false, is timeout false, is buffer full false
5.673 [D] writeRecord: bufferPointer: 0, batchPointer: 0, _bufferCeiling: 0
5.752 [D] Flushing buffer: is oversized false, is timeout false, is buffer full false
5.842 [D] writeRecord: bufferPointer: 0, batchPointer: 0, _bufferCeiling: 0
5.921 [D] Flushing buffer: is oversized false, is timeout false, is buffer full false
6.011 [D] writeRecord: bufferPointer: 0, batchPointer: 0, _bufferCeiling: 0
6.090 [D] Flushing buffer: is oversized false, is timeout false, is buffer full false
6.179 [D] writeRecord: bufferPointer: 0, batchPointer: 0, _bufferCeiling: 0

This is confirmed in Grafana. Every time I refresh my dashboard, I see new data (even when I refresh it at like 2 Hz). It shouldn't be batching and flushing this frequently as I have my max batch size set to 2000 currently.
Here is my current loop() method, in case you want to inspect it:
void loop() {
  for (int i = 0; i < MAX_BATCH_SIZE; i++){
    sensorCosine.addField("cos", 10.0*cos(0.04*t)+10.0);

    if (!client.writePoint(sensorCosine)) {
        Serial.print("InfluxDB write failed: ");
        Serial.println(client.getLastErrorMessage());
    }

      t += 0.01;
      sensorCosine.clearFields();
  }

  // If no Wifi signal, try to reconnect it
  if (wifiMulti.run() != WL_CONNECTED) {
    Serial.println("Wifi connection lost");
  }

  // End of the iteration - force write of all the values into InfluxDB as single transaction
  Serial.println("Flushing data into InfluxDB");
  if (!client.flushBuffer()) {
    Serial.print("InfluxDB flush failed: ");
    Serial.println(client.getLastErrorMessage());
    Serial.print("Full buffer: ");
    Serial.println(client.isBufferFull() ? "Yes" : "No");
  }
}

The debug output [D] Flushing buffer: is oversized false, is timeout false, is buffer full false just says: there is no reason for automatic flushing.
If you have milliseconds precicion, some points will be overwritten because of the same timestamp, as writting to buffer is fast.
You don't need to call client.flushBuffer(), as data is written automatically when buffer if full. This would happen at the last cycle of the for  loop, however, not in case of batch size 2000. More bellow.
Actually, maximum supported batch size is 255 and  higher number are trimmed to this.  It is due to memory restrictions, mainly on ESP8266 device.  Regular points are usually long (several tags and fields), so this limit is set to prevent OOM crash.

Oh ok. I wasn't aware that 255 was the max batch size. I remember reading somewhere that the optimal batch size for Influx is 2000, but it must not be applicable in my situation.
Since ESP32 has more memory, is it possible to change the library code to increase the maximum support batch size?

If fact, recommended bachsize is 5000. However this is meant for normal computer/server, not embeded device.
Even 2000 points will be problem. Maximum allocable memory even on ESP32 is only 110K, despite there is more RAM available.
This means you would need to be sure that points will have at maximum 50 chars!

I have struggled with this also. I have a high data rate application with 8k sample rate on esp32 reading vibration data with external psram and did a custom implementation for buffering to it. However the overhead is too big when sending packets even when queuing from the external buffer and for some reason I encounter random crashes when I still have heap space, I guess line protocol just isn't good for this. Can you recommend a binary alternative of it, I've tried looking up the header specifications for influx to do something, but it's pretty hard.

I’ve had better success recently using Bluetooth rather than Wifi. I’m using Python with the Bleak library to ingest the data into Influx. I can get close to 1kHz sample rate with no batching

Internet communication is required in this project. The main issue with line protocol here is the use of strings in multiple buffers and then they are copied in the tcp packet buffer again from what I reckon, that's a lot of overhead when you have a lot of points. I guess a tcp packet can't be build as a stream of strings to create it while sending it in order to save resources, or isn't implemented. Best option is to send binary data and I would have preferred to send directly to the db with this, tried with mqtt before but had some problems configuring the broker with the latest telegraf server. Another potential solution would be if I try to patch this lib to use the psram directly for its buffers, I'm not sure that would help either considering the overhead.

I have good news for you.
I have already rewritten posting data by streaming batch data. This means, no more max alloc block size limitation. As each line is separately allocated, it should allow a big batch on ESP32.
Increasing batch size limit is not done yet, but you can try zero-alloc writing in the feat/stream_batch branch

I was away for the holidays, will go testing it 👍. This feature should be mainstream


Haskell support

Add haskell support based on #20.

It was branched off of that one. Is there anything else I should do to make that work?

Changed it?


Add hosts to ingress spec when tls not enabled

The simplest way to achieve https://github.com/polyaxon/polyaxon/issues/278
@mouradmourafiq please let me know if this breaks backward compatibility and what the best way to go about preserving that would be and I'll update it accordingly.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Includes subject_ids in the classification CSV export

Requested by @ggdhines in #1798 (and originally in PFE issue #2543). Included as an array, but whoever's manning the aggregation station now that Greg has left is welcome make suggestions.
Review checklist

[ ] First, the most important one: is this PR small enough that you can actually review it? Feel free to just reject a branch if the changes are hard to review due to the length of the diff.
[ ] If there are any migrations, will they the previous version of the app work correctly after they've been run (e.g. the don't remove columns still known about by ActiveRecord).
[ ] If anything changed with regards to the public API, are those changes also documented in the apiary.apib file?
[ ] Are all the changes covered by tests? Think about any possible edge cases that might be left untested.

LGTM

Closes #1798


upgraded

upgraded
Originally posted by @joe198 in https://github.com/facebook/facebook-android-sdk/pull/832#issuecomment-708694896

ok


Updated readme for running esv-intersection in machine

How's this going @ShikhaMishra9?

We shouldn't have stale PR's. Can you merge this in @ShikhaMishra9?


How to add toolbar button for all the labels

I am using gmail.js ( gmail.tools.add_toolbar_button() ) to add toolbar button in gmail ui. But it adds the button only in inbox label, I want to see this button in all available labels like Sent, Draft etc.
How to achieve that. I think I need to modify the gmail.js code itself
Thanks in advance.

This issue was discussed in https://github.com/KartikTalwar/gmail.js/issues/440
I did,
function locationHashChanged() {
// if (location.hash === "#somecoolfeature") {
// somecoolfeature();
refresh(main);
console.log('cool features are waiting', location.hash);
// }
}
window.onhashchange = locationHashChanged;
to handle this


support OpenShift OAuth

OpenShift V3 is a docker/kubernetes based PaaS for on premise or public cloud: http://www.openshift.org/
a PR is on its way shortly for this...

I've not quite verified this totally works; when configured, an OpenShift button appears in the Gogs login page which then logs in via OpenShift and redirects back; but then barfs with the following error message:
2015-05-13T10:24:51.209542389Z [Macaron] Started GET /user/login for <IP_ADDRESS>
2015-05-13T10:24:51.215101246Z [Macaron] Completed /user/login 200 OK in 5.29264ms
2015-05-13T10:24:51.215101246Z 2015/05/13 10:24:51 [T] auto-login cookie cleared: jstrachan
2015-05-13T10:24:53.269067653Z [Macaron] Started GET /user/login/oauth2/openshift?next=/user/info/openshift for <IP_ADDRESS>
2015-05-13T10:24:53.269067653Z [Macaron] Completed /user/login/oauth2/openshift?next=/user/info/openshift 302 Found in 241.102µs
2015-05-13T10:25:23.488985266Z [Macaron] Started GET /user/login/openshift?code=YjQ5OTdhMzYtYTFlMy00YTNlLTg4MjEtYjM2ZDBjOTczNTIz&state=%2Fuser%2Finfo%2Fopenshift for <IP_ADDRESS>
2015-05-13T10:25:23.521187009Z Post https://my.openshift.server:8443/oauth/token: x509: certificate signed by unknown authority
2015-05-13T10:25:23.522246187Z [Macaron] Completed /user/login/openshift?code=YjQ5OTdhMzYtYTFlMy00YTNlLTg4MjEtYjM2ZDBjOTczNTIz&state=%2Fuser%2Finfo%2Fopenshift 302 Found in 32.192257ms
2015-05-13T10:25:23.689361979Z [Macaron] Started GET /oauth2error for <IP_ADDRESS>
2015-05-13T10:25:23.691146266Z 2015/05/13 10:25:23 [home.go:179 Profile()] [E] GetUserByName: User does not exist
2015-05-13T10:25:23.693843039Z [Macaron] Completed /oauth2error 404 Not Found in 4.506247ms

I figured I'd wait until I can verify that all the changes to gogs work for OAuth with OpenShift V3 before submitting the PR.
If anyone wants to play with the branch, its here...
https://github.com/fabric8io/gogs
there's a release with the patch if you wanna try it out: https://github.com/fabric8io/gogs/releases/tag/v0.6.2-openshift-m2
here's a sample section of the app.ini file to enable OpenShift V3 OAuth for gogs:
[oauth.twitter]
ENABLED=false

[oauth]
ENABLED=true

[oauth.openshift]
SCOPES=https://my.openshift.server:8443/console/user
TOKEN_URL=https://my.openshift.server:8443/oauth/token
ENABLED=true
AUTH_URL=https://my.openshift.server:8443/oauth/authorize
CLIENT_ID=gogs

[security]
INSTALL_LOCK=true

BTW here's the kubernetes json to run this gogs docker container with openshift OAuth enabled:
{
  "apiVersion" : "v1beta3",
  "kind" : "Template",
  "labels" : { },
  "metadata" : {
    "annotations" : { },
    "labels" : { },
    "name" : "gogs"
  },
  "objects" : [ {
    "apiVersion" : "v1beta3",
    "kind" : "Service",
    "metadata" : {
      "annotations" : { },
      "labels" : {
        "component" : "gogs",
        "provider" : "fabric8"
      },
      "name" : "gogs"
    },
    "spec" : {
      "ports" : [ {
        "port" : 80,
        "protocol" : "TCP",
        "targetPort" : 3000
      } ],
      "publicIPs" : [ ],
      "selector" : {
        "component" : "gogs",
        "provider" : "fabric8"
      }
    }
  }, {
    "apiVersion" : "v1beta3",
    "kind" : "ReplicationController",
    "metadata" : {
      "annotations" : { },
      "labels" : {
        "component" : "gogs",
        "provider" : "fabric8"
      },
      "name" : "gogs-controller"
    },
    "spec" : {
      "replicas" : 1,
      "selector" : {
        "component" : "gogs",
        "provider" : "fabric8"
      },
      "template" : {
        "metadata" : {
          "annotations" : { },
          "labels" : {
            "component" : "gogs",
            "provider" : "fabric8"
          }
        },
        "spec" : {
          "containers" : [ {
            "args" : [ ],
            "command" : [ ],
            "env" : [ {
              "name" : "GOGS_SERVER__ROOT_URL",
              "value" : "${GOGS_SERVER_PROTOCOL}${GOGS_SERVER_DOMAIN}"
            }, {
              "name" : "GOGS_SERVER__DOMAIN",
              "value" : "${GOGS_SERVER_DOMAIN}"
            }, {
              "name" : "GOGS_OAUTH_OPENSHIFT__SCOPES",
              "value" : "https://${DOMAIN}:8443/console/user"
            }, {
              "name" : "GOGS_OAUTH_OPENSHIFT__CLIENT_ID",
              "value" : "gogs"
            }, {
              "name" : "GOGS_OAUTH_OPENSHIFT__AUTH_URL",
              "value" : "https://${DOMAIN}:8443/oauth/authorize"
            }, {
              "name" : "GOGS_OAUTH__ENABLED",
              "value" : "true"
            }, {
              "name" : "GOGS_OAUTH_OPENSHIFT__ENABLED",
              "value" : "true"
            }, {
              "name" : "GOGS_WEBHOOK__SKIP_TLS_VERIFY",
              "value" : "true"
            }, {
              "name" : "GOGS_WEBHOOK__TASK_INTERVAL",
              "value" : "true"
            }, {
              "name" : "GOGS_OAUTH_TWITTER__ENABLED",
              "value" : "false"
            }, {
              "name" : "GOGS_OAUTH_OPENSHIFT__TOKEN_URL",
              "value" : "https://${DOMAIN}:8443/oauth/token"
            }, {
              "name" : "GOGS_SERVER__PROTOCOL",
              "value" : "http"
            } ],
            "image" : "fabric8/gogs",
            "imagePullPolicy" : "IfNotPresent",
            "name" : "gogs-container",
            "ports" : [ {
              "containerPort" : 3000,
              "name" : "http"
            }, {
              "containerPort" : 22,
              "name" : "ssh"
            } ],
            "volumeMounts" : [ ]
          } ],
          "nodeSelector" : { },
          "volumes" : [ ]
        }
      }
    }
  } ],
  "parameters" : [ {
    "description" : "The gogs server name",
    "name" : "GOGS_SERVER_DOMAIN",
    "value" : "gogs.my.openshift.server"
  }, {
    "description" : "The protocol for gogs to use; either http or https",
    "name" : "GOGS_SERVER_PROTOCOL",
    "value" : "http://"
  }, {
    "description" : "The domain name to append to the service name to access gogs repositories",
    "name" : "DOMAIN",
    "value" : "my.openshift.server"
  } ]
}

@Unknwon @dsimansk

what's the status of it?

I couldn't get it to work; we're now experimenting with an alternative approach which we should hopefully be able to get going soon:
https://github.com/fabric8io/openshift-auth-proxy

As far as I know @Unknwon has removed all the OAuth stuff :).

The openshift-auth-proxy should work well with Gogs's reverse proxy auth without any OAuth feature.

@jstrachan Was OpenShift OAuth ever implemented for gogs? I tested your template above but that doesn't seem to give me a "login with openshift" page.

@daniyalj I never got it to work I'm afraid

@jstrachan will the "openshift-auth-proxy" service work?


About save depth image

The depth image is captured with the format "sl::Mat sl::MAT_TYPE_32F_C1 ", while the "sl::Mat  read" and "sl::Mat  write" function can only read and write "MAT_TYPE_8U_C1, MAT_TYPE_8U_C3, MAT_TYPE_8U_C4". So I have the ｑｕestion than how can I save the depth data as a image? And how can I read again the saved depth image?
Thank you.

Hi @liuhang0727,
I have a working solution for you:

Write an .EXR with 32FC1 cv::Mat (no flags, just name).
Read the .EXR with cv::IMREAD_UNCHANGED.

That's it.


I want to use two recycler views in the same layout  with diffrent adapters. How can i implement the same using fast adapter?

@rakshithsj30 you just create 2 fastAdapter's.
There's nothing special about it.

thanks :)


Incorporate tracking of QR codes and subsequent notifications

Summary | Résumé
As a user, I wish to better track not only my problematic proximity contacts (#1304) but also the public places I have visited where my risks are presumably increased. I wish to support businesses that have invested in QR code upgrades during the pandemic, help them lessen their burden in recording my visit and encourage other businesses to similarly invest. Also applicable to public buildings such as schools, public libraries, hotels, flights, etc. Finally I wish to be notified sooner rather than later via the app rather than the problematic contact tracing, should a propagation be linked to that business.
Design detail | Précisions au sujet de la conception
The app would record a QR encounter as an event much similar to a proximity contact and report event (including actual business) in a daily log. The event could be temporarily notified by the app to the screen. Should a propagation be linked to a businesss I have frequented over a specific period, my app should be notified by the app server similarly to diagnosed contacts reporting back.
Unresolved questions / Out of scope | Questions non résolues ou hors sujet
The proposed enhancement is not expected to infringe on personal privacy (or individual freedom). Yes, the user of the app will be aware that visit is recorded similarly to an app “proximity contact”. Lesser even than signing guest register for people without the app. This proposed enhancement would make the app more useful, lessen the burden on existing contact tracing and help drive toward a higher adoption rate. This request does not cover the app server part. Business could be expected to run the app and report back a diagnosed case to the system. Yes this enhancement request also repurposes the app similarly to social distancing #1304 and reporting localized app adoption rate/ 3 digits postal code #1305. It could be reviewed as part of a relaunch driving towards higher adoption rates in response to a new virus strain propagating faster, hence requiring a stronger app solution.  And let's remember lessons from #Ebola when reviewing: "Moving quickly is essential" and avoid aiming for perfection before making a move”.

Ref: #1384

Hi @LeCanardQuoi - thank you for your comments. Please refer to this doc for an overview of our work on QR codes and notifications for venue outbreaks. We'll be updating documentation and details in that repo as we go forward.

Still not sure on process to exchange on related requirements. Two additional sub-reqs: 1) Tracking as proximity events non signatory QR scan code sites would have merits for an app user wishing to track their whereabouts (without depending on Google map trace). Would be useful before more comprehensive support gets off the ground, likely involving bureaucratic delays. 2) Make sure the feature can be individually activated and deactivated, similarly to activating/deactivating Bluetooth proximity events, and log these commands to the Google setting log.

In keeping with our newly really release cds-snc/covid-alert-documentation#31 practices, I've moved this up into the covid-alert-documentation repo as our central point for taking input from folks outside the team, and am converting the issue into a Discussion.


Refactor creation of private fields

Alaska Airlines Pull Request
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Resolves: #300
Summary:
Please summarize the scope of the changes you have submitted, what the intent of the work is and anything that describes the before/after state of the project.
Move the instantiation of the private fields into a function called privateDefaults, which is then called in the connectedCallback function to ensure that the fields get instantiated only when the component is attached to the DOM.
Type of change:
Please delete options that are not relevant.

[ ] New capability
[x] Revision of an existing capability
[ ] Infrastructure change (automation, etc.)
[ ] Other (please elaborate)

Checklist:

[x] My update follows the CONTRIBUTING guidelines of this project
[x] I have performed a self-review of my own update

By submitting this Pull Request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Pull Requests will be evaluated by their quality of update and whether it is consistent with the goals and values of this project. Any submission is to be considered a conversation between the submitter and the maintainers of this project and may require changes to your submission.
Thank you for your submission!
-- Auro Design System Team

:tada: This PR is included in version 2.17.10 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Bug 195608: add sigterm handler to insecurereadyz

This PR reopens https://github.com/openshift/cluster-kube-apiserver-operator/pull/1130 to add graceful termination shutdown to the readyz endpoint. Originally we thought the bug had been fully fixed.

/bugzilla refresh

/lgtm
/approve

/test k8s-e2e-gcp

/retest
/lgtm

/test e2e-aws-single-node

/retest

/test e2e-gcp-operator

/cherrypick release-4.8


[Bug] - Cap login signup input field length

No login / signup input fields can be > 256 chars (I think this is the db standard for strings).

username
name
password
email

Closing as this has already been fixed by PR https://github.com/garageScript/c0d3-app/pull/185.
Max length is 64 characters.


Features Page: Fix Page Speed with Video Embeds

Please change the preload:auto to preload:none on all local video embeds and make poster images for them too.

Would this have fixed the vertical thrashing of the page while it was loading?  It's working so much better, now.


implement TryAccountUpdateCode

In order to abstract code writing, add a TryUpdateCode function to the trie

On second thoughts, it's probably unnecessary if TryUpdateAccount is modified to also take a code argument.

it is unnecessary, closing


Kubernetes CIS: apiserver: EnablePodSecurityPolicy

What this PR does / why we need it: See #3507
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
If applicable:

[ ] documentation
[ ] unit tests
[ ] tested backward compatibility (ie. deploy with previous version, upgrade with this branch)

Release note:

Kubernetes CIS: apiserver: EnablePodSecurityPolicy

@marrobi FYI
@weinong @nicolerenee Please audit for any negative AKS effect. The proposal is to enable PodSecurityPolicy and RBAC by default for CIS benchmark adherence


Add a way to detect out of sync Chart.lock files

It is possible for a chart writer to add a dependency to the Chart.yaml but to forget to run helm dependency update before committing the change to version control. It would be useful to have a quick way to detect these mismatches.
❯ cd $(mktemp -d)
❯ helm create test
❯ cat <<EOT >> test/Chart.yaml
dependencies:
- name: zookeeper
  version: "2.0.0"
  repository: https://kubernetes-charts-incubator.storage.googleapis.com/
EOT
❯ helm dependency build test
...
# Bump the zookeeper version to 2.1.0 in the Chart.yaml
❯ vim test/Chart.yaml
❯ helm dependency build test
Error: the lock file (Chart.lock) is out of sync with the dependencies file (Chart.yaml). Please update the dependencies

It would be nice to have a command to run that could be used to detect this mismatch.

While I agree that some way to check that the Chart.yaml and Chart.lock are in-sync, I don't believe helm lint is the right place for this. helm lint runs a series of tests to verify that the chart is well-formed. It is supposed to catch errors that will cause the chart to fail a helm install due to missing or malformed fields. As an example, is not the responsibility of go lint to make sure that go.mod and go.sum are in sync, either.
Perhaps a helm dependency check is more aligned with your suggestion, similar to go mod verify.
For reference:
><> go help mod verify
usage: go mod verify

Verify checks that the dependencies of the current module,
which are stored in a local downloaded source cache, have not been
modified since being downloaded. If all the modules are unmodified,
verify prints "all modules verified." Otherwise it reports which
modules have been changed and causes 'go mod' to exit with a
non-zero status.

This would be a nice feature to have. Are there any tools outside of Helm that already implement this?


NGO Registration error #457: Change max_length=12

Issue Reference
This PR addresses the Issue : Fixes #457
Summarize
Please, describe what the PR does, the changes you have made.

Also, mention the key points if any to look into while code reviews

see https://github.com/IEEEKeralaSection/rescuekerala/pull/470

This PR should be closed. #480 #470


contact form added

added the contact form to the index.html file

added the contact form to the index.html file

Looks good, well done!

Merged

This is an automated message from Fork, Commit, Merge [BOT].

Thank you for your contribution! Your pull request has been merged. The files have been reset for the next contributor.
What's next?
If you're looking for more ways to contribute, I invite you to check out my other projects. Just click here to find more. These projects contain real issues that you can help resolve. You can also check out the Influences section in the README to find more projects similar to this one.
Also please leave a star to this project if you feel it helped you, i would really appreciate it.
I look forward to seeing your contributions!


Release 2023.06.04

This release adds the following PRs:

#40

:tada: This PR is included in version 1.1.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


.get_route_to() throwing uncatched errror

Description of Issue/Question
When attempting to get information about a connected route, an error is thrown
Setup
None
napalm-eos version
(Paste verbatim output from pip freeze | grep napalm-eos between quotes below)
napalm-eos==0.5.0

eOS version
(Paste verbatim output from show version | json between quotes below)
{
    "modelName": "vEOS",
    "internalVersion": "4.16.6M-3205780.4166M",
    "systemMacAddress": "00:15:5d:1b:db:cf",
    "serialNumber": "",
    "memTotal": 1897592,
    "bootupTimestamp":<PHONE_NUMBER>.0,
    "memFree": 92896,
    "version": "4.16.6M",
    "architecture": "i386",
    "isIntlVersion": false,
    "internalBuildId": "e796e94c-ba3b-4355-afcf-ef0abfbfaee3",
    "hardwareRevision": ""
}

Steps to Reproduce the Issue
Attempt to get_route_to() directly connected route
Error Traceback
(Paste the complete traceback of the exception between quotes below)
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from napalm_base import get_network_driver
>>> import pprint,getpass
>>> password = getpass.getpass()
Password:
>>> driver = get_network_driver('eos')
>>> device = driver('<IP_ADDRESS>', 'bewing', password)
>>> device.open()
>>> device.get_route_to(u'<IP_ADDRESS>')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "napalm_eos/eos.py", line 1071, in get_route_to
    'next_hop': napalm_base.helpers.ip(next_hop.get('nexthopAddr')),
  File "/home/bewing/napalm-eos/local/lib/python2.7/site-packages/napalm_base/helpers.py", line 242, in ip
    return py23_compat.text_type(IPAddress(addr))
  File "/home/bewing/napalm-eos/local/lib/python2.7/site-packages/netaddr/ip/__init__.py", line 306, in __init__
    'address from %r' % addr)
netaddr.core.AddrFormatError: failed to detect a valid IP address from None

JSON from the router for "show ip route":
localhost#show ip route <IP_ADDRESS> | json
{
    "vrfs": {
        "default": {
            "routes": {
                "<IP_ADDRESS>/24": {
                    "kernelProgrammed": true,
                    "directlyConnected": true,
                    "routeAction": "forward",
                    "vias": [
                        {
                            "interface": "Ethernet1"
                        }
                    ],
                    "hardwareProgrammed": true,
                    "routeType": "connected"
                }
            },
            "allRoutesProgrammedKernel": true,
            "routingDisabled": false,
            "allRoutesProgrammedHardware": true,
            "defaultRouteState": "reachable"
        }
    }
}

Do we want to return None, an empty string, or just leave the key unset for a directly connected route's nextHop?

When calling this method you should specify the protocol too. This implementation does not really reflect that, but for reference see: https://github.com/napalm-automation/napalm-junos/blob/develop/napalm_junos/junos.py#L961
So, if you do device.get_route_to(u'<IP_ADDRESS>', 'bgp') won't be displayed any connected routes.
So, in order to follow the same implementation patters as for JunOS and IOS-XR, we need to make the protocol arg mandatory and use it to filter out any route learned through a different protocol.

Why would we not want to support connected routes?  They are present in the route table.  Should it not be a supported type, with an empty string for next_hop?
Furthermore, the base implementation states that protocol is optional, and infers that leaving it blank would search all route types.

Why would we not want to support connected routes?

I didn't say that. I said that if you request BGP routes, only BGP routes will be returned.

Furthermore, the base implementation states that protocol is optional, and infers that leaving it blank would search all route types.

If napalm-base says that, then we should leave it. It also makes more sense to me and we need to fix JunOS and IOS-XR then accordingly.

If napalm-base says that, then we should leave it. It also makes more sense to me and we need to fix JunOS and IOS-XR then accordingly.

Problem with that is that we are actually lying. We don't support everything. Or are we planning to support any potential IGP plus BGP on any platform? The benefit of forcing to ask for a platform is that if someone asks for EIGRP on IOS we can just say sorry Notimplemented for protocol EIGRP but if we allow None to signal we want everything we might be lying for those running EIGRP/RIPor any other protocol we are not parsing.
I would actually change base and state that protocol is mandatory. Thoughts?

I would actually change base and state that protocol is mandatory. Thoughts?

I personally would prefer the ability to utilize get_route_to() to get the state of the FIB, so being able to query without a protocol is highly desired.  If we want to implement another method to get forwarding state, that would work as well.

Another problem is that napalm_base says that get_route_to "Returns a dictionary of dictionaries containing details of all available routes to a destination" but looks like we are only checking the installed routes. That's is contradictory with the description of the method and the fact that we have a field called current_active and inactive_reason.

If we want to implement another method to get forwarding state, that would work as well

Yes, I think that would make more sense. Either that or modify this one. Let's wait and see what @mirceaulinic thinks as he is the one that came up with this method.

Another problem is that napalm_base says that get_route_to "Returns a dictionary of dictionaries containing details of all available routes to a destination" but looks like we are only checking the installed routes. That's is contradictory with the description of the method and the fact that we have a field called current_active and inactive_reason.

True :) I think we should change the docstring in that case - I we should return all possible installed routes.

If we want to implement another method to get forwarding state, that would work as well

Yes, I agree on this point.

I would actually change base and state that protocol is mandatory. Thoughts?

No - by default I would find easier to be able to retrieve all learned routes to a destination. And also to filter by a protocol if possible, but not make this mandatory.

I personally think we should deprecate this method. I think we should have a method like:
get_forwarding_information_for(ip_net=None)
Which only returns forwarding information; prefix, prefix-length, next-hop, next-hop-interface
And then have protocol specific methos:
get_bgp_route(ip_net=None)
get_ospf_route(ip_net=None)
get_isis_route(ip_net=None)
get_eigrp_route(ip_net=None)
...

And then include more protocol-relevant information on those.
The reason I want to deprecate this one is because it's too overly complex and I think it's lying to the user as it gives a false expectation on the information it might return.

Which only returns forwarding information; prefix, prefix-length, next-hop, next-hop-interface

Are you specifically excluding listing what RIB/protocol sourced the installed route?  Because I still feel that would be relevant information.

The FIB doesn't know/care about it but I guess we could include that info anyway as that might help then checking for more information with another method.
In any case, this was just a generic proposal. If we decide to go down this path we will need a proper discussion on a dedicated issue/PR potentially looking at the OC implementation to see if it makes sense implementing their model.

I personally think we should deprecate this method.

No, I don't agree. I think we should keep them both (at least for a while).

get_forwarding_information_for(ip_net=None)

Can we move this please in napalm-base? I think we'll have many to discuss about that as, again, that's another complex example.

No, I don't agree. I think we should keep them both (at least for a while).

With deprecating I meant raising a DeprecationWarning exception which doesn't really fail your program but prints a noisy Warning that you should stop using it in favor of something else.
https://docs.python.org/2/library/warnings.html#module-warnings


Show INFO message when syncing files between remote / local

When syncing large files between remote / local, the task icon in SoS Notebook does not spinning, nor does it show anything. It was not an issue until just now I wrongfully sync-ed very large files without noticing and depleted one of my disk. It was a bit annoying.
Since syncing can sometimes be a harmful behavior, it would be great to add more INFO prompt to the notebook while rsync is doing its job.

This stage is called "pending" and is executed as a preparation step for submission. It does not have its own dedicated icon because the job might be waiting after the files are synched. I will see what I can do here though.

One way to solve this problem is to replace the timer with a status line (maybe limit to 50 character) and update what is doing from time to time.

There are already info messages, although they were displayed after the files are synced.

Yes exactly. But by that time mess was already made :(

Good enough?

Looks like so! I do not (yet again) have a large file to test sync for now, but it seems exactly what i'd expect. Close it for now until there is an issue in field test soon.


Make communicate users mutually

Hi everybody,
I started the first project of friendlychat, android-start, and all thing will done, now i want to complet this project to make communicate users mutually (user_1 with user_2 only no on group chat like first project).
can you help me, Is there any other google project can do that?
lahssen think you

@lahssen This codelab creates a global "room" where all users can see all messages. However you can adjust it so that you create individual rooms for each conversation. So instead of:

messages

msg_1
...
msg_2
...

You will have:

conversations

conversation_1

msg_1
...
msg_2
...

conversation_2

msg_1
...
msg_2
...

Each conversation would only contain messages from the users that are in that conversation. You can use database rules to enforce these restrictions.


Update testcase

Param Authority will have 1000 stake tokens
6 validators
Each validator will have one token delated to them from Param Authority account

Closing. Stale.


Tyleryasaka/simplify attestations

Checklist:

[x] Code contains relevant tests for the problem you are solving
[x] Ensure all new and existing tests pass
[x] Update any relevant READMEs and docs
[x] Submit to the develop branch instead of master

Description:
Adds support for redirect url handlers and uses those automatically, rather than accepting redirect url as a param.

LGTM!


Fast-forward to upstream master

Prerequisite to https://github.com/transmission/transmission/issues/5609.
Do not click "merge": this should be fast-forwarded to keep the same commit as upstream.

@ckerr

updated to miniupnp/miniupnp:master


Compiling model in callback causes TypeError

System information.

Have I written custom code (as opposed to using a stock example script provided in Keras): yes
TensorFlow version (use command below): 2.8.0 (2.4 too)
Python version: 3.7

Describe the problem.
In a fine-tuning case I would like to do transfer-learning phase first (with fine-tuned layers frozen) and after that,  all layers should be unfrozen. I wrote a callback that unfreeze the layers after few epochs. Unfortunately, after changing the layers' trainable attribute, the model should be recompiled - and the recompilation causes the TypeError (see colab).
I am aware that I can workaround this by compiling and fitting model twice - for both phases separately - but the usage of callback seems more elegant to me.
Standalone code to reproduce the issue.
https://colab.research.google.com/drive/1u6VlH6EIQGXSp7vEIngTasp3v2EE42Wi?usp=sharing
Source code / logs.
TypeError: 'NoneType' object is not callable

@jvishnuvardhan Was able to replicate the issue on colab using TF v2.8.0 and tf-nightly(2.9.0.dev20220314), please find the gist here .Thanks!

@dmirecki
I really like the idea of having a callback for fine-tuning rather than training again.
But is the correct way to do is to recompile the model or to set the value of layer.trainable as False like here. Although this also raises an error AttributeError: 'bool' object has no attribute 'dtype'

interesting bug - thanks for the report.

Is compiling the model the right way to do it or should we just use backend.set_value like we set lr of optimizers in callbacks.

It seems that adding self.model.make_train_function() indeed fixed the problem! Thanks a lot!

您好，您的邮件已经收到 ！                   杨显洁


Fix cargo-readme test to pull from main.rs instead of lib.rs

Will need to be manually merged, since this fixes a broken test.
Signed-off-by: Mark Bestavros<EMAIL_ADDRESS>

What about cases where there's just a lib.rs?

@MikeCamel We aren't changing the lint configuration of all other repos; just this one, which has special requirements (the doc comment is in main.rs instead of lib.rs). All others will use the default cargo readme behavior, which is to look at lib.rs  first, then main.rs if no lib.rs.

@MikeCamel I've updated the test such that it'll fall back to default behavior if no main.rs is present.


Initcap behaves differently in Spark and in DataFusion (also Comet)

Describe the bug
DataFusion's initcap behaves differently than Spark's. While both do "upper-case the first letter of each word and lowercase others", Spark considers as words anything separated by whitespace (' '), while DataFusion considers anything separated by non-ascii-alphanumeric as words. (DF's code would also fail to uppercase or lowercase non-ascii chars, but that doesn't materialize as a separate issue as it considers them separators already in the first place.)
https://github.com/apache/datafusion-comet/pull/1051 shows the problem by adding two cases to the test, one using a dash and one using non-ascii letters (from Finnish).
== Results ==
!== Correct Answer - 7 ==       == Spark Answer - 7 ==
 struct<initcap(name):string>   struct<initcap(name):string>
 [James Smith]                  [James Smith]
 [James Smith]                  [James Smith]
![James Ähtäri]                 [James äHtäRi]
 [Michael Rose]                 [Michael Rose]
 [Rames Rose]                   [Rames Rose]
![Robert Rose-smith]            [Robert Rose-Smith]
 [Robert Williams]              [Robert Williams]

Steps to reproduce
Call initcap with an input containing non-ascii-alphanumeric non-whitespace characters
Expected behavior
Match Spark
Additional context
No response

DataFusion InitCap doesn't support nonASCII
Tracked in https://github.com/apache/datafusion/issues/13711

![Robert Rose-smith]            [Robert Rose-Smith]

already fixed in latest DF
> select initcap('robert rose-smith');
+------------------------------------+
| initcap(Utf8("robert rose-smith")) |
+------------------------------------+
| Robert Rose-Smith                  |
+------------------------------------+
1 row(s) fetched.
Elapsed 0.005 seconds.

already fixed in latest DF

How is that fixed? :) The diff I showed is exactly the same as the diff you meantion below, no?

Please refer to message update above

Looks like unicode init cap is getting fixed in DF https://github.com/apache/datafusion/pull/13752


Is there optimized kernels for hifi4  conv_2d in xtensa ?

Does it just support Fusion F1 and Hifi5 optimizing， How can I do in hifi4 dsp
Thanks

The Fusion F1 optimizations support Hifi4 as well. We're going to be changing things to make that clearer in the coming weeks in the documentation, code and makefile commands.
For now, you can run the following command (copied from here):
make -f tensorflow/lite/micro/tools/make/Makefile third_party_downloads

make -f tensorflow/lite/micro/tools/make/Makefile \
  TARGET=xtensa \
  TARGET_ARCH=fusion_f1 \
  OPTIMIZED_KERNEL_DIR=xtensa \
  XTENSA_CORE=F1_190305_swupgrade \
  test -j8

and replace XTENSA_CORE=<your core> to use the optimized kernels for your core. (while leaving TARGET_ARCH=fusion_f1).

@advaitjain
Thanks,  my problem has been solved.

Please note that with https://github.com/tensorflow/tflite-micro/pull/305, we use TARGET_ARCH=hifi4 for Fusion F1, Hifi3, Hifi3Z and Hifi4. This matches up with the naming convention for the underlying Xtensa nnlib: https://github.com/foss-xtensa/nnlib-hifi4


Disable external tests on non-docker enabled platform

related with #70
Signed-off-by: Sophia Guo<EMAIL_ADDRESS>

@sophia-guo You can tell if a build is running inside a docker container or not by checking if it has the dockerImage param set in the build configs. See https://github.com/AdoptOpenJDK/ci-jenkins-pipelines#data-fields

@smlambert Thanks for explanation. Updated.


Update reqs-wd-app-guard.md

Removing:
|Windows Defender Exploit Protection settings|The following settings should be configured or verified in the Windows Security app under App & browser control > Exploit protection > Exploit protection settings > System Settings.Control flow guard (CFG) must be set to Use default (On) or Off by default. If set to On by default, Windows Defender Application Guard will not launch.Randomize memory allocations (Bottom-up ASLR) must be set to Use default (On) or Off by default. If set to "On by default", the Vmmem process will have high CPU utilization while a Windows Defender Application Guard window is open.|

This is not part of Application Guard.

Thanks @qrscharmed

Hi, thanks for making updates. Also do you know how I can remove the QRScharmed? That was my pre-Microsoft Git account that I linked to my msft account and now it won’t go away. I’d like to just show my name…
Thanks!
Rona
From: Denise Vangel-MSFT<EMAIL_ADDRESS>Sent: Thursday, April 16, 2020 11:50 AM
To: MicrosoftDocs/windows-itpro-docs<EMAIL_ADDRESS>Cc: Rona Song<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/windows-itpro-docs] Update reqs-wd-app-guard.md (#6490)
Thanks @qrscharmedhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fqrscharmed&data=02|01|rona.song%40microsoft.com|69f8ec55bfbf4639115b08d7e236f931|72f988bf86f141af91ab2d7cd011db47|1|0|637226598050488218&sdata=i8krjdMmrqU0l4rPFQNGhcXrRUQIjf3Wl1EYiNTu%2FO8%3D&reserved=0
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fwindows-itpro-docs%2Fpull%2F6490%23issuecomment-614832561&data=02|01|rona.song%40microsoft.com|69f8ec55bfbf4639115b08d7e236f931|72f988bf86f141af91ab2d7cd011db47|1|0|637226598050488218&sdata=vK40tbotjzPGnKhOaBFRsMMuNGRbPnE9omVbE5gFeCE%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJCRRQJ5MBGZPIG6S2C3GR3RM5HNXANCNFSM4MHJGWPA&data=02|01|rona.song%40microsoft.com|69f8ec55bfbf4639115b08d7e236f931|72f988bf86f141af91ab2d7cd011db47|1|0|637226598050498174&sdata=6kULd5pPsXtpLz4%2BM7ZbIYEMilJijknTisTil3MSENY%3D&reserved=0.

@qrscharmed : There are 2 options you can choose from:

Add your full name to your profile settings, https://github.com/settings/profile . This will only display the full name in the note pop-up and in actual commits contributed by you via Pull Requests.
Rename your account, https://github.com/settings/admin -- Please note: This will break any links to your old comments made under the account name qrscharmed and those old comments will appear to be posted by @ghost (Deleted user) instead.

So I’ve had this done before, but still shows QRScharmed. Should I update this elsewhere?
<EMAIL_ADDRESS>From: Trond B. Krokli<EMAIL_ADDRESS>Sent: Thursday, April 16, 2020 5:29 PM
To: MicrosoftDocs/windows-itpro-docs<EMAIL_ADDRESS>Cc: Rona Song<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/windows-itpro-docs] Update reqs-wd-app-guard.md (#6490)
@qrscharmedhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fqrscharmed&data=02|01|rona.song%40microsoft.com|56113271eb204d92e3f008d7e2664703|72f988bf86f141af91ab2d7cd011db47|1|0|637226801220294052&sdata=BgbZLptXmY5Yy%2Fsp6P93pnAoqOBSuAiiDibk4sZF90s%3D&reserved=0 : There are 2 options you can choose from:

Add your full name to your profile settings, https://github.com/settings/profilehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsettings%2Fprofile&data=02|01|rona.song%40microsoft.com|56113271eb204d92e3f008d7e2664703|72f988bf86f141af91ab2d7cd011db47|1|0|637226801220304009&sdata=NJx%2BvBR9WluI%2Bp4XXHzIbnuB2cyEsukAyjroL%2Bs%2Fxf8%3D&reserved=0 . This will only display the full name in the note pop-up and in actual commits contributed by you via Pull Requests.
Rename your account, https://github.com/settings/adminhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsettings%2Fadmin&data=02|01|rona.song%40microsoft.com|56113271eb204d92e3f008d7e2664703|72f988bf86f141af91ab2d7cd011db47|1|0|637226801220304009&sdata=2V%2F6bgSaQ8FhYrw54Lu1KTIRQ5jE1qR%2BGMYo7%2B2Q5Qk%3D&reserved=0 -- Please note: This will break any links to your old comments made under the account name qrscharmed and those old comments will appear to be posted by @ghosthttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fghost&data=02|01|rona.song%40microsoft.com|56113271eb204d92e3f008d7e2664703|72f988bf86f141af91ab2d7cd011db47|1|0|637226801220313964&sdata=hsPYNBvBo7%2FUJ9B1y9R%2BmEENznEzcJ9oCnQ%2FcOfs600%3D&reserved=0 (Deleted user) instead.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fwindows-itpro-docs%2Fpull%2F6490%23issuecomment-614967126&data=02|01|rona.song%40microsoft.com|56113271eb204d92e3f008d7e2664703|72f988bf86f141af91ab2d7cd011db47|1|0|637226801220313964&sdata=y7f%2FaVmW3X%2FamhbWB9EHxGtI%2Bc%2B4NVvzPzaQWNjdtPg%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJCRRQNCRLQBW7Z26TZVNBLRM6PDPANCNFSM4MHJGWPA&data=02|01|rona.song%40microsoft.com|56113271eb204d92e3f008d7e2664703|72f988bf86f141af91ab2d7cd011db47|1|0|637226801220323924&sdata=L8F%2FqiYfWKP3UCuBKadhaZsm48Ve3Ht9CjLMBp2uqzs%3D&reserved=0.

Well, you have updated your Name field in your profile, which shows when hovering the pointer over your username link.
What you seem to want, is to rename your account name. As mentioned earlier, that is possible, but with some consequences worth considering before doing so.

Please note that your username (= account name ) can not contain spaces.

Well, you have updated your Name field in your profile, which shows when hovering the pointer over your username link.
What you seem to want, is to rename your account name. As mentioned earlier, that is possible, but with some consequences worth considering before doing so.
If it is more important to you to see your account name as either RonaSong, Rona_Song, or ronasong, instead of "qrscharmed", than having access to your existing posts/comments, commit history and so on, you can rename your account here: ►►  https://github.com/settings/admin ◄◄ Warning: this can remove your existing user history ("Here Be Dragons")
Please note that your username (= account name ) can not contain spaces.

See also: https://help.github.com/en/github/setting-up-and-managing-your-github-user-account/changing-your-github-username


HostPath does not enforce quotas

OpenEBS LocalPV HostPath do not enforce quotas
This is a known limitation of OpenEBS LocalPV HostPath, but it's one that seriously limits its usefulness in production.
I'd like for OpenEBS LocalPV HostPath to collect disk usage stats and render pods unscheduleable when they exceed quotas
I know this is not as straight forward as it sounds, but I'd like to make OpenEBS HostPath render pods unschedulable when quotas are exceeded. For existing pods, I'd like to kill them if they are exceeding quotas. For pods to be scheduled, I'd like for this quota to be enforced in advance.
Anything else you would like to add:
I know this is a roadmap feature.

Started experimenting with the feature of creating a sparse file within the hostpath to enforce quota. (inline with what Rawfile Local Pv does).
The PV spec will be changed to:
    local:
      fsType: ""
      path: /var/openebs/local/pvc-4ae3d213-0a93-4b77-9851-d5af29fc07f1/disk.img

The above results in Kubelet rejecting to mount the device with the following error:
MountVolume.NewMounter initialization failed for volume "pvc-4ae3d213-0a93-4b77-9851-d5af29fc07f1" : only directory and block device are supported

Need to figure out if I can easily convert the sparse file into a block device.
The current code is located at: https://github.com/kmova/dynamic-localpv-provisioner/tree/enforce-quota

Thanks! I'll see if I can help in any way. I doubt it, but I'd like to learn how to set up a dev environment for this anyway because we may be using it heavily.

Another related question with hostpath, which filesystem do you plan to use - xfs, ext4 or anything else.
Specifically, would it be ok to use XFS - as there is a possibility of using set quota on top of XFS.

I prefer using XFS only

I use XFS only as well.

This feature also needs to account for

volume resize

This feature needs a good implementation of generating unique xfs project ids per node, that are also unique across clusters in cases the PVs are migrated/merged onto same node.

Hi, I have submitted an application and excited to work on this. 😄

Implementation notes:
Prerequisites:

The base path used by the provisioner should be mounted with XFS project quotas enabled.
Refer: https://fabianlee.org/2020/01/13/linux-using-xfs-project-quotas-to-limit-capacity-within-a-subdirectory/
Example:
sudo mkfs.xfs /dev/sdb
sudo mkdir -p /mnt/xfs_hostpath
sudo mount -o pquota /dev/sdb /mnt/xfs_hostpath

Approach 1:

Provision the volume. Example :  /mnt/xfs_hostpath/pvc-uuid-xyz
Launch another job that sets the project quota on the volume directory.
Example:sudo xfs_quota -x -c 'project -s -p /mnt/xfs_hostpath/vol5m 100' /mnt/xfs_hostpath
sudo xfs_quota -x -c 'limit -p bsoft=5m bhard=5m 100' /mnt/xfs_hostpath

Approach 2:

Provision the volume. Example :  /mnt/xfs_hostpath/pvc-uuid-xyz
Run a daemon set that watches for new PVCs created on the node that sets the project quota on the volume directory.
Example:sudo xfs_quota -x -c 'project -s -p /mnt/xfs_hostpath/vol5m 100' /mnt/xfs_hostpath
sudo xfs_quota -x -c 'limit -p bsoft=5m bhard=5m 100' /mnt/xfs_hostpath

In both of the above cases, the solution needs to take care of finding a unique project id to be associated with a given volume.
Also for inspiration on how to set XFS project quota you can check the following:
https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner/blob/master/pkg/volume/quota.go

This feature also needs to account for
* volume resize

@kmova
currently quota pod is running just after helper pod by taking parameters from storage class
for this feature should I also add an standalone quota pod builder which by taking name of pv can resize the limits of quota if user wants to?

@jshin47 Hostpath now supports enforcing quota for xfs fs in OpenEBS 3.0 release. Please check the latest release and if everything seems good to you can you close this issue?

@Ab-hishek Thank you so much! We will test in the next week or so and confirm it works. Thanks again, this will be so helpful!

This feature seems to be working fine. Many users have already started using this feature and it works fine. Therefore closing the issue.

This feature also needs to account for
* volume resize

@kmova currently quota pod is running just after helper pod by taking parameters from storage class for this feature should I also add an standalone quota pod which by taking name of pv can resize the limits of quota if user wants to?

Openebs hostpath now support volume resize?

@Icedroid  No, its not exactly resize. The quotas(supported for xfs file systems only to be precise) limits the amount of storage hostpath volumes can consume instead of using the entire underlying storage.
References:

https://github.com/openebs/dynamic-localpv-provisioner/pull/78
https://github.com/openebs/dynamic-localpv-provisioner/tree/develop/docs/tutorials/hostpath/xfs_quota

Could support be added for BTRFS as well? BTRFS support quotas via subvolumes and has easy volume snapshots.


cast SERVER_PORT env var into int to be used properly

Also error handling if the casting cannot be done, and default to port 8000.

Looks good!


Fix #filePath related warnings

https://github.com/apple/swift-distributed-actors/issues/635 caused new warnings:
warning: parameter 'file' with default argument '#filePath' passed to parameter 'file', whose default argument is '#file'

This is because standard library's assertion and error functions use #file.
Also fix a few other trivial warnings.

Oh, interesting... thank you for following up

@swift-server-bot test this please


Issue running the grunt init on Ghost

This is my full error text:
BuildingBuilding.Build failed.
ENOENT, no such file or directory 'C:\Users\Uday\Documents\GitHub\uvelop.io\core\client\node_modules\ember-cli-
ic-ajax\node_modules'
Error: ENOENT, no such file or directory 'C:\Users\Uday\Documents\GitHub\uvelop.io\core\client\node_modules\emb
er-cli-ic-ajax\node_modules'
at Error (native)
at Object.fs.readdirSync (fs.js:761:18)
at C:\Users\Uday\Documents\GitHub\uvelop.io\core\client\node_modules\broccoli-merge-trees\index.js:32:21
at Array.map (native)
at mergeRelativePath (C:\Users\Uday\Documents\GitHub\uvelop.io\core\client\node_modules\broccoli-merge-tree
s\index.js:30:29)
at C:\Users\Uday\Documents\GitHub\uvelop.io\core\client\node_modules\broccoli-merge-trees\index.js:23:5
at lib$rsvp$$internal$$tryCatch (C:\Users\Uday\Documents\GitHub\uvelop.io\core\client\node_modules\rsvp\dis
t\rsvp.js:489:16)
at lib$rsvp$$internal$$invokeCallback (C:\Users\Uday\Documents\GitHub\uvelop.io\core\client\node_modules\rs
vp\dist\rsvp.js:501:17)
at lib$rsvp$$internal$$publish (C:\Users\Uday\Documents\GitHub\uvelop.io\core\client\node_modules\rsvp\dist
\rsvp.js:472:11)
at lib$rsvp$asap$$flush (C:\Users\Uday\Documents\GitHub\uvelop.io\core\client\node_modules\rsvp\dist\rsvp.j
s:1290:9)
Warning: Command failed: C:\Windows\system32\cmd.exe /s /c "C:\Users\Uday\Documents\GitHub\uvelop.io\core\clien
t\node_modules.bin\ember build"
Use --force to continue.
Aborted due to warnings.
any ideas?

Hi @uvirk, please don't ask for help on GitHub - it is for tracking issues with the software itself. We have a #help channel in our slack team, where you can come and chat real time with our community members. Thanks :+1:

Hello @ErisDS,
thanks for the prompt answer. Truly, as you expressed, it is so much more easier now than before. I am not sure whether this is right place to ask, but I have a follow-up question whether it is possible in Ghost to achieve a URL behavior as for example on qz.com, where there is a unique sequential number before a URL slug and the same content can be reached simply by number in URL path as with full URL. This is great as it eliminates any need to URL shortening services while keeping the branding intact. Just try it at URL bellow to experience what I mean.
http://qz.com/663944/dramatic-drone-footage-of-earthquake-damage-on-japans-kyushu-island/
http://qz.com/663944/
Besides channels, this is something I would love to see implemented in Ghost if it is not yet there.

@JanSichula That is not currently possible. To request a feature for Ghost you can add your idea (or vote on a similar one) over on our wishlist. We use this list to help determine what goes into the roadmap and how to prioritise features. You can also access the wishlist any time from inside the Ghost admin:


Use a different field for the upload field

Need to use a field in the entity which won't be cast by the marshaller. So it'll need to be a field in the data which is different to the one stored in the database.
A few ideas to try

Try just storing the file upload array in the entity using a different name to the saved db field. Then when the data has been saved I can query the entity to process the data. Relates to #46
Might be possible to use an entity virtual field

Then I can populate the actual entity field name in the entity after processing the

This has already been solved by using the data type class to prevent casting. I think that the removal of this solution and the development of a new one, which would include more configuration isn't an ideal situation.

On reflection I think the different field will be more beneficial as it will allow direct comparisons between data allowing the entity to be marked dirty more easily. It will also prevent a string field being compared to an array, which is currently possible.
Such as
$entity->_properties['image'] = ['name' => 'newly-uploaded-image.jpg'];
$entity->_original['image'] = 'Current-Image.jpg';

Ref: https://github.com/cakephp/cakephp/issues/5998

I do not believe that the change will bring benefit. Changing code in the table class versus changing code in the template.


Upgrade to create-react-app v2

CRA v2 is out https://reactjs.org/blog/2018/10/01/create-react-app-v2.html
It has CSS Modules, postcss, import svgs as component, and new babel, webpack and jest, which'll simplify a few things for us.

Just finishing this so I can then solve https://github.com/ipfs-shipyard/ipfs-webui/issues/986.


chore: log priority queues sizes in SUMMARY log

Purpose
Log priority queues sizes in SUMMARY logs to see what is inside tp

prnet cleaned-up successfully


Support descending ordered queries

Queries are returning rows ascending order of timestamp, but generally we prefer recent results to be shown first.

+1

This has been added in 0.15.0:
try:
SELECT * FROM "wikiticker"
WHERE "__time" >= CURRENT_TIMESTAMP - INTERVAL '1' HOUR
ORDER BY __time ASC

and:
SELECT * FROM "wikiticker"
WHERE "__time" >= CURRENT_TIMESTAMP - INTERVAL '1' HOUR
ORDER BY __time DESC

Smash the Explain button to see how it works


How to install this extension in VSCode

Question
Thank you for taking these projects over and maintaining them. How do we install the extension to VSCode, can't find any link. Only found the un-maintained extension.

Hey @oyalhi! 👋
How to install ?
styled-jsx Syntax Highlighting
Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.
ext install Divlo.vscode-styled-jsx-syntax

vscode-styled-jsx-languageserver
Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.
ext install Divlo.vscode-styled-jsx-languageserver

Do you think we should add this section in the README, maybe it wasn't intuitive enough ?

Hey @oyalhi! 👋
How to install ?
styled-jsx Syntax Highlighting
Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.
ext install Divlo.vscode-styled-jsx-syntax

vscode-styled-jsx-languageserver
Launch VS Code Quick Open (⌘+P), paste the following command, and press enter.
ext install Divlo.vscode-styled-jsx-languageserver

Do you think we should add this section in the README, maybe it wasn't intuitive enough ?

It does thank you.

It does thank you.


Missing name for the registry value

On chapter "Configure the TLS Handle Expiry Time on NPS" between step 5 and 6 is something missing?
How should we call the new registry value?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c7a6ed8a-3ca9-0473-0d87-ea0b5ef49c4c
Version Independent ID: 83a515a2-1ac2-717a-a414-4709dce9961c
Content: Manage Certificates Used with NPS
Content Source: WindowsServerDocs/networking/technologies/nps/nps-manage-certificates.md
Product: windows-server-threshold
Technology: networking
GitHub Login: @shortpatti
Microsoft Alias: pashort

Thanks for your dedication to our documentation. If we have been unable to review your issue in a timely manner, we sincerely apologize for the delayed response. We have captured the informtion you have provided for review. We are closing this issue for now, but if you feel that it's still a concern, please respond and let us know. If you determine another possible update to our documentation, please don't hesitate to reach out again. #please-close


Using RDP between clients

Thank you for this script to install WG & PiHole on Oracle. I have previously used another script on Ionos and then on Google cloud to compare performance from Africa.  Then I discovered Oracle and tried to set up a VM using the other script but couldn't get it to work.  I went looking I found this script and now I am able to connect from my clients perfectly. Thank you!
I am however having trouble connecting between clients using RDP. I have tried various configurations mirroring what works on the other Wireguard servers without success.  I am suspecting a difference in iptables but not knowledgeable enough to determine that.
Do you have any suggestions?
Thank you again for the helpful script! Rob
For quick reference, here is wg config from Oracle plus the script that it references which works:
WG CONFIG FILE (ORACLE)
Address = <IP_ADDRESS>/24,fd42:42:42::1/64
.......
PostUp = /etc/wireguard/ipt/start.sh; iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o ens3 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o ens3 -j MASQUERADE
PostDown = /etc/wireguard/ipt/stop.sh; iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o ens3 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o ens3 -j MASQUERADE
iptables -t nat -I POSTROUTING 1 -s <IP_ADDRESS>/24 -o ens3 -j MASQUERADE
ORACLE START.SH
iptables -I INPUT 1 -i wg0 -j ACCEPT
iptables -I FORWARD 1 -i ens3 -o wg0 -j ACCEPT
iptables -I FORWARD 1 -i wg0 -o ens3 -j ACCEPT
iptables -I INPUT 1 -i ens3 -p udp --dport 51515 -j ACCEPT
ip6tables -t nat -I POSTROUTING 1 -s fd42:42:42::1/64 -o ens3 -j MASQUERADE
ip6tables -I INPUT 1 -i wg0 -j ACCEPT
ip6tables -I FORWARD 1 -i ens3 -o wg0 -j ACCEPT
ip6tables -I FORWARD 1 -i wg0 -o ens3 -j ACCEPT
Here is the content of the Google WG server that does allow RDP.
GOOGLE CLOUD WG CONFIG
[Interface]
Address = <IP_ADDRESS>/24,fd42:42:42::1/64
.......
PostUp = iptables -I INPUT -p udp --dport 54209 -j ACCEPT
PostUp = iptables -I FORWARD -i ens4 -o wg0 -j ACCEPT
PostUp = iptables -I FORWARD -i wg0 -j ACCEPT
PostUp = iptables -t nat -A POSTROUTING -o ens4 -j MASQUERADE
PostUp = ip6tables -I FORWARD -i wg0 -j ACCEPT
PostUp = ip6tables -t nat -A POSTROUTING -o ens4 -j MASQUERADE

I assume you need to add this in the rules . Make sure you use the right port no.
PostUp = iptables -I INPUT -p udp --dport 51515 -j ACCEPT
Let me know the result.
Check this for more info.

I don't have vast knowledge in iptables. here is the outcome of the start.sh stored in /etc/wireguard/ipt/
#!/bin/bash

/sbin/iptables -t nat -I POSTROUTING 1 -s <IP_ADDRESS>/24 -o ens3 -j MASQUERADE
/sbin/iptables -I INPUT 1 -i wg0 -j ACCEPT
/sbin/iptables -I FORWARD 1 -i ens3 -o wg0 -j ACCEPT
/sbin/iptables -I FORWARD 1 -i wg0 -o ens3 -j ACCEPT
/sbin/iptables -I INPUT 1 -i ens3 -p udp --dport 51515 -j ACCEPT

/sbin/ip6tables -t nat -I POSTROUTING 1 -s fd42:42:42::1/64 -o ens3 -j MASQUERADE
/sbin/ip6tables -I INPUT 1 -i wg0 -j ACCEPT
/sbin/ip6tables -I FORWARD 1 -i ens3 -o wg0 -j ACCEPT
/sbin/ip6tables -I FORWARD 1 -i wg0 -o ens3 -j ACCEPT

So the line that seems similar is this one in the Orace Script:
/sbin/iptables -I INPUT 1 -i ens3 -p udp --dport 51515 -j ACCEPT
It includes "-i ens"  that is not in the google script.
Remove the "-i ens" and try?

Tried it.    Didn't break VPN connectivity but didn't allow RDP.

You can try using tailscale, they provide easy way to connect the clients. I'm sorry, I couldn't help you.


Consent link in custom Web Api

Could you please explain how I can let to consent to App in my tenant for my customer in order to get Service Principal in his tenant. I see that in this article Azure function endpoint is used and it somehow prompts for consent, but I use my custom Web Api

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ecbbb69a-f2dd-ed42-d887-14a72494000f
Version Independent ID: 7e5c254a-fd12-0fc6-fc7d-61f4ef009226
Content: Consume multi-tenant enterprise APIs secured with Azure AD in SharePoint Framework
Content Source: docs/spfx/use-aadhttpclient-enterpriseapi-multitenant.md
Product: sharepoint
Technology: sharepoint-framework
GitHub Login: @spdevdocs
Microsoft Alias: spdevdocs

To consent using the API, you would typically navigate to the API URL in the browser which would trigger authentication and the consent flow the first time.

This page may offer additional info for you: https://docs.microsoft.com/azure/active-directory/develop/consent-framework


fix(spacelift-trigger): corrects action failure for uninitialized stacks

what

creates uninitialized_stacks array to store stacks that error because they do not exist.
does not include uninitialized_stacks in error_stacks so the action still succeeds

why

If a PR is creating a stack, atmos tries to trigger the stack (which it computes from code) despite the stack not yet being created by a corresponding <tenant>-infrastructure run

1 Success / 1 Non-existent Stack
TRANKIN:gha-spacelift-script tylerrankin$ ./spacelift-trigger.sh
Running spacectl stack preview --id "mgmt-sandbox-infrastructure" --sha "cc10e08ddb4d463191571d90569aa618b8f27abc"
Duration: 0.000 seconds

Running spacectl stack preview --id "not-a-stack" --sha "cc10e08ddb4d463191571d90569aa618b8f27abc"
Duration: 0.000 seconds

The following stacks triggered successfully:

- mgmt-sandbox-infrastructure
Total Duration: 0.000 seconds

The following stacks have not been created. Confirm that related <tenant>-infrastructure stack runs will create these stacks:

- not-a-stack

1 Error / 1 Non-existent Stack
TRANKIN:gha-spacelift-script tylerrankin$ ./spacelift-trigger.sh
Running spacectl stack preview --id "mgmt-sandbox-infrastructure" --sha "5fb5fea358f0b347b6d51ef01778cd2730ebZZZZ"
Duration: 0.000 seconds

Running spacectl stack preview --id "not-a-stack" --sha "5fb5fea358f0b347b6d51ef01778cd2730ebZZZZ"
Duration: 0.000 seconds

No stacks triggered successfully.
Total Duration: 0.000 seconds

The following stacks have not been created. Confirm that related <tenant>-infrastructure stack runs will create these stacks:

- not-a-stack

The following stacks had errors while triggering:

- mgmt-sandbox-infrastructure: 2023/10/26 15:04:02 could not get the commit by SHA (5fb5fea358f0b347b6d51ef01778cd2730ebZZZZ): could not query GitHub API: Variable $sha of type GitObjectID! was provided invalid value

I think we should not even trigger a stack that does not exist. To skip stacks that do not exist you could the following:

Get the list of the current existing stacks and save it to a variable:

EXISTING_STACKS=$(spacectl stack list | cut -d '|' -f 2)

Put this block inside an if statement:

if  [[ $EXISTING_STACKS =~ "$spacelift_stack" ]]; then
  <block code>
fi


Quick access items stop working after keys are invalidated

Preflight Checklist

[X] I have installed the latest version of Storage Explorer.
[X] I have checked existing resources, including the troubleshooting guide and the release notes.
[X] I have searched for similar issues.

Storage Explorer Version
1.26.1
Regression From
No response
Architecture
x64
Storage Explorer Build Number
20221018.5
Platform
Windows
OS Version
Windows 11
Bug Description
Quick access items stop working after keys are invalidated.
After unpinning and creating new quick access items, it is working.
Steps to Reproduce

Create Quick access item for some storage.
Rotate keys for that storage.
Try using the Quick access item - it should fail, specifying permission error.

Actual Experience
Got unauthorized access error.
Expected Experience
No response
Additional Context
No response

@motikadosh I'm a bit confused. You're saying you expected a "permission error". Right? And then you say you got an "unauthorized access error". Yes? Aren't those basically the same thing? You don't have the right permissions anymore so your attempt at access is unathorized?

Closing due to lack of response. If you are still encountering this issue we recommend that you go ahead and open a ticket via the Azure Portal. Alternatively you can open a new issue here.


error: ../../../.pub-cache/hosted/pub.dev/dio-5.4.3/lib/src/form_data.dart:175:54: Error: The argument type 'List' can't be assigned to the parameter type 'Uint8List'.

Package
dio
Version
5.4.3
Operating-System
iOS
Adapter
Default Dio
Output of flutter doctor -v
! Doctor found issues in 1 category.
shashank@Shashanks-MacBook-Air bluebits-mobile % flutter doctor -v
[✓] Flutter (Channel stable, 3.7.10, on macOS 13.6 22G120 darwin-arm64, locale en-IN)
    • Flutter version 3.7.10 on channel stable at /Users/shashank/Documents/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 4b12645012 (1 year ago), 2023-04-03 17:46:48 -0700
    • Engine revision ec975089ac
    • Dart version 2.19.6
    • DevTools version 2.20.1

[!] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/shashank/Library/Android/sdk
    ✗ cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.
    ✗ Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/macos#android-setup for more details.

[✓] Xcode - develop for iOS and macOS (Xcode 14.3.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14E300c
    • CocoaPods version 1.13.0

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)

[✓] Connected device (3 available)
    • iPad (10th generation) (mobile) • 58251097-0265-42AE-A0E5-9A5ECE1091B7 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-16-4 (simulator)
    • macOS (desktop)                 • macos                                • darwin-arm64   • macOS 13.6 22G120 darwin-arm64
    • Chrome (web)                    • chrome                               • web-javascript • Google Chrome 123.0.6312.123

[✓] HTTP Host Availability
    • All required HTTP hosts are available

Dart Version
2.19.6
Steps to Reproduce
Build app
Get this error
Expected Result
No Error and build should complete successfully
Actual Result
error: ../../../.pub-cache/hosted/pub.dev/dio-5.4.3/lib/src/form_data.dart:175:54: Error: The argument type 'List' can't be assigned to the parameter type 'Uint8List'.

Waiting for this checks to pass. This just stop my project from running

This is fixed in v5.4.3+1.


Local storage is insecure

Local Storage is insecure and shouldn't be used for storing sensitive information.
Storing authentication info (e.g. oidc.session.privateKey here) in local storage is not acceptable in production environment. The above linked OWASP document suggests alternatives.

Thanks @mrkvon, this indeed has been a concern for some time.
@dmitrizagidulin Does this invalidate the entire idea of PoPtokens? We use PoPtokens to avoid extra requests to the IDP, but the good thing is that those requests can only be made by one origin. With PoPtokens, tokens for any domain can be generated anywhere by anyone who steals local storage. Not having PoPtokens would reduce the attack surface, it seems.

Disclaimer: I don't understand details of OIDC or WebId-OIDC.
Is it not possible to use http-only cookies or some other storage mechanism that can be considered secure?

Is it not possible to use http-only cookies

No, the client needs access to the tokens to make authenticated requests.

or some other storage mechanism

That might be possible, but OWASP is very liberal in what they consider threats, as it includes cross-site scripting (which lets you get away with more or less everything). Having short-lived tokens is important.

@RubenVerborgh thank you for explaining!
If I understand well: After the user gets authenticated with the identity provider, they receive some kind of token which they're supposed to use to authenticate themselves with various solid pods. They need to use JavaScript for this.
Indeed, AFAIK, using 3rd party scripts to steal or manipulate the local storage is its main security issue (includes XSS, but also just using any 3rd party library). Therefore any storage that is accessible by JavaScript is vulnerable in the same way, isn't it?
Feeling stuck.

Therefore any storage that is accessible by JavaScript is vulnerable in the same way, isn't it?

It depends. I have an idea in the back of my mind, where the local storage is encrypted, and the key to that encryption is only known by a background worker. That background worker can only run on certain domain, so keys cannot be stolen.
Cross-site scripting might then still make unauthorized requests, but that is hard to stop in the first place.

Just saw this issue.
Basically, it's just as @RubenVerborgh said.
There are no authentication methods available in the browser that are not vulnerable to Cross Site Scripting attacks. Local storage (where the PoP session tokens are stored) is vulnerable. But so are cookies. And so is WebID-TLS (browser certificates).
So, as such, PoP tokens are no more vulnerable than any other authentication mechanism.
Basically, if a hostile script is injected into the page (the kind that has access to local storage), it can make any sort of request it wants, regardless of authentication method, since it can get access to the auth client itself.
That said, there are steps we can take to mitigate this threat. But there's basically only a handful of strategies available (several of which are outlined in the Threat: Obtaining Access Tokens section of the OAuth2 Threat and Security Model document:

Solid apps should ensure input sanitization (apply the usual OWASP recommendations to help minimize script injections).
Auth token scope should be limited wherever possible (this is where Origin-based access controls comes in -- there is some work to be done in this area)
Limit token lifetime (although this is inversely proportional to user convenience, unfortunately :) )

@RubenVerborgh

PoPtokens, tokens for any domain can be generated anywhere by anyone who steals local storage. Not having PoPtokens would reduce the attack surface, it seems.

Not quite. PoP tokens have an ID token embedded in them, that has a limited lifetime (it expires when the session expires). And they are only issued to one particular domain (enforced by redirect_uri restrictions). So, basically, PoP tokens have the same attack surface as do regular OAuth2 ID and access tokens.

@RubenVerborgh Thank you for sharing your idea. I still try to understand it. If it works, why don't we just store the sensitive data directly in web worker? What is the difference between keeping the encryption key and the data themselves with the web worker? (I have only just started learning about web workers.)
@dmitrizagidulin When XSS takes place, the attacker can do any actions they defined. However, with http-only cookies the stored keys don't get exposed to them and can't be edited by them. In this sense local storage and http-only cookies are not equal. Is there an issue in my reasoning?
Further thoughts
The http-only cookies can be sent to one domain only. That's why we have hard time using them to authenticate in a distributed system, is that right?
Would the following conversation between client, identity provider and data server be feasible to solve the issue?
Let's assume client has set a http-only cookie from identity provider.

client to server: Hey, I want this and this data! (creates and sends one-time unique identifier)
client to identity provider: Hey, I just asked that server for that and that data! (sends the one-time unique identifier)
identity provider to client: Thanks, I'll remember for a while.
server to identity provider: Hey, do you know about this request? Is it really the authenticated user?
identity provider to server: Yes!
server to client: Here's your data!

If this is diverging too much, is there a better place for the discussion?

@dmitrizagidulin

Auth token scope should be limited wherever possible (this is where Origin-based access controls comes in -- there is some work to be done in this area)

And those ID tokens are only issued to one particular domain (enforced by redirect_uri restrictions).

Those restrictions only work in the browser. If an attacker steals localStorage, they can send its contents to a server, which can fake any Origin header, thus removing the domain-based protection. That's why I argue the attack surface is broader: steal one set of tokens, and have access to any server.

Limit token lifetime

That seems one of the most important mechanisms.
Additionally, encrypting localStorage with a key that only can be obtained from the Origin with a secret (HTTP-only) cookie, is something we should explore.

Additionally, encrypting localStorage with a key that only can be obtained from the Origin with a secret (HTTP-only) cookie, is something we should explore.

…but nothing prevents an XSS script to just make another request to the IDP and get another set of tokens, so even if we don't use localStorage, XSS still can do whatever.

why don't we just store the sensitive data directly in web worker?

Because that data is not persisted, so you would have to log in again every time.

However, with http-only cookies the stored keys don't get exposed to them and can't be edited by them.

Yes, but we want the client to make privileged requests to other servers. We don't want to go through the server every time.

The http-only cookies can be sent to one domain only. That's why we have hard time using them to authenticate in a distributed system, is that right?

Yes.

Would the following conversation between client, identity provider and data server be feasible to solve the issue?

Yes, but loads of roundtrips, which is what we want to avoid.
But also, with XSS, anyone can make those requests.

But also, with XSS, anyone can make those requests.

I see. With XSS we have a problem, whether we use centralized or distributed system. It's not a problem to be solved by this library.
So it comes down to:

Can we be as safe with a Solid App as with a Common Centralized App?
If the XSS happens, how bad it gets? (Judged by point 1.)

I'm looking forward to seeing where this goes.

Additionally, encrypting localStorage with a key that only can be obtained from the Origin with a secret (HTTP-only) cookie, is something we should explore.

Yes! I suggest that there is at least an option for all the user's data to be encrypted at rest, using a key shared by everyone who is authorized to access that data.
So, for example if I want to share some data with a group "friends" then that data is encrypted with the "friends" key and the "friends" key is encrypted with the public keys of each of the members of that group. The encrypted keys can either be stored with the data or sent to each member when they are added to the "friends" group.
There are a couple of possible ways to handle removing a person from the list of people authorized to access a resource:

re-encrypt the data with a different key and only encrypt that key with the public keys of members who still belong to the group.
or
continue to check that each person belongs to the list of authorized users, before granting them access to the data - so this encryption is in addition to the existing security, rather than replacing it.

If all of the encryption and decryption is performed in the client, then the server never needs to have access to the unencrypted data.
WebID already has some support for public keys so that's a start...

But so are cookies. And so is WebID-TLS (browser certificates).

Please explain how this extends to WebID-TLS? Ditto WebID-TLS+Delegation.

@dmitrizagidulin @RubenVerborgh ,
Any comment regarding the question I posed?
We do need to keep matter clear regarding WebID-OIDC and other protocols such as WebID-TLS and WebID-TLS+Delegation which do not use Local Storage in the Browser to store credentials.
Keychain for macOS and the Keystore provided by Windows provide safe and secure storage for credentials associated with PKI, that has always been the case.
One can also use secure PEM and PKCS#12 Files to store credentials.

Not from my side; I’m in agreement with you.

@RubenVerborgh ,
Okay we are clear on that then :)
@dmitrizagidulin ,
You made the claim about WebID-TLS, so over to you regarding clarification.

@kidehen My point was to remind that no authentication method, including WebID-TLS, is immune if the user runs a compromised or malicious app (because the app can just make authenticated requests and do whatever it wants with the user's data).

@dmitrizagidulin,
You are making a generic point about vulnerability that's universal right now.
My concern with your earlier comment.
My reading of that comment was that it appeared to be that you were speaking about a specific vulnerability that arises for Single Page Applications (SPAs) that handle authentication via WebID-OIDC (which depends on Local Browser Storage re public and private credentials). That specific vulnerability doesn't apply to either WebID-TLS or WebID-TLS+Delegation due to the fact that neither depends on Local Browser Storage for handling both credentials (public or private).
I don't want WebID-OIDC and WebID-TLS and WebID-TLS+Delegation conflated. Solid offers developers options which is a major benefit to be exploited, IMHO.

Though WebID-TLS doesn't use localStorage and uses credentials held in a presumably more secure keychain/keystore, isn't it still vulnerable to attack by session hijacking? If the creation of a WebID session results in an authentication cookie, couldn't a malicious script still gain access to this cookie from cookie storage? Pass the Cookie session hijacking points out that "this technique bypasses most multi-factor authentication protocols. The reason for this is that the final authentication token that the attacker steals is issued after all factors have been validated." - so the security of the protocol used to establish the session is immaterial.
If the WebID-TLS session used an httpOnly cookie, I gather session hijacking is much less of a threat - From OWASP: HttpOnly: "the majority of XSS attacks target theft of session cookies. A server could help mitigate this issue by setting the HttpOnly flag on a cookie it creates".

The cookie mechanism is a performance optimization that, IMHO, needs to go.


[AuthZ] Revisit Authorize default RequireAuthenticatedUser behavior

Currently its looking for User.Identity.IsAuthenticated, this probably needs to be changed to look for ANY Identity in the ClaimsPrincipal where IsAuthenitcated is true.
cc @blowdart @lodejard

Yup. Worth considering is what we do for claims checks. Do we limit to authenticated identities or not? That's probably what people expect.

I still need to revist (and reimplement) what ActiveAuthenticationTypes means on a policy, when set, we still further filter the user's identities?  We certainly can't have Schemes at this point unless we are going to rebuild the user entirely using only the principals from the policy


Fixed caching autodoc (failed to build). Added link to cache_file.

Added links to the cache_file source documentation. Fixed formatting error which prevented the fuel.utils.cache source docs from building.

@nouiz Here it is

@nouiz The docs build correctly locally, and the link to the autodoc works correctly.


Palindrome-Number.cpp

Added the solution for the leetcode problem number 9 Palindrome Number.

Don't forget to star this repository and share on your social media about this repo for hacktoberfest. Thank you for the contribution.


Remove openaddresses as a query param

Hey, is it possible to make sources as an input param? It limits results only to addresses and in my project I need to search over all items
Thanks!

Yep. Can do. Seems like a great enhancement.

For the record: https://mapzen.com/documentation/search/data-sources/