[CT-2296] [Spike] Remove the distinction between configs and non-configs

copying from internal Slack
At first blush, these feel like desirable UX outcomes for our artifacts / metadata interface:

Flatten all "known" fields (defined & validated in dbt-core) to be top-level in generated artifacts.
Group together related fields (incremental-related, test-related, snapshot-related)
Possibly stick the "unresolved" forms of database/schema/alias in their own nested dictionary
Stick user-provided configs (unknown to dbt-core, can be any data type) into an extra dictionary. That would still be our "anything goes" grab bag.

And for the developer UX:

Support setting all attributes within config and top-level attributes equivalent in yaml files. (I foresee lots of little changes, e.g. we'd need to add doc blocks to model rendering context)
Support a mechanism to explicitly prohibit setting certain attributes in dbt_project.yml, for multiple models at once — e.g. access, description, tests
Make config.get('something') and model['something'] equivalent
We rewrite these docs (again): "Configs, properties, same thing!"

Internal Notion with more background & context: https://www.notion.so/dbtlabs/Why-Configs-0ab08bbf3bb34c84bc70c3e57b4f575e?pvs=4
Example
models:
  - name: my_model
    description: ...
    materialized: ...
    any_field_i_want: ...

{{ config(description = ..., materialized = ..., any_field_i_want = ...) }}

With the addition of frontmatter (#7100):
---
description: ...
materialized: ...
any_field_i_want: ...
---

select ...

Does any_field_i_want imply breaking out of the meta sandbox, or do you just mean any_already_defined_field_i_want?

Does any_field_i_want imply breaking out of the meta sandbox, or do you just mean any_already_defined_field_i_want?

Fair point! Users (and adapter maintainers) can already define whatever custom configs they want, and access them via {{ config.get() }}.
If this is the route we go down, there will be very little functional difference between sticking a totally custom field in the "extra" dict (implicitly), versus putting it in the meta dict (explicitly). We'd either want to:

Preserve both patterns, and document the reasons for doing each. E.g. "You should add custom top-level fields to change the model's runtime behavior in a custom way (custom config). By contrast, meta allows you to decorate it with some extra fields that appear in more-predictable ways in downstream metadata (artifacts & logs)."
Deprecate the ability to pass whatever fields you want into meta... knowing that we'd need to keep around this back-compat for a long long time, and we'd also need to get "adapter-specific" configs actually working again, lest we be recommending that adapter maintainers continue relying on an officially deprecated pattern.

Huh! TIL.
I would lean towards the first option - keep them both but explain what context is appropriate for each.

If we totally removed the config dict, and allowed users to specify whatever (valid) keys they wanted as top-level properties, it would make it more difficult for us to catch typos & provide a helpful warning: https://github.com/dbt-labs/dbt-core/issues/5605#issuecomment-1209317030
But it's already the case that we don't / can't catch a typo for a config entry, e.g.:
config:
  matrialized: table

@gshank and I discussed potentially pulling this as follow-on work, as part of stabilizing artifact (manifest) interfaces. It's still not top priority must-have, but it would be thematic with that effort.


onSelectItem fires when selectAll happens

How I can prevent the function for select one item from happening when select all happens? This has 100s of records and I do not want runs my select item function?

I second this issue. Also I am experiencing issue with event not firing when 'onUnselectAll'.

I am also facing this problem. "onItemSelect" gets fires for all the items in the multi-select when i choose selectAll.  This is an emergency for me, anybody please provide the solution ASAP. Your effort will be appreciated. Thanks

Same here. I went ahead and almost implemented lot of code. I cannot pick another plugin. Help to resolve this issue will be much appreciated.

I have solved this problem by doing some changes in multiselect js file. Find and replace below given function in original multiselect js and then check-
$scope.selectAll = function () {
$scope.deselectAll(false);
$scope.externalEvents.onSelectAll();
                angular.forEach($scope.options, function (value) {
                    $scope.setSelectedItem(value[$scope.settings.idProp], true,true);
                });
            };

$scope.setSelectedItem = function (id, dontRemove, sendEvent) {
var findObj = getFindObj(id);
var finalObj = null;
                if ($scope.settings.externalIdProp === '') {
                    finalObj = _.find($scope.options, findObj);
                } else {
                    finalObj = findObj;
                }

                if ($scope.singleSelection) {
                    clearObject($scope.selectedModel);
                    angular.extend($scope.selectedModel, finalObj);
                    $scope.externalEvents.onItemSelect(finalObj);
                    if ($scope.settings.closeOnSelect) $scope.open = false;

                    return;
                }

                dontRemove = dontRemove || false;

                var exists = _.findIndex($scope.selectedModel, findObj) !== -1;

                if (!dontRemove && exists) {
                    $scope.selectedModel.splice(_.findIndex($scope.selectedModel, findObj), 1);
                    if (sendEvent != true)
                        $scope.externalEvents.onItemDeselect(findObj);
                } else if (!exists && ($scope.settings.selectionLimit === 0 || $scope.selectedModel.length < $scope.settings.selectionLimit)) {
                    $scope.selectedModel.push(finalObj);
                    if (sendEvent != true)
                        $scope.externalEvents.onItemSelect(finalObj);
                }
                if ($scope.settings.closeOnSelect) $scope.open = false;
            };

Thanks a lot. This worked great!

On Feb 22, 2016, at 12:18 AM, gauravg418<EMAIL_ADDRESS>wrote:
I have solved this problem by doing some changes in multiselect js file. Find and replace below given function in original multiselect js and then check-
$scope.selectAll = function () {
$scope.deselectAll(false);
$scope.externalEvents.onSelectAll();
            angular.forEach($scope.options, function (value) {
                $scope.setSelectedItem(value[$scope.settings.idProp], true,true);
            });
        };

$scope.setSelectedItem = function (id, dontRemove, sendEvent) {
var findObj = getFindObj(id);
var finalObj = null;
            if ($scope.settings.externalIdProp === '') {
                finalObj = _.find($scope.options, findObj);
            } else {
                finalObj = findObj;
            }

            if ($scope.singleSelection) {
                clearObject($scope.selectedModel);
                angular.extend($scope.selectedModel, finalObj);
                $scope.externalEvents.onItemSelect(finalObj);
                if ($scope.settings.closeOnSelect) $scope.open = false;

                return;
            }

            dontRemove = dontRemove || false;

            var exists = _.findIndex($scope.selectedModel, findObj) !== -1;

            if (!dontRemove && exists) {
                $scope.selectedModel.splice(_.findIndex($scope.selectedModel, findObj), 1);
                if (sendEvent != true)
                    $scope.externalEvents.onItemDeselect(findObj);
            } else if (!exists && ($scope.settings.selectionLimit === 0 || $scope.selectedModel.length < $scope.settings.selectionLimit)) {
                $scope.selectedModel.push(finalObj);
                if (sendEvent != true)
                    $scope.externalEvents.onItemSelect(finalObj);
            }
            if ($scope.settings.closeOnSelect) $scope.open = false;
        };

—
Reply to this email directly or view it on GitHub https://github.com/dotansimha/angularjs-dropdown-multiselect/issues/172#issuecomment-187018633.

just realized that the selectAll is getting invoked twice. Appreciate your help in advance.

On Feb 22, 2016, at 10:29 PM, admin admin<EMAIL_ADDRESS>wrote:
Thanks a lot. This worked great!

On Feb 22, 2016, at 12:18 AM, gauravg418<EMAIL_ADDRESS><EMAIL_ADDRESS>wrote:
I have solved this problem by doing some changes in multiselect js file. Find and replace below given function in original multiselect js and then check-
$scope.selectAll = function () {
$scope.deselectAll(false);
$scope.externalEvents.onSelectAll();
            angular.forEach($scope.options, function (value) {
                $scope.setSelectedItem(value[$scope.settings.idProp], true,true);
            });
        };

$scope.setSelectedItem = function (id, dontRemove, sendEvent) {
var findObj = getFindObj(id);
var finalObj = null;
            if ($scope.settings.externalIdProp === '') {
                finalObj = _.find($scope.options, findObj);
            } else {
                finalObj = findObj;
            }

            if ($scope.singleSelection) {
                clearObject($scope.selectedModel);
                angular.extend($scope.selectedModel, finalObj);
                $scope.externalEvents.onItemSelect(finalObj);
                if ($scope.settings.closeOnSelect) $scope.open = false;

                return;
            }

            dontRemove = dontRemove || false;

            var exists = _.findIndex($scope.selectedModel, findObj) !== -1;

            if (!dontRemove && exists) {
                $scope.selectedModel.splice(_.findIndex($scope.selectedModel, findObj), 1);
                if (sendEvent != true)
                    $scope.externalEvents.onItemDeselect(findObj);
            } else if (!exists && ($scope.settings.selectionLimit === 0 || $scope.selectedModel.length < $scope.settings.selectionLimit)) {
                $scope.selectedModel.push(finalObj);
                if (sendEvent != true)
                    $scope.externalEvents.onItemSelect(finalObj);
            }
            if ($scope.settings.closeOnSelect) $scope.open = false;
        };

—
Reply to this email directly or view it on GitHub https://github.com/dotansimha/angularjs-dropdown-multiselect/issues/172#issuecomment-187018633.

most aggravating 2 hours ever till this.  thank you @gauravg418!

@dotansimha , It seems that there is already a pull request addressing this issue. Would you be so kind as to check it please? It would help us a lot. Thanks :)


Cannot find module 'sinon'

I am trying to install prose locally:
npm install && mkdir -p dist && gulp
Results in:
localhost:prose nathanh$ npm install && mkdir -p dist && gulp
npm WARN EPEERINVALID<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS><2 but none was installed.
npm WARN EPACKAGEJSON<EMAIL_ADDRESS>license should be a valid SPDX license expression
[09:29:15] Using gulpfile ~/Git/tmp/prose/gulpfile.js
[09:29:15] Starting 'templates'...
[09:29:15] Starting 'oauth'...
[09:29:15] Starting 'vendor'...
[09:29:15] Finished 'vendor' after 5.92 ms
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0[09:29:15] Finished 'templates' after 118 ms
100   169  100   169    0     0    192      0 --:--:-- --:--:-- --:--:--   192
[09:29:16] Finished 'oauth' after 924 ms
[09:29:16] Starting 'build-tests'...
[09:29:16] Starting 'build-app'...
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: Cannot find module 'sinon' from '/Users/nathanh/Git/tmp/prose/test'
    at /Users/nathanh/Git/tmp/prose/node_modules/browser-resolve/node_modules/resolve/lib/async.js:46:17
    at process (/Users/nathanh/Git/tmp/prose/node_modules/browser-resolve/node_modules/resolve/lib/async.js:173:43)
    at ondir (/Users/nathanh/Git/tmp/prose/node_modules/browser-resolve/node_modules/resolve/lib/async.js:188:17)
    at load (/Users/nathanh/Git/tmp/prose/node_modules/browser-resolve/node_modules/resolve/lib/async.js:69:43)
    at onex (/Users/nathanh/Git/tmp/prose/node_modules/browser-resolve/node_modules/resolve/lib/async.js:92:31)
    at /Users/nathanh/Git/tmp/prose/node_modules/browser-resolve/node_modules/resolve/lib/async.js:22:47
    at FSReqWrap.oncomplete (fs.js:95:15)

@nasht00 I'm having trouble re-producing this. What version of node.js are you running?

localhost:~ nathanh$ node --version
v0.12.7
localhost:~ nathanh$ npm --version
3.3.4

Should be fixed by #910. Marking this as closed.


Warning option & Template Exceptions

Add --warnings option to command line
Add file name for template exceptions

Coverage increased (+0.01%) to 94.773% when pulling b25fb6349ac210eb0a3084f45b5172c301c96ec9 on jeromesimeon:js-release-0.13.0 into 35c583b016bdd77abe56bd19840dfac6591c69df on accordproject:master.


Lowercase authorization header creates a broken request

Given the following example code:
<?php

use Psr\Http\Message\ResponseInterface;
use Psr\Http\Message\ServerRequestInterface;
use Slim\Factory\AppFactory;

require __DIR__  . '/vendor/autoload.php';

$app = AppFactory::create();

$app->get("/foo", function(ServerRequestInterface $request, ResponseInterface $response, array $arguments) {
    $line = $request->getHeaderLine("Authorization");
    $response->getBody()->write("$line\n");
    return $response;
});

$app->get("/bar", function(ServerRequestInterface $request, ResponseInterface $response, array $arguments) {
    $line = $request->getHeaderLine("X-Something");
    $response->getBody()->write("$line\n");
    return $response;
});

$app->run();

Yields the following results:
$ curl --include --header "Authorization: Basic cm9vdDp0MDBy" localhost:8080/foo
HTTP/1.1 200 OK
Host: localhost:8080
Date: Mon, 01 Mar 2021 10:52:46 GMT
Connection: close
X-Powered-By: PHP/8.0.1
Content-type: text/html; charset=UTF-8

Basic cm9vdDp0MDBy

$ curl --include --header "authorization: Basic cm9vdDp0MDBy" localhost:8080/foo
HTTP/1.1 200 OK
Host: localhost:8080
Date: Mon, 01 Mar 2021 10:53:07 GMT
Connection: close
X-Powered-By: PHP/8.0.1
Content-type: text/html; charset=UTF-8

Basic cm9vdDp0MDBy,Basic cm9vdDp0MDBy

The latter result can be considered broken. Taken from rfc2616:
"Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]. It MUST be possible to combine the multiple header fields into one "field-name: field-value" pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma."
Also author of the spec says it is not valid to specify multiple Authorization fields.
"You can only use multiple header fields when they are defined using list syntax."
I assume the bug is somewhere in slim/psr7 since problem goes away when changing the PSR-7 implementation:
$ composer remove slim/psr7
$ composer require nyholm/psr7
$ composer require nyholm/psr7-server

$ curl --include --header "authorization: Basic cm9vdDp0MDBy" localhost:8080/foo
HTTP/1.1 200 OK
Host: localhost:8080
Date: Mon, 01 Mar 2021 11:08:12 GMT
Connection: close
X-Powered-By: PHP/8.0.1
Content-type: text/html; charset=UTF-8

Basic cm9vdDp0MDBy

See also: https://github.com/tuupola/slim-basic-auth/issues/105, https://github.com/tuupola/slim-basic-auth/issues/89, https://github.com/slimphp/Slim-Psr7/issues/179

We should probably make it case insensitive then? Why do I feel like we've had this debate over and over again 🤔

I think so yes. RFC says header names are case-insensitive.
https://tools.ietf.org/html/rfc7230#section-3.2
"Each header field consists of a case-insensitive field name followed by a colon (":"), optional leading whitespace, the field value, and optional trailing whitespace."

hey, any news on this?


What is the best practices for creating vertices and edges in AGE to avoid duplicates and maximize performance?

Hi folks! I've been working on a Python project that collects 3851 vertices and creates 12507 edges. I want to store this data in AGE, but I'm not quite sure about the best approach.
The data collection is performed using Python's Beautiful Soup, a web scraping library. I then create objects representing vertices, edges, and the graph. Vertices and edges are stored in their respective arrays within the graph class.
For creating vertices in AGE, I use the MERGE clause to check if a vertex with the same label and properties already exists in the graph before creating it:
query = f'''
SELECT * FROM cypher('{graph_name}', $$
MERGE (:{vertex.label} {properties})
$$) AS (n agtype);
'''

This works well for creating each vertex independently.
However, during testing with AGE, I noticed that when vertices are created this way and the following query is executed:
query=f'''
SELECT * FROM cypher('{graph_name}', $$
MERGE (a:{from_v_label} {from_v_properties})-[e:{e_label}]->(b:{to_v_label} {to_v_properties})
$$) AS (e agtype);
'''

If the vertices already exist, it creates duplicates of the vertices.
So, my question is, what is the best way to create vertices and edges without duplicating vertices that may already exist and with good performance? Thank you for taking the time to read my question.

If all vertices already exist before any edges, can't we just use-
MATCH (vertex1) MATCH (vertex2) CREATE (vertex1)-[:e]->(vertex2)

Hi Rafsun! The vertices can be created before and then create the edges after. I thought about this solution before, but wouldn't this be O(n) for creating the vertices and then O(2n) to check if the vertices exist to just then create the edge? I've been thinking that this would take quite a while if we consider the total complexity of this operation as O(n + 2n × m) - considering n to be the number of vertices and m the number of edges. I'm also considering that AGE doesn't use indexes on vertices and edges tables.

I've checked the same queries with Neo4J and it works the same way.

@MatheusFarias03
Do you think the following would work? It should not duplicate the vertices.
MERGE (n:Person {name:'abc'})  -- creates the start vertex if does not exist
MERGE (m:Person {name:'xyz'}) -- creates the end vertex if does not exist
MERGE (n)-[:rel]->(m) -- creates the edge if does not exist
RETURN n


SSL error

I am trying to run this service on NixOS but I am facing the following error with both the lastest tagged and master commit.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: "https://api.cloudflare.com/client/v4/zones", source: hyper::Error(Connect, Ssl(Error { code: ErrorCode(5), cause: Some(Io(Os { code: 32, kind: BrokenPipe, message: "Broken pipe" })) }, X509VerifyResult { code: 0, error: "ok" })) }', src/main.rs:59:10
stack backtrace:
   0:     0x560ea9ef40fc - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hf2ea00d5f3b53d9f
   1:     0x560ea9f230ae - core::fmt::write::ha076a7e65c6b1874
   2:     0x560ea9f034a1 - std::io::Write::write_fmt::h9630bb3b0fe886f5
   3:     0x560ea9f0b575 - std::panicking::default_hook::{{closure}}::h64163197f96398c1
   4:     0x560ea9f0b1d2 - std::panicking::default_hook::h92625161492e6b48
   5:     0x560ea9f0ba91 - std::panicking::rust_panic_with_hook::hfdd7d6bb6788c6c2
   6:     0x560ea9ef4437 - std::panicking::begin_panic_handler::{{closure}}::he81a60a7ef84ad4a
   7:     0x560ea9ef4214 - std::sys_common::backtrace::__rust_end_short_backtrace::h62801bf9780239e6
   8:     0x560ea9f0b742 - rust_begin_unwind
   9:     0x560ea9cd3733 - core::panicking::panic_fmt::hfd221b9f37504b4a
  10:     0x560ea9cd3873 - core::result::unwrap_failed::h7a3045feac200afb
  11:     0x560ea9cf052b - cfdyndns::cloudflare_api::h2c8c2ef748c7b9e7
  12:     0x560ea9cf0cfb - cfdyndns::main::h890b4ef85ecd2099
  13:     0x560ea9cdbfe3 - std::sys_common::backtrace::__rust_begin_short_backtrace::h7b2f491300ee378a
  14:     0x560ea9ceb8a9 - std::rt::lang_start::{{closure}}::hec8c72fe99e36f09
  15:     0x560ea9efc345 - std::rt::lang_start_internal::h09bcc9fffd987f14
  16:     0x560ea9cf2112 - main
  17:     0x7fa9f902924e - __libc_start_call_main
  18:     0x7fa9f9029309 - __libc_start_main_alias_1
  19:     0x560ea9cd3925 - _start
  20:                0x0 - <unknown>

Seems like a SSL error, but not sure since I don't really master rust

Can you try again with the latest release tag?

closing for now as this should be fixed in the latest release


Add new item doesn't support setting an initial stock

Narrow it down to the conversion of Product -> BaseProduct, working on solution now


net/sendfile: adapt sendfile() to support new driver model

Summary
net/sendfile: adapt sendfile() to support new driver model
Bug report:
https://github.com/apache/nuttx/pull/7616#issuecomment-1427088714
Signed-off-by: chao an<EMAIL_ADDRESS>Impact
N/A
Testing
tcp sendfile test

I just tested this on actual hardware. Everything seems fine.
This PR can be merged.


Is it possible to initialize codemirror from a contenteditable?

Because this is awkward: http://jsbin.com/hakipo/edit?html,js,output

This is out of scope for CodeMirror, but yes, it's definitely possible (get HTML, create editor with that HTML as content, I guess).

Thanks marijnh, yes I can get innerHTML, but how do you create the editor from that string, how do you position the editor for exemple as an overlay on top of the element, and how do you retrieve that content before destroying it? many questions sorry but that's exactly what I'd like


Mbyrdantonio9nelson IG

Few clothes to put on and less to say about it, tuck away the garment show yourself on Instagram.


Update overview.md (Typo "MySQL Flexible server"→"MySQL Flexible Serv…

…er")
https://learn.microsoft.com/en-us/azure/mysql/flexible-server/overview
https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/mysql/flexible-server/overview.md
#PingMSFTDocs

@hyoshioka0128 - Thanks for bringing this article to our attention. The content would benefit from an internal copy edit to ensure that it follows Microsoft style and formatting guidelines. We'll close this PR and submit the file for an internal copy edit.


CIAB: Dynamically detect server netmask

What does this PR do?
Since CIAB change to use Dynamic DNS, the netmask will also change.
Therefore, it should be dynamically detect.
Which TC components are affected by this PR?

[ ] Documentation
[ ] Grove
[ ] Traffic Analytics
[ ] Traffic Monitor
[ ] Traffic Ops
[ ] Traffic Ops ORT
[ ] Traffic Portal
[ ] Traffic Router
[ ] Traffic Stats
[ ] Traffic Vault
[x] Other CIAB

What is the best way to verify this PR?
Start CIAB
Check the netmask of db, trafficvault, enroller, or dns.
Their netmask should be the real netmask (<IP_ADDRESS>)
Check all that apply

[ ] This PR includes tests
[ ] This PR includes documentation updates
[ ] This PR includes an update to CHANGELOG.md
[ ] This PR includes all required license headers
[ ] This PR includes a database migration (ensure that migration sequence is correct)
[ ] This PR fixes a serious security flaw. Read more: www.apache.org/security

@JBevillC This PR is updated today so please make sure you are using the latest code.

@Shihta I'm trying to test this, and it looks like the netmask didn't get set properly in my CiaB.
Everything started, and I was able to request the delivery service successfully.
But, the dns, db, enroller, and trafficvault netmasks are empty:
traffic_ops=# select host_name, ip_netmask from server;
    host_name    | ip_netmask
-----------------+-------------
 dns             |
 db              |
 enroller        |
 trafficvault    |
 trafficportal   | <IP_ADDRESS>
 trafficops      | <IP_ADDRESS>
 trafficops-perl | <IP_ADDRESS>
 mid             | <IP_ADDRESS>
 edge            | <IP_ADDRESS>
 trafficrouter   | <IP_ADDRESS>
 trafficmonitor  | <IP_ADDRESS>
(11 rows)

Any ideas?

It's quite weird... I'm curious if your source code is the same as this branch.
There are 2 processes that handle the IP obtaining:

P1: dns, db and enroller use set-to-ips-from-dns.sh to make enrollment. You can check this folder
P2: the others call function to-enroll to make enrollment. TV belongs to P2 after the PR #3287 is merged

If there is an error in this commit, it should not affect TV...
maybe you should rebuild your container image, especially trafficops-perl and trafficvault.
everything seems correct in my environment:

So there are something I need your help:

Could you send me your trafficops-perl log? You can get it by command: docker logs CONTAINER_ID_OF_TO_PERL > LOG_FILE_NAME 2>&1. I want to check if there are some log like this:

setting ips from dns: checking file for dir '/traffic_ops_data/servers' service 'db'
setting ips from dns: trying service file '/traffic_ops_data/servers/020-db_server.json'
setting ips from dns: service file '/traffic_ops_data/servers/020-db_server.json' exists, adding IPs
service "trafficops.infra.ciab.test" not found in dns, count=1, waiting ...
setting ips from dns: checking file for dir '/traffic_ops_data/servers' service 'trafficops'
setting ips from dns: trying service file ''
setting ips from dns: checking file for dir '/traffic_ops_data/servers' service 'trafficops-perl'
setting ips from dns: trying service file ''
setting ips from dns: checking file for dir '/traffic_ops_data/servers' service 'trafficportal'
setting ips from dns: trying service file ''
setting ips from dns: checking file for dir '/traffic_ops_data/servers' service 'trafficmonitor'
setting ips from dns: trying service file ''
setting ips from dns: checking file for dir '/traffic_ops_data/servers' service 'trafficvault'
setting ips from dns: trying service file ''
setting ips from dns: checking file for dir '/traffic_ops_data/servers' service 'trafficrouter'
setting ips from dns: trying service file ''
setting ips from dns: checking file for dir '/traffic_ops_data/servers' service 'enroller'
setting ips from dns: trying service file '/traffic_ops_data/servers/030-enroller_server.json'
setting ips from dns: service file '/traffic_ops_data/servers/030-enroller_server.json' exists, adding IPs
setting ips from dns: checking file for dir '/traffic_ops_data/servers' service 'dns'
setting ips from dns: trying service file '/traffic_ops_data/servers/010-dns_server.json'
setting ips from dns: service file '/traffic_ops_data/servers/010-dns_server.json' exists, adding IPs

The old file 040-trafficvault_server.json should not appear here.

Check my modified command of obtaining netmask can be correctly executed in your environment:

# docker exec -ti CONTAINER_ID_OF_TO_PERL bash
[root@trafficops-perl app]# INTERFACE=$(find /sys/class/net -type l '!' -name lo -exec basename '{}' \;)
[root@trafficops-perl app]# echo $INTERFACE
eth0
[root@trafficops-perl app]# NETMASK=$(route |grep -v default |grep $INTERFACE |awk '{print $3}')
[root@trafficops-perl app]# echo $NETMASK
<IP_ADDRESS>

Thanks!

I see the same problem -- here's the output from your method:   ```
[root@trafficops-perl app]# INTERFACE=$(find /sys/class/net -type l '!' -name lo -exec basename '{}' ;)
[root@trafficops-perl app]# echo $INTERFACE
tunl0 eth0 ip6tnl0
[root@trafficops-perl app]# NETMASK=$(route |grep -v default |grep $INTERFACE |awk '{print $3}')
grep: eth0: No such file or directory
grep: ip6tnl0: No such file or directory

So the above is not eliminating the tunnel interfaces tunl0 and ip6tnl0.   Wondering if that's specific to docker on mac?     I found the following works for my case, and seems should be more general:
INTERFACE=$(ip link | awk '/\<UP\>/ && !/LOOPBACK/<EMAIL_ADDRESS>"", $2); print $2}')
NETMASK=$(route | awk -v INTERFACE=$INTERFACE '$8 ~ INTERFACE && $1 !~ "default"  {print $3}')

the command of getting network mask looks awesome
but I'm confused that is that necessary to detect interface dynamically
docker / k8s seem to use eth0 as the default name in the major network interface
so hard coding it seems a right and simple way
any idea?

I would think that would depend on the image being used in docker,  and the above seems to be a good way to pick up the right one..  At any rate,  I'd like to get this in, but the find method you've used doesn't work for us -- it returns multiple names..

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3960/
Test PASSed.

The 2 commands for INTERFACE and NETMASK have been updated.
I verified this PR with the latest code as well, it seems okay.


"Missing content-type boundary" sending file

I'm currently switching from an AutoRest generated client to an Nswag generated c# client.  Everything seems to be working except for uploading to IFormFile parameters.  There are two problems.
First, here is the swagger:
 / api / Test / UploadFile: {
	post: {
		tags: [
			"Test"
		],
		parameters: [{
				name: "file",
				schema: {
					type: "file"
				}
			}
		],
		requestBody: {
			content: {
				multipart / form - data: {
					schema: {
						type: "object",
						properties: {
							ContentType: {
								type: "string"
							},
							ContentDisposition: {
								type: "string"
							},
							Headers: {
								type: "object",
								additionalProperties: {
									type: "array",
									items: {
										type: "string"
									}
								}
							},
							Length: {
								type: "integer",
								format: "int64"
							},
							Name: {
								type: "string"
							},
							FileName: {
								type: "string"
							}
						}
					},
					encoding: {
						ContentType: {
							style: "form"
						},
						ContentDisposition: {
							style: "form"
						},
						Headers: {
							style: "form"
						},
						Length: {
							style: "form"
						},
						Name: {
							style: "form"
						},
						FileName: {
							style: "form"
						}
					}
				}
			}
		},
		responses: {
			200: {
				description: "Success"
			}
		}
	}
},

The first problem is that the generated client method has both a "body" parameter and a "file" parameter.  I would expect just one... and browsing through the generated code, it looks like the "file" parameter is not even used.
public async System.Threading.Tasks.Task UploadFileAsync(FileParameter file = null, System.IO.Stream body = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken))

Second, when I try to use this method, I get an error from Asp.Net Core 3.0:
Missing content-type boundary.
This API operation worked fine with the AutoRest generated client, so I just need to figure out what I need to do differently to get the NSwag generated client to work here.

I think "multipart/form-data" might be not supported currently...
An option is to make this method protected (ProtectedMethods setting) and implement it yourself within a partial class.

Ref https://github.com/RicoSuter/NSwag/issues/2419

Hi. Realted to the above:
I am using Nswag and generating typescript API definition from an api that has the attribute [OpenApiBodyParameter("application/octet-stream")]
public async Task Upload()
However the generated api code specifies Blob instead of File
upload(body: Blob | null | undefined): Promise<FileResponse | null> {
which generates an error when trying to read the content-type boundary. is there any other attribute I should be using so it generates the api with File instead of Blob?

Hi. Realted to the above:
I am using Nswag and generating typescript API definition from an api that has the attribute [OpenApiBodyParameter("application/octet-stream")]
public async Task Upload()
However the generated api code specifies Blob instead of File
upload(body: Blob | null | undefined): Promise<FileResponse | null> {
which generates an error when trying to read the content-type boundary. is there any other attribute I should be using so it generates the api with File instead of Blob?

@Coops9 How are you reading your data from the asp.net core side?
application/octet-stream is about raw binary, not about files with a file name.

@Coops9 How are you reading your data from the asp.net core side?
application/octet-stream is about raw binary, not about files with a file name.

@jeremyVignelles Hi, thanks for responding. I only chose application/octet-stream as I want to support any filetype being uploaded and that was suggested as the best mime-type to use to mean a generic file (of any type)
I am implementing uploading of files streaming them to a file to a controller action instead of using IFormFile - this gives better support for large files uploaded by multiple people.
Adding the attribute [OpenApiBodyParameter("application/octet-stream")] produces close to what I need except that it generates a signature with a Blob type... I don't know if that's because of the mime-type I used. When I call this method passing in the file I get the "Missing content-type boundary" error. If the method signature was of type File, this error doesn't happen. File extends Blob.
This is the code that throws the error mentioned in this Git issue
using System;
using System.IO;
using Microsoft.Net.Http.Headers;

namespace SampleApp.Utilities
{
    public static class MultipartRequestHelper
    {
        // Content-Type: multipart/form-data; boundary="----WebKitFormBoundarymx2fSWqWSd0OxQqq"
        // The spec at https://tools.ietf.org/html/rfc2046#section-5.1 states that 70 characters is a reasonable limit.
        public static string GetBoundary(MediaTypeHeaderValue contentType, int lengthLimit)
        {
            var boundary = HeaderUtilities.RemoveQuotes(contentType.Boundary).Value;

            if (string.IsNullOrWhiteSpace(boundary))
            {
                throw new InvalidDataException("Missing content-type boundary.");
            }

            if (boundary.Length > lengthLimit)
            {
                throw new InvalidDataException(
                    $"Multipart boundary length limit {lengthLimit} exceeded.");
            }

            return boundary;
        }
...

Here is the article in question and the suggested to code to help illustrate why I am doing what I am doing https://docs.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-3.1#upload-large-files-with-streaming-1
[HttpPost]
[DisableFormValueModelBinding]
[ValidateAntiForgeryToken]
public async Task<IActionResult> UploadDatabase()
{
    if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
    {
        ModelState.AddModelError("File",
            $"The request couldn't be processed (Error 1).");
        // Log error

        return BadRequest(ModelState);
    }

    // Accumulate the form data key-value pairs in the request (formAccumulator).
    var formAccumulator = new KeyValueAccumulator();
    var trustedFileNameForDisplay = string.Empty;
    var untrustedFileNameForStorage = string.Empty;
    var streamedFileContent = new byte[0];

    var boundary = MultipartRequestHelper.GetBoundary(
        MediaTypeHeaderValue.Parse(Request.ContentType),
        _defaultFormOptions.MultipartBoundaryLengthLimit);
    var reader = new MultipartReader(boundary, HttpContext.Request.Body);

    var section = await reader.ReadNextSectionAsync();

    while (section != null)
    {
        var hasContentDispositionHeader =
            ContentDispositionHeaderValue.TryParse(
                section.ContentDisposition, out var contentDisposition);

        if (hasContentDispositionHeader)
        {
            if (MultipartRequestHelper
                .HasFileContentDisposition(contentDisposition))
            {
                untrustedFileNameForStorage = contentDisposition.FileName.Value;
                // Don't trust the file name sent by the client. To display
                // the file name, HTML-encode the value.
                trustedFileNameForDisplay = WebUtility.HtmlEncode(
                        contentDisposition.FileName.Value);

                streamedFileContent =
                    await FileHelpers.ProcessStreamedFile(section, contentDisposition,
                        ModelState, _permittedExtensions, _fileSizeLimit);

                if (!ModelState.IsValid)
                {
                    return BadRequest(ModelState);
                }
            }
            else if (MultipartRequestHelper
                .HasFormDataContentDisposition(contentDisposition))
            {
                // Don't limit the key name length because the
                // multipart headers length limit is already in effect.
                var key = HeaderUtilities
                    .RemoveQuotes(contentDisposition.Name).Value;
                var encoding = GetEncoding(section);

                if (encoding == null)
                {
                    ModelState.AddModelError("File",
                        $"The request couldn't be processed (Error 2).");
                    // Log error

                    return BadRequest(ModelState);
                }

                using (var streamReader = new StreamReader(
                    section.Body,
                    encoding,
                    detectEncodingFromByteOrderMarks: true,
                    bufferSize: 1024,
                    leaveOpen: true))
                {
                    // The value length limit is enforced by
                    // MultipartBodyLengthLimit
                    var value = await streamReader.ReadToEndAsync();

                    if (string.Equals(value, "undefined",
                        StringComparison.OrdinalIgnoreCase))
                    {
                        value = string.Empty;
                    }

                    formAccumulator.Append(key, value);

                    if (formAccumulator.ValueCount >
                        _defaultFormOptions.ValueCountLimit)
                    {
                        // Form key count limit of
                        // _defaultFormOptions.ValueCountLimit
                        // is exceeded.
                        ModelState.AddModelError("File",
                            $"The request couldn't be processed (Error 3).");
                        // Log error

                        return BadRequest(ModelState);
                    }
                }
            }
        }

        // Drain any remaining section body that hasn't been consumed and
        // read the headers for the next section.
        section = await reader.ReadNextSectionAsync();
    }

    // Bind form data to the model
    var formData = new FormData();
    var formValueProvider = new FormValueProvider(
        BindingSource.Form,
        new FormCollection(formAccumulator.GetResults()),
        CultureInfo.CurrentCulture);
    var bindingSuccessful = await TryUpdateModelAsync(formData, prefix: "",
        valueProvider: formValueProvider);

    if (!bindingSuccessful)
    {
        ModelState.AddModelError("File",
            "The request couldn't be processed (Error 5).");
        // Log error

        return BadRequest(ModelState);
    }

    // **WARNING!**
    // In the following example, the file is saved without
    // scanning the file's contents. In most production
    // scenarios, an anti-virus/anti-malware scanner API
    // is used on the file before making the file available
    // for download or for use by other systems.
    // For more information, see the topic that accompanies
    // this sample app.

    var file = new AppFile()
    {
        Content = streamedFileContent,
        UntrustedName = untrustedFileNameForStorage,
        Note = formData.Note,
        Size = streamedFileContent.Length,
        UploadDT = DateTime.UtcNow
    };

    _context.File.Add(file);
    await _context.SaveChangesAsync();

    return Created(nameof(StreamingController), null);
}

I don't know if that's because of the mime-type I used.

I think so. File upload using boundaries is only for multipart/form-data, as described in the code you pasted:
    if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
    {
        ModelState.AddModelError("File",
            $"The request couldn't be processed (Error 1).");
        // Log error

        return BadRequest(ModelState);
    }

If you're using application/octet-stream there's no concept of file name, as there is in multipart/form-data and IFormFile. That's just basically a stream of bytes. Then, on the client side, that maps to a Blob.

@jeremyVignelles thanks again for getting back so quickly!
What mime-type I should be using? So that I can call the code api endpoint above. I only chose application/octect-stream as a default. I want to be able to upload any file types
Regardless of what mime-type I use for
[OpenApiBodyParameter]
the generated api method is as follows:
(generated TS from C# API code)
upload(body: Blob | null | undefined): Promise<FileResponse | null> {
        let url_ = this.baseUrl + "/api/Data/DataImport";
        url_ = url_.replace(/[?&]$/, "");

        const content_ = body;

        let options_ = <RequestInit>{
            body: content_,
            method: "POST",
            headers: {
                "Content-Type": "multipart/form-data",
                "Accept": "application/json"
            }
        };

        return this.http.fetch(url_, options_).then((_response: Response) => {
            return this.processUpload(_response);
        });
    }

the Content-Type is correct.
How can I generate the correct api method? I can't give it FormData because the generated type is Blob
Ideally what I really want to send to the api end-point is something of type FormData:

    const formData = new FormData();

    if (!fileList.length) return;

    // append the files to FormData
    Array
      .from(Array(fileList.length).keys())
      .map(x => {
        formData.append(fieldName, fileList[x], fileList[x].name);
      });

await this.generatedApi.upload(formData)

Hey @Coops9  How did you generate swagger documentation for your method since it doesn't have method parameters?
[HttpPost]
[DisableFormValueModelBinding]
[ValidateAntiForgeryToken]
public async Task<IActionResult> UploadDatabase()

sorry for the off-topic

Did a nyone ever get this to work? Running into the same issue.

WebApi client with a POST endpoint with [FromForm] body
generated OpenApi document via nswag
loaded OpenApi document onto Web Application as a service reference
generated class now has endpoint input param as System.IO.Stream
execute request with a valid stream object and get the described error in this ticket: "failed to read the request form. missing content-type boundary"

This is funny. It worked for me earlier, but not it just stopped. And no FileParameter is generated either. What is happening to NSwag?

In my case, it was the package version. Fixing that solved it.

Workaround https://github.com/RicoSuter/NSwag/issues/2419#issuecomment-818433921


Introduce CoreText control

A Text item with the most commonly used property values.

Update from 13558f1 to 32a11f3:

Added wrap property as suggested by jarolrod


Create fragment for C4B license package creation

What New Or Updated Would You Like To See?
Centralize documentation on creating a package for your C4B license
Why Is It Needed?
Docs for this are spread out across different sections of docs. If we update how we do things, we need to right now worry about editing several docs. If we centralize it into a fragment, we can then just link to that one central doc across all aspects and only need to make one change in the future
Additional Context?

/cc @JPRuskin @Windos, will centralizing this break anything across the Azure Environment?

/cc @JPRuskin @Windos, will centralizing this break anything across the Azure Environment?

@steviecoaster our initial license package creation is embedded somewhere in the deployment of the environment, and our docs are "hey you've got a new license, here's how to update it" so centralizing it won't break anything.
General note, if something changes re naming/versioning the package we'd need to replicate that in the deployment and/or provide context in the Azure Environment specific docs.

This will impact that area, as there is a 4th version segment added to account for the node count the license is valid for as the 4th octet, so we should raise an issue on your end of things.
If you want, I can raise that issue an assign it to myself, although that's likely touching your template json and you might not want me touching that :joy:


Protect password value in API

This might help on the frontend: https://pub.dev/packages/crypt
Currently the request api is passing the password in plain text.
{"email":"stephanie@garden.com","password":"maci"}
The backend is hashing the password and saving the password as a hash in the db.
However, when creating a user the password and confirm password appear in plain text in the api. This should also be corrected.
{"id":"365a0612-483d-4f97-9303-59fd33dad5bb","first_name":"Stephanie","last_name":"Hall","email":"stephanie@garden.com","password_hash":"$2a$10$7GUELZpckaKL4wl0FBBkT.nIyBiHd9xsPF.rd/xwvc2wCeMKcrTku","password":"maci","password_confirmation":"maci","created_at":"2024-05-19T15:16:39.766623-04:00","updated_at":"2024-05-19T15:16:39.766623-04:00"}

may also consider this: https://pub.dev/packages/passwordfield


Feature - plugin to make building http requests easier

@hanqiuzh and I were doing some work and were having to build curl commands to test some API behaviors.  Sometimes it's hard to remember the syntax or flags needed to write curl commands or whatever http client tool you want to use.  If KUI had a way to help build/make http requests like Postman can do and make those requests inside KUI, that might be a cool feature?

this sounds like a neat idea!

@showeimer I just release HTTP Shell v0.0.1 that is build from Kui v9.0.0 that is

A CLI tool that provides developers a modern alternative to http clients for interacting with APIs.

Find it at https://github.com/SoftInstigate/http-shell
@starpitinterested on merging the plugin into kui? let me know...

hey, so sorry for the delay! we'll check it out. it sounds interesting.

@ujibang Great! We are glad to merge http-shell into Kui. Please feel free to PR it. Recently, we've been enhancing Sidecar to be more terminal involved (you can find the proposal here: https://github.com/IBM/kui/issues/6007). We're going to try it out with http-shell. The get command result will then be shown inline, just like the set auth command. We will soon release a live version of this for feedback. We'll really appreciate it if you could give us some feedback on this.

The "inline sidecar" work currently has an open PR: #6008. In addition to the screenshots in the associated issue, we have published a read-only form for perusal to http://next.kui.tools; from there, you can browse some of the notebooks, or, better, you can link directly in to one of the kubernetes example notebooks: http://q.next.kui.tools/index.html?command=replay -c /kui/kubernetes/crud-operations.json
btw, this ability to host a read-only Kui as a static single-page web app, hosted in a plain object storage bucket is something you can also use. i think we're ready to publicize this avenue (it's just a plain build, using the plugin-client-notebook client, and copied to an s3 bucket). you could craft your own notebooks to help publicize your tool. we can chat about that separately.
Let us know what you think of the inline sidecar!

I went through the proposal #6007 about inline sidecars and also looked the screenshot in the PR https://github.com/SoftInstigate/http-shell/pull/1

Well, inline sidecars are definitely the way to go to show response data! This means that different responses will linearly be available in the shell, a great improvement.
The online notebooks are super cool! I will create a notebook and publish it for sure. Notebooks really make documenting some features of our product RESTHeart quite effective. Being RESTHeart an API server, sometimes it takes different steps requests to explain some use cases, and a notebook can definitely visualize the process very well!


Round robin consistent hash not working.

dest_domain=. method=get parent="parent1, parent2, parent3" scheme=https round_robin=consistent_hash

The above is my parent.config file. According to the docs :
- consistent hash of the url so that one parent is chosen for a given url. If a parent is down, the traffic that would go to the down parent is rehashed amongst the remaining parents.

But when i request a content from a child node, the same content request goes out to all the parent servers. To be exact, i am testing with a vod so the content requested are video segments. It's not like different segments are asked from different servers. There are redundant requests as well.
For example, the same segment is requested by the child from all the parent servers.

@amoghyermalkar123 which ATS version?  I'll see if I can duplicate it.

8.1.x Version.
Yes I do get the expected responses. I'm not sure how to turn on debugging "parent_select", can't see any option in records.config, can you let me know how to do that?
That's the part, parents are not even failing, all the parents are serving the content. I thought about parent retries but in that case the request logs should have mentioned failed, but they don't.
Both parents are serving the request i.e. 200 OK. They are even fetching the content from the origin if it's a TCP_MISS 200.

A few more points, firstly, A bit context on the test you should know - i sent out parallel requests for the vod on the same child server.
I expected that only one parent server would be serving up the content which as mentioned above wasn't the case.
Secondly, I want to understand how parent selection is implemented - the algorithm. Basically how it is decided what content should go to a specific parent.

Update, I realized this later that we have an option for debug logs to watch parent selection in action, I'll provide a trimmed down sequence of logs for the aforementioned issue, please help me in understanding the activity.
[ET_NET 5] DEBUG: <HttpTransact.cc:3009 (HandleCacheOpenReadMiss)> (http_trans) [1] [HandleCacheOpenReadMiss] --- MISS
[ET_NET 5] DEBUG: <HttpTransact.cc:3010 (HandleCacheOpenReadMiss)> (http_seq) [1] [HttpTransact::HandleCacheOpenReadMiss] Miss in cache
[ET_NET 5] DEBUG: <HttpTransact.cc:4957 (get_ka_info_from_config)> (http_trans) [1] get_ka_info_from_config, server_info->http_version 65537, check_hostdb 0
[ET_NET 5] DEBUG: <ParentSelection.cc:117 (findParent)> (parent_select) In ParentConfigParams::findParent(): parent_table: 0x556d29afbe90.
[ET_NET 5] DEBUG: <ParentSelection.cc:121 (findParent)> (parent_select) policy.ParentEnable: 1
[ET_NET 5] DEBUG: <ParentSelection.cc:840 (UpdateMatch)> (parent_select) Matched with 0x556d29a2b7a8 parent node from line 1
[ET_NET 5] DEBUG: <ParentConsistentHash.cc:146 (selectParent)> (parent_select) ParentConsistentHash::selectParent(): Using a consistent hash parent selection strategy.
[ET_NET 5] DEBUG: <ParentConsistentHash.cc:210 (selectParent)> (parent_select) Initial parent lookups: 1
[ET_NET 5] DEBUG: <ParentConsistentHash.cc:309 (selectParent)> (parent_select) Additional parent lookups: 1
[ET_NET 5] DEBUG: <ParentConsistentHash.cc:334 (selectParent)> (parent_select) Chosen parent: parent1.com.443.com.443
[ET_NET 5] DEBUG: <ParentSelection.cc:181 (findParent)> (parent_select) PARENT_SPECIFIED
[ET_NET 5] DEBUG: <ParentSelection.cc:182 (findParent)> (parent_select) Result for request-domain was parent parent1.com.443:443

.....

[ET_NET 0] DEBUG: <DNS.cc:1503 (dns_process)> (dns) Got 1 DNS records for [parent1.com]
[ET_NET 0] DEBUG: <DNS.cc:1721 (dns_process)> (dns) received A name = parent1.com
[ET_NET 0] DEBUG: <DNS.cc:1737 (dns_process)> (dns) received A = parent1-ip

DEBUG: <HttpSM.cc:4685 (do_http_server_open)> (http) [1] open connection to parent1.com: parent1-ip:443
DEBUG: <HttpSM.cc:4698 (do_http_server_open)> (http_seq) [HttpSM::do_http_server_open] Sending request to server
DEBUG: <HttpTransact.cc:3385 (handle_response_from_parent)> (http_trans) [1] [1] failed to connect to parent

[ET_NET 5] DEBUG: <HttpTransact.cc:3413 (handle_response_from_parent)> (http_trans) [1] [handle_response_from_parent] 3 per parent attempts exhausted
[ET_NET 5] NOTE: Parent initially marked as down parent1.com:443

[ET_NET 5] DEBUG: <ParentSelection.cc:196 (nextParent)> (parent_select) ParentConfigParams::nextParent(): parent_table: 0x556d29afbe90, result->rec: 0x556d29a2b7a8
[ET_NET 5] DEBUG: <ParentSelection.cc:212 (nextParent)> (parent_select) ParentConfigParams::nextParent(): result->r: 2, tablePtr: 0x556d29afbe90
[ET_NET 5] DEBUG: <ParentSelection.cc:215 (nextParent)> (parent_select) Calling selectParent() from nextParent
[ET_NET 5] DEBUG: <ParentConsistentHash.cc:146 (selectParent)> (parent_select) ParentConsistentHash::selectParent(): Using a consistent hash parent selection strategy.
[ET_NET 5] DEBUG: <ParentConsistentHash.cc:210 (selectParent)> (parent_select) Initial parent lookups: 1
[ET_NET 5] DEBUG: <ParentConsistentHash.cc:309 (selectParent)> (parent_select) Additional parent lookups: 1
[ET_NET 5] DEBUG: <ParentConsistentHash.cc:334 (selectParent)> (parent_select) Chosen parent: parent2.com.443
[ET_NET 5] DEBUG: <ParentSelection.cc:234 (nextParent)> (parent_select) Retry result for request-domain was parent parent2.com:443

The log says that it couldn't connect to the parent server or Response is not valid, while I see TCP_MISS 200 on both the parent servers.

@jrushford appreciate the detailed explanation. I see the behavior on the servers exactly as described above. Closing this issue.
Also if I run into a few problems I hope i'll get the help :)
Thanks.


1351 | Count Negative Numbers in a Sorted Matrix| C++

Title -  Optimised Code  / Improved Solution
What will change -   Adding solution for the code and comment the previous code

[x] Adding New Code
[x] Improving Code
[x] Improving Documentation
[x] Bug Fix

Programming Language
Please add/delete options that are not relevant.

[ ] Python
[x] C++
[ ] Java
[ ] SQL

Self Check

Ask for issue assignment and wait to get assigned to it before making Pull Request.
Add your file in the proper folder
Clean Code and Documentation for better readability
Add Title and Description of the program in the file

:star2: Star it :fork_and_knife:Fork it :handshake: Contribute to it!
Discord server ✉ - https://discord.gg/CXksGdn
Happy Coding,

please assign it to me @juhikushwah

Have a look 👀


When using self-referential extras in pyproject.toml, the package is added to the requirements

If I run
pip-compile --annotation-style=line --no-header --all-extras --output-file=pyproject.lock pyproject.toml

with the following pyproject.toml
[project]
name = "my-pkg"
version = "0.0.1"

[project.optional-dependencies]
tests = ["pytest"]
dev = ["ruff", "my-pkg[tests]"]

The result includes a self-reference
iniconfig==2.0.0          # via pytest
my-pkg[tests] @ file:///home/...  # via my-pkg (pyproject.toml)
packaging==23.2           # via pytest
pluggy==1.3.0             # via pytest
pytest==7.4.2             # via my-pkg, my-pkg (pyproject.toml)
ruff==0.0.292             # via my-pkg (pyproject.toml)

I would expect
iniconfig==2.0.0          # via pytest
packaging==23.2           # via pytest
pluggy==1.3.0             # via pytest
pytest==7.4.2             # via my-pkg,(pyproject.toml)
ruff==0.0.292             # via my-pkg (pyproject.toml)

"my-pkg[tests]" means "this package with the tests extras. my-pkg is then added to extras because you're explicitly requesting it.

I understand what you're actually trying to do, but I'm not sure that it's possible to express.
Personally, I would just include pytest as a dev dependency.

Alternative opinion: I dislike the use of extras for test/dev deps — they are basically your public API so why would you expose them to the non-contributing end-users? I think, it's semantically wrong.
 So why not just use regular requirements as they were designed?

Though, I've seen this @ file:///home/ thing in other places (editable + relative?) and thought it'd be nice to fix it (there's difference between -e . and . for some reason..

The example I gave above was intended to be easy to replicate.
Here is a real world example using pandas pyproject.toml
[project.optional-dependencies]
test = ['hypothesis>=6.46.1', 'pytest>=7.3.2', 'pytest-xdist>=2.2.0', 'pytest-asyncio>=0.17.0']
performance = ['bottleneck>=1.3.4', 'numba>=0.55.2', 'numexpr>=2.8.0']
computation = ['scipy>=1.8.1', 'xarray>=2022.03.0']
fss = ['fsspec>=2022.05.0']
aws = ['s3fs>=2022.05.0']
gcp = ['gcsfs>=2022.05.0', 'pandas-gbq>=0.17.5']
excel = ['odfpy>=1.4.1', 'openpyxl>=3.0.10', 'python-calamine>=0.1.6', 'pyxlsb>=1.0.9', 'xlrd>=2.0.1', 'xlsxwriter>=3.0.3']
parquet = ['pyarrow>=7.0.0']
feather = ['pyarrow>=7.0.0']
hdf5 = [# blosc only available on conda (https://github.com/Blosc/python-blosc/issues/297)
        #'blosc>=1.20.1',
        'tables>=3.7.0']
spss = ['pyreadstat>=1.1.5']
postgresql = ['SQLAlchemy>=1.4.36', 'psycopg2>=2.9.3']
mysql = ['SQLAlchemy>=1.4.36', 'pymysql>=1.0.2']
sql-other = ['SQLAlchemy>=1.4.36']
html = ['beautifulsoup4>=4.11.1', 'html5lib>=1.1', 'lxml>=4.8.0']
xml = ['lxml>=4.8.0']
plot = ['matplotlib>=3.6.1']
output-formatting = ['jinja2>=3.1.2', 'tabulate>=0.8.10']
clipboard = ['PyQt5>=5.15.6', 'qtpy>=2.2.0']
compression = ['zstandard>=0.17.0']
consortium-standard = ['dataframe-api-compat>=0.1.7']
all = ['beautifulsoup4>=4.11.1',
       # blosc only available on conda (https://github.com/Blosc/python-blosc/issues/297)
       #'blosc>=1.21.0',
       'bottleneck>=1.3.4',
       'dataframe-api-compat>=0.1.7',
       'fastparquet>=0.8.1',
       'fsspec>=2022.05.0',
       'gcsfs>=2022.05.0',
       'html5lib>=1.1',
       'hypothesis>=6.46.1',
       'jinja2>=3.1.2',
       'lxml>=4.8.0',
       'matplotlib>=3.6.1',
       'numba>=0.55.2',
       'numexpr>=2.8.0',
       'odfpy>=1.4.1',
       'openpyxl>=3.0.10',
       'pandas-gbq>=0.17.5',
       'psycopg2>=2.9.3',
       'pyarrow>=7.0.0',
       'pymysql>=1.0.2',
       'PyQt5>=5.15.6',
       'pyreadstat>=1.1.5',
       'pytest>=7.3.2',
       'pytest-xdist>=2.2.0',
       'pytest-asyncio>=0.17.0',
       'python-calamine>=0.1.6',
       'pyxlsb>=1.0.9',
       'qtpy>=2.2.0',
       'scipy>=1.8.1',
       's3fs>=2022.05.0',
       'SQLAlchemy>=1.4.36',
       'tables>=3.7.0',
       'tabulate>=0.8.10',
       'xarray>=2022.03.0',
       'xlrd>=2.0.1',
       'xlsxwriter>=3.0.3',
       'zstandard>=0.17.0']

It would be possible to create a group of optional dependencies. For example, sql = ['pandas[postgresql]', 'pandas[mysql]', 'pandas[sql-other]']. This way you don't have to write more than once a dependency constrains (as they did in the all extra) which is prone to errors because you could update one and forget to update the other.
The "my-pkg[tests]" self-reference is because pip expects this syntax (it's supported since
pip 21.1 and they are working on documenting it, link). Self-references are also been discussed here

It would be possible to create a group of optional dependencies. For example, sql = ['pandas[postgresql]', 'pandas[mysql]', 'pandas[sql-other]']. This way you don't have to write more than once a dependency constrains (as they did in the all extra) which is prone to errors because you could update one and forget to update the other.

AFAIK, there's no way to express dependencies this way in pyproject.toml.

The "my-pkg[tests]" self-reference is because pip expects this syntax (it's supported since
pip 21.1 and they are working on documenting it, https://github.com/pypa/pip/issues/11296). Self-references are also been discussed here

pip installs my-pkg too if you use this syntax. You want "my-pkg[tests]" minus my-pkg. I am under the impression that pip does not have a way to express this.

but having a reference to the local filesystem

Pip supports only this way. Maybe add an additional flag to remove annotations for self-referential extras only.

As a (hopefully temporary) workaround, I found applying the following CLI options works
pip-compile --unsafe-package my-pkg --no-allow-unsafe

Like this, the breaking reference of my-pkg to the local filesystem will not be included, which saves me from manually editing the resulting requirements.txt file after running pip-compile.
Note: this command now pins the previously "unsafe" packages (like pip or setuptools), but excluding these packages from being pinned will be deprecated anyways (see #989). Hence the --no-allow-unsafe to future-proof the command.

Another thought in the arena of workarounds (sorry):
For me, requirements files still trump pyproject.toml as a source of truth, so I use the former and some scripting to (re)populate the latter. For the first example in this issue, I'd have dev-requirements.in and tests-requirements.in:
$ <dev-requirements.in
ruff
-r tests-requirements.in

$ <tests-requirements.in
pytest

Then running the pypc function from my pip-tools frontend, inject new values into pyproject.toml. It calls a bit of Python using tomlkit, you can steal the logic:
$ pypc
$ <pyproject.toml

[project]
name = "my-pkg"
version = "0.0.1"

[project.optional-dependencies]
tests = ["pytest"]
dev = ["pytest", "ruff"]

You want "my-pkg[tests]" minus my-pkg. I am under the impression that pip does not have a way to express this.

FYI there's a draft PEP 735 attempting to address this.


{vm-repair} Adding longoptions prefix to Run script Parameters

This PR allows users to specify some parameters that are passed to the script in their entirety, not splitting on =. This expands us in our support of other scenarios, including 1P ones in alar2 script that need the entire parameter passed for 'initiator'.
This checklist is used to make sure that common guidelines for a pull request are followed.
Related command

General Guidelines

[ ] Have you run azdev style <YOUR_EXT> locally? (pip install azdev required)
[ ] Have you run python scripts/ci/test_index.py -q locally? (pip install wheel==0.30.0 required)
[ ] My extension version conforms to the Extension version schema

For new extensions:

[ ] My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify src/index.json.

Thank you for your contribution! We will review the pull request and get back to you soon.


PHP icon

The php icon change to blackberry icon.

The icon file changed icon order, just restart atom and it'll reload the font.


Clarification needed on what large means and what does data collection agents mean.

In the beginning of the specification, I see following sentence: "Open Agent Management Protocol (OpAMP) is a network protocol for remote management of large fleets of data collection Agents." ,  here it is not described what large means(is it hundred or thousands or millions of agents) . Also it is not described what does term "data collection agents" mean. I believe it will be helpful to provide these clarifications.

I believe there are no practical upper bounds for OpAMP-managed fleet size.
As for "data collection agents", OpAMP is thought to be a generic solution here, not limited to a single type of agent. What would you prefer to see here @anuraggujral

@pmm-sumo Thanks for your comment. If protocol can be used for any kind of agent, then I believe we should change "Open Agent Management Protocol (OpAMP) is a network protocol for remote management of large fleets of data collection Agents." to "Open Agent Management Protocol (OpAMP) is a network protocol for remote management of large fleets of agents."

Apologies for the confusion I might have made @anuraggujral but OpAMP's scope is still "data collection agents". Just not one specific kind of agent (like OpenTelemetry Collector) but many of them (potentially it could be used with many existing and future data collection agents)


Regex in general_phpdoc_annotation_remove

Support question
We want to forbid an annotation @group.
We want to use the PHP-CS-Fixer general_phpdoc_annotation_remove rule for this.
'general_phpdoc_annotation_remove' => [
    'annotations' => [
        'group',
    ],
],

/**
 * @group example
 * @GROup example
/*
function example() { ... }

The first annotation get removed, the second not. The rule is not case insensitive.
Can I, how can I use regex for this rule to catch all annotations?

I thikn it would require a new config for the fixer that is saying if casing should be ignored.
would you like to propose such improvement PR?

@keradus oh man, awesome :heart:
Thank you very much.

glad to help! observe the new release in upcoming day and enjoy using this new feature!


Fix issues with setup and README

Fixes # None

Merging!

This pull request marks the first allpha release of einsteinpy


Using a custom Robo Class

In the change log it mentions: RoboFile can be changed to any provided class.
This is not immediately obvious to me.
Is the idea that I simply create my own custom cut down version of the Robo\Runner class.
Removing all the logic around loading the RoboFile.php.
Or is there something I am missing here???

Maybe have a look at #95.

If I use the example from #95 I get:
Call to undefined method Symfony\Component\Console\Application::execute()
If I change execute() to run() I get a working Symfony Console Application, however no output.
I believe the correct usage would be:
\Robo\Config::setOutput(new Symfony\Component\Console\Output\ConsoleOutput());
$runner = new \Robo\Runner();
$app = $runner->createApplication('ClassNameWithTasks');
$app->run($optionalInputGoesHere);

In the end I created my own runner.
see: https://github.com/brad-jones/robo-tasks/blob/master/src/Runner.php

Couldn't we add an -f <file> option to use a different file?

it would also be good having an environment variable, to store the relative path of the RoboFile.php which should be used
export ROBO_FILE=src/Robo

This has been implemented. You can use the --load-from option, or see framework.md