Implement e2e testing to make sure that the simulator API behaves as expected

Currently, unit tests for API service layer and transport layer (handler) are not enough. This is because most packages of these layers are just a wrapper for client-go for now, with less complex logic.
It is useful to have an e2e test to make sure that simulator API is working as expected. The goal of this issue is to be able to run e2e tests on github actions prow job.
/kind feature

Hi @sanposhiho I would like to work on this issue if this is not taken already. I think this would be very interesting and maybe challenging to implement through it's full course. As we are already in works for enabling prow as part of CI, what are your thoughts in enabling e2e testing in prow itself.

Hi. Feel free to take this issue, thanks :)

what are your thoughts in enabling e2e testing using prow itself instead of GH Actions?

Yeah. This e2e test also should be run in prow job instead of GH action. I update the issue description 👍

Hi @sanposhiho, I had few points in mind to implement e2e for kube-scheduler-simulator. The testing should cover all the below features that are provided by simulator to create/delete/edit resources like:

Nodes
Pods
Persistent Volumes, Volume Claims
Storage Classes
Priority Classes

Also I think, we'll have to create a new cluster (or utilize an existing one) so as to run our e2e tests on everytime those are triggered? Any ideas or suggestions you'd like to share, please let me know.

Hello @pravarag.

The testing should cover all the below features that are provided by simulator to create/delete/edit resources like:

We plan to deprecate all APIs for create/delete/edit resources.
https://github.com/kubernetes-sigs/kube-scheduler-simulator/issues/95
So, for now, e2e tests should only cover other APIs in these paths:

/export
/import
/schedulerconfiguration

https://github.com/kubernetes-sigs/kube-scheduler-simulator/blob/master/server/server.go#L48-L83
And, /reset will be added soon. (We can ask @sivchari to add e2e test cases for /reset if he is interested.)
https://github.com/kubernetes-sigs/kube-scheduler-simulator/issues/26

Also I think, we'll have to create a new cluster (or utilize an existing one) so as to run our e2e tests on everytime those are triggered? Any ideas or suggestions you'd like to share, please let me know.

The simulator doesn't require the real cluster.
It starts only the following components when it starts.

pv controller
scheduler
api-server (etcd)

See the entry point of simulator server about it:
https://github.com/kubernetes-sigs/kube-scheduler-simulator/blob/master/simulator.go#L33-L52
So, when each e2e test starts, we can start these components in the same way as we do in the entrypoint.

I know we don't have enough documentation to fully understand the simulator's behavior. 😓 Please feel free to ask me anything you want to know for this.

However, in order to run the e2e test in prow job, we need to figure out how to run etcd in job. 🤔 💭

However, in order to run the e2e test in prow job, we need to figure out how to run etcd in job. thinking thought_balloon

@sanposhiho Thanks for sharing the detailed info, I'm gathering some doubts which I'll share here as well. Also, wanted to check if we'll be requiring to start a new simulator as well before we start any of scheduler, api-server, pv-controller? or these can work independently.
On a different note - as you mentioned we don't need to run e2e tests for create/delete/edit resources like Pods, Nodes, PVs etc. And if I look under /export like here, we have different apis like get request for all the resources, (pods, nodes, PVs, PVCs etc.). So, we want to include these all under e2e testing scenarios right? Kindly let me know if I've missed something here.

@pravarag
(This is a minor point, but, first of all, this simulator is composed of the simulator API(api-server for frontendn), kube-apiserver, pv-controller, and scheduler (scheduler plugins). Therefore, note that "simulator" is a generic term for all of these components.)

I'm gathering some doubts which I'll share here as well. Also, wanted to check if we'll be requiring to start a new simulator as well before we start any of scheduler, api-server, pv-controller? or these can work independently.

Unless we use mocks, we need to start kube-apiserver and scheduler before starting simulator API.

On a different note - as you mentioned we don't need to run e2e tests for create/delete/edit resources like Pods, Nodes, PVs etc. And if I look under /export like here, we have different apis like get request for all the resources, (pods, nodes, PVs, PVCs etc.). So, we want to include these all under e2e testing scenarios right? Kindly let me know if I've missed something here.

Sorry, I didn't catch what you wanted to say exactly.
My earlier comment doesn't mean we don't need to use create/delete/edit resources in e2e tests. But it means we don't need to create e2e tests for all these APIs.
Like /export API, some APIs handles resources in kube-apiserver, so, we need to store resources to kube-apiserver (or get resources from kube-apiserver) in e2e tests to judge if the e2e test is passed or not, and to prepare e2e tests.

Maybe my explanation was not so enough that it was difficult to understand what was needed in this issue. There are many types of e2e tests in this world, and we need to be clear about how we are going to do that.
Here, I would like to describe more specifically what kind of e2e testing I am expecting to see in this issue.

Before e2e test, start the simulator (i.e. the simulator API, kube-apiserver, pv-controller, and scheduler) by calling the same functions as in startSimulator.
In e2e test, we send requests to APIs. (this API means "API server for frontend", doesn't mean kube-apiserver)
Some e2e tests may need to store some resources (Pods, Nodes...etc) before sending request. In that cases, we can request kube-apiserver directly via client returned here in startSimulator.

So, for example, to test /export API, we store some Pods and Nodes by communicating directly with kube-apiserver and send a request to /api/v1/export. Then, if /api/v1/export response contains the Pods and Nodes, that test is marked as a success.

(My expectations are not always correct and the best ones. So, please feel free to let me know if your imagined e2e test is different from this. In that case, let's discuss which way we should go :) Of course, it is possible that both of them will be necessary.)

Thanks for the above suggestions @sanposhiho and I've raised a WIP PR for now to follow updating the e2e tests as I finish those. One suggestion I'll need, as I'm trying to implement like you mentioned any references I can use for including direct api-testing test cases? Currently, I'm trying to start all the components like apiserver, scheduler etc as in here but, while writing test-cases not able to determine how to validate against current responses? Like, do I need to use something like a MUX based router which will get the result from the endpoint like /schedulerconfiguration or /export ? Any ideas around it :)

Oops, I missed the comment.

while writing test-cases not able to determine how to validate against current responses? Like, do I need to use something like a MUX based router which will get the result from the endpoint like /schedulerconfiguration or /export ? Any ideas around it :)

sorry I didn't catch what you mean here. May I ask you to rephrase your point a little more simply? 🙇
In startComponents, you want to start all needed components, right?
And what we need to do in the e2e tests is basically just send requests to api-server in each test case and validate responses.
Also, we can know kube-apiserver's URL in startComponents. So if we need to prepare some resources for some test cases, we can create the resources by requesting to kube-apiserver before running a test case.

Apologies for the delayed response.

sorry I didn't catch what you mean here. May I ask you to rephrase your point a little more simply? bow
In startComponents, you want to start all needed components, right? And what we need to do in the e2e tests is basically just send requests to api-server in each test case and validate responses. Also, we can know kube-apiserver's URL in startComponents. So if we need to prepare some resources for some test cases, we can create the resources by requesting to kube-apiserver before running a test case.

I think you mentioned right on point what I was looking for. I was actually looking for ways to get the validation responses from different components like kube-apiserver you mentioned above. I'll follow the same and make changes to current code. Thanks @sanposhiho

@sanposhiho , apologies for the delay on this PR as this is pending from long time. I'll work this on priority and will try to finish soon.

/area simulator

@pravarag
We need to have a function to initialize simulator first.
By executing almost the same functions as in ↓, we can run all required components (kube-apiserver, scheduler, controllers etc) + simulation server.
https://github.com/kubernetes-sigs/kube-scheduler-simulator/blob/1a5f4be5b0ec75796d56bd5f967e0658062088d2/simulator/simulator.go#L36-L79
Yes, it's what you are going to do in startComponents
https://github.com/kubernetes-sigs/kube-scheduler-simulator/blob/d8e29d254f6da8debe6d2a04bc38393728b09394/test/e2e/e2e_test.go#L19

Then, next step is writing e2e tests for each endpoint.
We have two options for the style here, writing Go tests with table-driven style as other unit tests in this repo, or writing Go tests with ginkgo.
Actually, scheduer's e2e test in kubernetes upstream is written with ginkgo.
https://github.com/kubernetes/kubernetes/blob/master/test/e2e/scheduling/predicates.go
TBH, I'm not familiar with ginkgo, but you can choose what you want to try.
Here, I write a pseudo example for /reset endpoint with table-driven tests.
func Test_Reset(t *testing.T) {
	// run components first
	startComponents()

	// initialize k8s client to communicate with the kube-apiserver in the simulator.
	k8sclient := createK8sClient()

	testcases := []struct {
		name                   string
		preInsertResources     []runtime.Object
		expectedResourcesAfterReset    []runtime.Object
		expectedHTTPStatusCode int
	}{
		{
			name: "/reset endpoint removes all preInsertResources",
			preInsertResources: []runtime.Object{
				&corev1.Pod{
					ObjectMeta: metav1.ObjectMeta{Name: "test-resource"},
				},
			},
			expectedResourcesAfterReset: nil, // all resource removed
			expectedHTTPStatusCode: http.StatusAccepted,
		},
	}
	for _, tt := range testcases {
		t.Run(tt.name, func(t *testing.T) {
			for _, resource := range tt.preInsertResources {
				// create all resources by k8s client
				k8sclient.Create(resource)
			}

			// call the reset endpoint
			resp, err := http.Get(fmt.Sprintf("%s/%s", simulatorServerURL, "reset"))
			if err != nil {
				t.Fatal()
			}
			defer resp.Body.Close()

			if resp.StatusCode != tt.expectedHTTPStatusCode {
				t.Fatal()
			}

			allresources := k8sclient.List()
			for _, r := range allresources {
				found := false
				for _, expected := range tt.expectedResourcesAfterReset {
					if reflect.DeepEqual(r, expected) {
						found = true
						break
					}
				}
				if !found { t.Fatal() } // unexpected resources are still remaining.
			}
		})
	}
}

/reopen

This one is pending from long time, I plan to finish it. In-case, anyone else is interested to work on it, please self-assign :)

/reopen


compatibility with nuxt Generate?

Is apollo-module compatible with generate?  When I run generate, I get the following with Error: ENOENT: no such file or directory, lstat '/Users/Alex/Sites/temp/nuxtlotest/dist/index.html.
yarn generate
yarn run v1.3.2
$ nuxt generate
  nuxt:generate Generating... +0ms
  nuxt:build App root: /Users/Alex/Sites/temp/nuxtlotest +0ms
  nuxt:build Generating /Users/Alex/Sites/temp/nuxtlotest/.nuxt files... +1ms
  nuxt:build Generating files... +14ms
  nuxt:build Generating routes... +5ms
  nuxt:build Building files... +31ms
Build completed in 12.368s

Hash: 8e589c9f082a2d25ae41
Version: webpack 3.8.1
Time: 12374ms
                                  Asset       Size  Chunks             Chunk Names
    pages/index.ba3b400b8c2db044bc9d.js    3.93 kB       0  [emitted]  pages/index
layouts/default.bb1fbffadd7a97ccccdd.js     1.4 kB       1  [emitted]  layouts/default
         common.82aed720f1f0ae6edbcd.js     274 kB       2  [emitted]  common
            app.91b26017724725808d3d.js    26.9 kB       3  [emitted]  app
       manifest.8e589c9f082a2d25ae41.js    1.59 kB       4  [emitted]  manifest
                               LICENSES  634 bytes          [emitted]
 + 8 hidden assets
Hash: ba275b1a3579d8eb0730
Version: webpack 3.8.1
Time: 1284ms
             Asset    Size  Chunks             Chunk Names
server-bundle.json  326 kB          [emitted]
  nuxt:generate Destination folder cleaned +0ms
  nuxt:generate Static & build files copied +34ms
Generating routes
  nuxt:render Rendering url / +0ms
{ Error: ENOENT: no such file or directory, lstat '/Users/Alex/Sites/temp/nuxtlotest/dist/index.html'
  errno: -2,
  code: 'ENOENT',
  syscall: 'lstat',
  path: '/Users/Alex/Sites/temp/nuxtlotest/dist/index.html' }
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

@apmontgomery vue-apollo is used on the component side where you should not see any issues on the rendered app. But: how does your code looks to generate each page? Because the generator runs on the server, you could use graphql-request if you need graphql on the server side to generate your pages - it might be easier to use.

Thanks @dohomi. Then maybe I’ve found a bug. The generate command will not complete when set up as documented. I’ll post a repo with the code shortly.

Here is a repo demonstrating the problem:
https://github.com/apmontgomery/nuxt-apollo-generate-error
After installing as documented, generate command fails to generate an index.html.  Note that I am not using apollo-nuxt to pull any content server side.
Any suggestions would be much appreciated. :)

ok strange. @Atinux @alexchopin do you have any idea what could cause this error? I never used generate command but seems that something breaks when the package is applied

maybe a separate issue, but it also appears that these dependencies should be included in the module:
graphql
graphql-tag
apollo-link-http

this dependency will raise depends how and what kind of implementation you are using. I'd rather keep it to the dev who is writing his own config which might be from usecase to usecase different

Hi @dohomi I just tried npm run generate for the first time on my Nuxt + Apollo / Strapi site. There were no errors noted but also no content was brought in via Apollo. Should this work? An example from a template:
import allActivities from<EMAIL_ADDRESS>
export default {
     data () {
          return {
               activities: []
          }
     },
     apollo: {
          activities: {
               prefetch: true,
               query: allActivities,
               variables() {
                    return {
                         slug: this.slug,
                    };
               },
          },
     },


curl: (60) server certificate verification failed

If Homebrew was updated on Aug 10-11th 2016 and brew update always says Already up-to-date. you need to run: cd "$(brew --repo)" && git fetch && git reset --hard origin/master && brew update.
Please follow the general troubleshooting steps first:

[x] Ran brew update and retried your prior step?
[x] Ran brew doctor, fixed as many issues as possible and retried your prior step?
[x] Confirmed this is problem with Linuxbrew/brew and not specific formulae? If it's a formulae-specific problem please file this issue at https://github.com/Linuxbrew/homebrew-core/issues/new

Bug reports:
I found the problem when installing erlang which error is server certificate verification failed, I already tried solution enabling the

-k

option as uninstall and reinstalled linux brew, but I still got same error, like here
==> Downloading https://www.erlang.org/download/otp_doc_man_19.2.tar.gz
/usr/bin/curl --remote-time --location --user-agent Linuxbrew/1.1.10 (Linux; x86_64 elementary OS 0.4 Loki) curl/7.47.0 --fail https://www.erlang.org/download/otp_doc_man_19.2.tar.gz -C 0 -o /home/satrio/.cache/Homebrew/erlang--man-19.2.tar.gz.incomplete
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
Error: Failed to download resource "erlang--man"
Download failed: https://www.erlang.org/download/otp_doc_man_19.2.tar.gz

any solution for this ?

You can update your out-of-date certs file /etc/ssl/certs/ca-certificates.crt if you have root access.
Try brew install curl

I already try update certs in ubuntu but still same result message
 sudo update-ca-certificates --fresh

Odd. I would have expected that to resolve the issue. You can try downloading the certs manually from https://curl.haxx.se/docs/caextract.html and then setting the SSL_CERT_FILE environment variable. Although I can't now seem to find any documentation for SSL_CERT_FILE.

Did brew install curl work?

yes, there is no problem with brew install curl
and I also copied and replace on
.linuxbrew/etc/openssl/
just wondering, did you have a problem or not when brew install erlang  on your computer ? just feeling like only me have this problem so that you close this issues.

just wondering, did you have a problem or not when brew install erlang on your computer ?

I can verify that this issue affects me as well.

Sorry that I didn't try reproducing your issue earlier. It affects me as well, and it doesn't affect Safari on my system, so it's specific to Linuxbrew.

Here's a PR to update cacert from version 2016-04-20 to 2017-01-18, but it does not unfortunately resolve the issue.
https://github.com/Linuxbrew/homebrew-core/pull/1918

As a workaround you can run brew edit erlang and replace all https with http.

Here's a PR attempting to reproduce the issue on CI: https://github.com/Linuxbrew/homebrew-core/pull/1919

I'm not able to reproduce the issue with brew fetch erlang with Homebrew on a Mac.

Another workaround: brew install --without-docs erlang

Here's another ticket reporting this same issue a year ago:
https://github.com/Linuxbrew/legacy-linuxbrew/issues/981

As a workaround you can run brew edit erlang and replace all https with http

working with this solution
.

I was able to reproduce this issue on a Mac with openssl, but not with the default DarwinSSL.
❯❯❯ brew install curl --with-openssl
❯❯❯ /usr/local/opt/curl/bin/curl https://www.erlang.org/
curl: (60) SSL certificate problem: unable to get local issuer certificate

With DarwinSSL
❯❯ curl -v https://www.erlang.org/ |head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying <IP_ADDRESS>...
* Connected to www.erlang.org (<IP_ADDRESS>) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: www.erlang.org
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5

I find VeriSign Class 3 Public Primary Certification Authority - G5 in cacert-2017-01-18.pem, but I don't find Symantec Class 3 Secure Server CA - G4. Is that a problem? I don't know much about CA certs.

$   brew uninstall --ignore-dependencies curl
 $  brew install curl --with-openssl
Warning: curl: this formula has no --with-openssl option so it will be ignored!
==> Downloading https://curl.haxx.se/download/curl-7.53.1.tar.bz2
Already downloaded: /home/satrio/.cache/Homebrew/curl-7.53.1.tar.bz2
==> ./configure --disable-silent-rules --prefix=/home/satrio/.linuxbrew/Cellar/curl/7.53.1 --with-ssl=/home/satrio/.linuxbrew/opt/openssl --with-ca-bundle=/home/satrio/.linuxbrew/etc/
==> make install
🍺  /home/satrio/.linuxbrew/Cellar/curl/7.53.1: 391 files, 3.0M, built in 46 seconds

there is no problem with curl just when install erlang there had problem

I've reported this issue upstream at https://github.com/openssl/openssl/issues/2833

The URL secure URL https://www.erlang.org/download/otp_doc_man_19.2.tar.gz redirects to the insecure URL http://erlang.org/download/otp_doc_man_19.2.tar.gz so the simple solution is to simply use the insecure URL.
❯❯❯ curl -v https://www.erlang.org/download/otp_doc_man_19.2.tar.gz 2>&1 | grep Location
< Location: http://erlang.org/download/otp_doc_man_19.2.tar.gz

I've changed the https URLs to http in PR https://github.com/Linuxbrew/homebrew-core/pull/1919

Resolved by https://github.com/Linuxbrew/homebrew-core/pull/1919


Merging the product list into one to resolve the issue of products no…

…t being displayed. Adding the display of the current page.
Merging the product list into one to resolve the issue of products not being displayed. Adding the display of the current page.

@cokll ?

@cokll ?

If you don't use the UI shop, you won't be able to see the product list because the game can't send two announcements in a row. Also, if the quantity purchased is too large, it will overflow and cause the purchase to succeed directly.

The issue with the notifications will be fixed if the user chooses to use the new notification system available for everyone as a crossplay mod.
I also checked the overflow and it doesn't seem to be an issue, plus writing 1000 as a magic number doesn't really help. If you are still concerned please open a new PR.


Stuck pending_open channel for days

Background
New to running a LND node, so inexperience may [likely] be a factor.  I've used autopilot to open some channels, and I've since closed them all (or tried to).  For the most part this worked OK, but I still have a zombie pending_open channel for multiple days with no limbo balance.  How do I kill this thing?  NOTE: my channel pending_open balance has also been stuck at this value, even though my wallet is/was empty.  (I've since added more funds to try having an active node again, but I'd like to clear out this zombie channel...)
I see no log details pertaining to the channel, and trying to do a closechannel states:
lncli closechannel 60ebbc8ed216604f74e8bc71f768060aa25d23d6d38aa758a2c121afd4b155f0
[lncli] rpc error: code = Unknown desc = channel not found
My node:
https://1ml.com/node/03f28b34c82c8f437d2c23f4dbcb0e2b9377475e3f154f12e5f249fc9e4df229f0
Current State:
lncli pendingchannels
{
"total_limbo_balance": "0",
"pending_open_channels": [
{
"channel": {
"remote_node_pub": "021779d56f6992431e959991a664d678a2f41054073adb48c5a351d791c9cb0781",
"channel_point": "60ebbc8ed216604f74e8bc71f768060aa25d23d6d38aa758a2c121afd4b155f0:1",
"capacity": "570935",
"local_balance": "570741",
"remote_balance": "0"
},
"confirmation_height": 0,
"commit_fee": "194",
"commit_weight": "600",
"fee_per_kw": "269"
}
],
"pending_closing_channels": [
],
"pending_force_closing_channels": [
],
"waiting_close_channels": [
]
}
Your environment
mainnet

version of lnd: .7 beta
which operating system (uname -a on *Nix): raspian buster lite 7/10/2019
version of btcd, bitcoind, or other backend .18
any other relevant environment details

Steps to reproduce

Fired up an LND with default autopilot values.
Stopped LND, changed default values to increase maxchannels and allocation
Started LND
Left it running overnight
Closed all channels, swept full wallet balance to other address
RESULT: still a zombie pending_open channel days later

Expected behaviour
No stuck pending_open channels?
Actual behaviour
Stuck pending_open channel, and stuck pending_open channel balance even though wallet empty.
Thanks!

Looks like this is just an invalid channel as a result of likely double spending an input. In the future we'll be able to detect this and remove it. For now, you should be able to remove it with the lncli abandonchannel command available in a debug build of lnd. You can obtain one by running make in lnd's source directory, which will create a lnd-debug binary. If the you believe the funds have actually been committed, you'll have to force a wallet rescan.

Looks like this is just an invalid channel as a result of likely double spending an input. In the future we'll be able to detect this and remove it. For now, you should be able to remove it with the lncli abandonchannel command available in a debug build of lnd. You can obtain one by running make in lnd's source directory, which will create a lnd-debug binary. If the you believe the funds have actually been committed, you'll have to force a wallet rescan.

This doesn't work.  After compiling the debug binary and swapping it in, I get:
lncli abandonchannel 60ebbc8ed216604f74e8bc71f768060aa25d23d6d38aa758a2c121afd4b155f0
[lncli] rpc error: code = Unknown desc = channel not found
So abandonchannel can't even find/see it.  Still in the pending_open list though.  I believe I've gotten the funds back, I'd just like to get rid of this thing so it's not throwing off my channel balance and cluttering up my pending list.
Still:
"pending_open_channels": [
    {
        "channel": {
            "remote_node_pub": "021779d56f6992431e959991a664d678a2f41054073adb48c5a351d791c9cb0781",
            "channel_point": "60ebbc8ed216604f74e8bc71f768060aa25d23d6d38aa758a2c121afd4b155f0:1",
            "capacity": "570935",
            "local_balance": "570741",
            "remote_balance": "0"
        },
        "confirmation_height": 0,
        "commit_fee": "194",
        "commit_weight": "600",
        "fee_per_kw": "269"
    },

I'd just like to get rid of this thing so it's not throwing off my channel balance and cluttering up my pending list.

Given that you've already closed all of your channels, you should start a new node and go through the recovery process then.


[incubator-kie-drools#5708] [new-parser] function definition causes a…

…n error
Issue:

https://github.com/apache/incubator-kie-drools/issues/5708

How to replicate CI configuration locally?

Build Chain tool does "simple" maven build(s), the builds are just Maven commands, but because the repositories relates and depends on each other and any change in API or class method could affect several of those repositories there is a need to use build-chain tool to handle cross repository builds and be sure that we always use latest version of the code for each repository.
build-chain tool is a build tool which can be used on command line locally or in Github Actions workflow(s), in case you need to change multiple repositories and send multiple dependent pull requests related with a change you can easily reproduce the same build by executing it on Github hosted environment or locally in your development environment. See local execution details to get more information about it.

How to retest this PR or trigger a specific build:

for pull request and downstream checks

Push a new commit to the PR. An empty commit would be enough.

for a full downstream build

for github actions job: add the label run_fdb

for Jenkins PR check only

If you are an ASF committer for KIE podling, login to Jenkins (https://ci-builds.apache.org/job/KIE/job/drools/), go to the specific PR job, and click on Build Now button.

Before PR
drools-model-codegen
[ERROR] Tests run: 2433, Failures: 419, Errors: 2, Skipped: 9

After PR
drools-model-codegen
[ERROR] Tests run: 2433, Failures: 409, Errors: 2, Skipped: 9

@yurloc @mariofusco @gitgabrio Please review, thanks!

Rebased to resolve conflicts.

Overall fixing progress: There were 396 failed tests.


Xcodeproj?

This project cannot be used in a workspace unless there's an xcodeproj.

Hey! You can generate an xcodeproj by running swift package generate-xcodeproj from the ClangSwift repo.

This issue was fixed in a4ca5a59!


VSCode not detecting Flutter/Dart plugins

Steps to Reproduce

start vscode and open the command pallete, run Flutter:doctor, or Flutter:upgrade
start android-studio with studio.sh and run the hello-world app ( i have it installed in /opt/android-studio with my path updated to /opt/android-studio/bin. )
flutter doctor -v from command line works fine

Logs
flutter_01.log

flutter --verbose create foo caught my device, which wasn't happening before either
4.0s)
[ +105 ms] Wrote 66 files.
[        ]
           All done!
[ +152 ms] java -version
[  +72 ms] executing: /home/t0m/Android/Sdk/tools/bin/sdkmanager
--licenses
[+1316 ms] executing: /home/t0m/Android/Sdk/platform-tools/adb devices
-l
[  +19 ms] Exit code 0 from: /home/t0m/Android/Sdk/platform-tools/adb
devices -l
[   +2 ms] List of devices attached
           ce07171788c3e940027e   device usb:1-1 product:greatqltesq
           model:SM_N950U device:greatqlte transport_id:1
[  +36 ms] /home/t0m/Android/Sdk/platform-tools/adb -s
ce07171788c3e940027e shell getprop
[ +105 ms] ro.hardware = qcom
[   +3 ms] [✓] Flutter is fully installed. (Channel stable, v1.2.1, on
Linux,
               locale en_US.UTF-8)
           [✓] Android toolchain - develop for Android devices is
fully
               installed. (Android SDK version 28.0.3)
           [!] Android Studio is partially installed; more components
are
               available.
           [✓] Connected device is fully installed. (1 available)

           Run "flutter doctor" for information about installing
additional
           components.

[   +2 ms] In order to run your application, type:

             $ cd foo
             $ flutter run

           Your application code is in foo/lib/main.dart.

[   +1 ms] "flutter create" took 7,205ms.
[        ] "flutter create" took 7,205ms.

flutter run output
└──╼ $flutter run
Launching lib/main.dart on SM N950U in debug mode...
Initializing gradle...                                                       ⣻
Oops; flutter has exited unexpectedly.
Sending crash report to Google.
Initializing gradle...                                                Crash report sent (report ID: 6391bcae9d334ef2)
Initializing gradle...                                                        Crash report written to /home/t0m/Documents/flutter-getting-started/foo/flutter_01.log;

┌─[t0m@parrot]─[~/Documents/flutter-getting-started/foo]
└──╼ $flutter doctor -v
[✓] Flutter (Channel stable, v1.2.1, on Linux, locale en_US.UTF-8)
    • Flutter version 1.2.1 at /opt/flutter
    • Framework revision 8661d8aecd (6 weeks ago), 2019-02-14 19:19:53
      -0800
    • Engine revision 3757390fa4
    • Dart version 2.1.2 (build 2.1.2-dev.0.0 0a7dcf17eb)

[✓] Android toolchain - develop for Android devices (Android SDK
    version 28.0.3)
    • Android SDK at /home/t0m/Android/Sdk
    • Android NDK location not configured (optional; useful for native
      profiling support)
    • Platform android-28, build-tools 28.0.3
    • Java binary at: /opt/android-studio/jre/bin/java
    • Java version OpenJDK Runtime Environment (build
      1.8.0_152-release-1248-b01)
    • All Android licenses accepted.

[!] Android Studio
    • Android Studio at /opt/android-studio
    ✗ Flutter plugin not installed; this adds Flutter specific
      functionality.
    ✗ Dart plugin not installed; this adds Dart specific
      functionality.
    • Java version OpenJDK Runtime Environment (build
      1.8.0_152-release-1248-b01)

[✓] Connected device (1 available)
    • SM N950U • ce07171788c3e940027e • android-arm64 • Android 8.0.0
      (API 26)

! Doctor found issues in 1 category.

The Output From my Terminal and Not VSCode is:

┌─[t0m@parrot]─[~]
└──╼ $flutter doctor -v
[✓] Flutter (Channel stable, v1.2.1, on Linux, locale en_US.UTF-8)
    • Flutter version 1.2.1 at /opt/flutter
    • Framework revision 8661d8aecd (6 weeks ago), 2019-02-14 19:19:53 -0800
    • Engine revision 3757390fa4
    • Dart version 2.1.2 (build 2.1.2-dev.0.0 0a7dcf17eb)

[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
    • Android SDK at /home/t0m/Android/Sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-28, build-tools 28.0.3
    • Java binary at: /opt/android-studio/jre/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1248-b01)
    • All Android licenses accepted.

[✓] Android Studio (version 3.3)
    • Android Studio at /opt/android-studio
    • Flutter plugin version 34.0.1
    • Dart plugin version 182.5215
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1248-b01)

[✓] Connected device (1 available)
    • SM N950U • ce07171788c3e940027e • android-arm64 • Android 8.0.0 (API 26)

• No issues found!

This looks related to the version. We look for plugins in "$HOME/.$studioAppName$major.$minor/config/plugins" and I notice that the header for that section is different in your two outputs:

[!] Android Studio
[✓] Android Studio (version 3.3)

The only variable I can see involved is $HOME - have you done anything to change that from the default that maybe VS Code isn't picking up?

Sorry for the late response, I haven't changed anything from the defaults

@t-0-m-1-3 if you run flutter doctor from VS Code's built-in terminal, does it behave the same as your external terminal, or the same as running the Flutter: Run Flutter Doctor command?


V3 Priorities

So this issue marks a development freeze on anything but bugfixes for any cogs written for Red (v2) in here.
This is in preparation for getting everything ready for v3.
Some cogs need rewriting, others had features I knew would be easier to get working with the rewrite of discord.py that didn't get implemented [on v2] as a result.
Here's my current priority list (subject to change pending feedback from people who use my cogs)

BanSync, ServerWhitelist / ServerBlacklist
SuggestionBox **
Autorooms & TempChannels
QuoteTools & ChannelRelays
Announcer & Embedmaker*

* These are both getting many more features.
** Considering making this a more generic form box with customization to the prompts
Anything not listed here is not a priority as of now.
Additionally, the following cogs are not being ported for various reasons:
advstatus (should be core feature in v3)
antimentionspam (should be a core feature in v3)
streamannouncer (This one might change, but realistically, I'd prefer to make the PR to red if needed)
audionotifier (insert Red Audio Meme here; serious note: I don't care enough on this one.)

Feedback about which cogs you'd like to see prioritized or what features you'd like to see are always welcome.

Bansync is v3 ready now.

dd5fcbf removes the 1st item on that list,
Update on remaining things:
Autorooms & TempChannels
Will be implemented soon
QuoteTools & ChannelRelays
Will be looking into a way to better handle a few things in these
This should be handled after AutoRooms, but possibly before TempChannels
Announcer
There's an announcement tool in core red now, considering if this is a sufficient reason to drop mine or if I should instead keep it since the functionalities are different
SuggestionBox
This is on the back burner as I still want to turn this into a more generic form tool, but have not decided the best means of this yet
EmbedMaker
This is getting an overhaul. I'll be adding a way to get a quick embed for people who already know what they want, custom color support, embeds from markdown blocks, and a few other things.

These are currently all waiting on a set of possibly breaking changes in V3 before continuing.

Relay stuff is in and tested. Embed stuff is next unless I get motivated to do something else first instead.

@HelmicNewciv report tool is in latest beta release of V3 (as reports) with added functionality of moderator interaction.
See: red#1280

Think that was the last major roadblock before switching over to V3 for me.  Definitely look forward to being able to report stuff to moderators on more Discords.

Embedmaker is in with quite a bit of additional functionality, including timezone aware timestamps with some level of natural language parsing. November 5th 2018 9pm EST would be correctly parsed
Relays are in
Autorooms, temp channels, and quotetools should also all be ready soon.
For anyone who was looking forward to the last project I sort of started, but never completed on v2 (the advanced role management system), It's a WIP for v3 now, the bare bones of which is in, but not ready for public use yet.

any plans with the role systems? when it comes to the suggestions or announcer cogs, what if there was a formatted style that the owner or admins could create to have different fields for info organized with an embed and the bot would relay it to a channel?
my idea with this is that the user would do a command like [p]review and the bot would send the user a DM much like the embed maker cog does where it asks them the questions/prompts/admin created fields and posts to #game-reviews.
[game title=]
[system=]
[time played=]
[rating/10=]
[review=]

@netflixandbill
The role systems are still a heavy WIP. Current Ideas:

The main role management cog with have stickyrole support baked in. There's also just the standalone stickyroles cog for people who dont need the full suite
Roles can have required roles to be joined, and those can be a list of any, a list of all, or some combination, or a list of exclusions
Roles can be set to be self-removable (but default to not)
Roles can be joined/left using reactions or commands (so far, only emoji is supported, and only for joining)

Regarding the formatted style options for suggestionbox:
I like the idea, and this will be a part of it.
Regarding embeded Announcements:
I won't be re-implementing the logic from embedmaker, but I'll make it available at a later date to users who have both cogs to use the embedmaker to create their announcement.

QuoteTools and RoomTools are both in now. RoomTools is a merge of autorooms and tempchannels with menuing hidden for any disabled. Quotetools's logic has been simplified down to a single command, while retaining most of the prior functionality. More to be added on this at a later date.

RSS & Announcer Soon(TM)
I had planned to work more on the role management system first, but I need to reconsider a bit of what is and isn't in scope for that before continuing with it.

Don't know the scope of the role management, but would it include automated ways to add roles?  I know we have a role that we're currently automatically adding to any users that have been in the server for 7 days, but we'd like to restrict it to those who've posted at least X times per day in non-junk channels for X amount of days to filter out idlers.
"Invisible" roles in the form of manually adding users to a channel/category's permissions would be ideal to keep a truly secret channel category.  Goal is to keep a secret community category where users post selfies and personal information, and keep it private and away from harassers.  Not a perfect solution, but it requires an attacker to at least be really familiar with the community and dedicated enough to stay active for a good while.  Most of the hostile folk we get will make their intentions obvious minutes within joining.
That bit's probably out of the scope of the cog, but I'd like to see that functionality somewhere.

@HelmicNewciv I'm up for discussing details on the invisible roles part as part of the cog, including an API for other cogs to say "give this user this invisible role" and just let the cog handle the updating required.
That isn't a yes or no on doing it (it may come down to rate limits on channel updates, or if not, there will probably be a warning of some sort when using it)
There's also the issue of 6(?) properties of channels not actually being as secret as the client may have you believe. People using things like BetterDiscord* can see the names, topics, ids, position, category, and permissions of every channel, not just the ones they have permissions to. This is done for optimization reasons (I'll try to find the link to a statement on this later)
*Using client mods like BetterDiscord can get you banned, but the types of people you are describing in your use-case may not care.
As for automated ways to add roles... I've been considering it for a while but creating that criteria in a way manageable from discord commands will be difficult to do in a way which satisfies a more general use case. (That doesn't mean I won't try, just that it is a little lower on the priority list than things with clearer paths to a good implementation.)
I'm definitely open to discussing it more either here or via DM on Discord though, as both of these things can be considered in scope if the implementation can be achieved with any level of sanity.

rss is done, work has been started on the suggestionbox cog (overhauled with more stuff) for v3


fix(17265): fix navigation to active adapter's table

This PR fixes the navigation from the workspace or the event log.
When clicking on a link to an adapter in either page, the user is now correctly navigated to the "Active Adapters" tab of the app/protocol-adapters page, with the editor opens whenthe source is a valide instance.
Before

After

Is there anything special to the workspace screenshot?

When clicking on the source on the EventLog page, I get some schema definition errors for HTTP and OPC UA adapter.

When clicking on the source on the EventLog page, I get some schema definition errors for HTTP and OPC UA adapter.

Should be fixed. and nothing special with the two screenshots :-)


there is something wrong when I assigned Caps Lock to Ctrl key in keyboard manger.

Microsoft PowerToys version
0.63.0
Installation method
PowerToys auto-update
Running as admin
Yes
Area(s) with issue?
Keyboard Manager
Steps to reproduce

Assign Caps Lock to Ctrl(left).
Assign Ctrl(left) to any key else.
use Win + v to paste something.
then something wrong with it.

✔ Expected Behavior
paste content to my context
❌ Actual Behavior

Other Software
No response

1

Directly modifying the registry to reassign Caps Lock and Ctrl can help avoid this issue. You can refer to the following link for guidance: https://gist.github.com/joshschmelzle/5e88dabc71014d7427ff01bca3fed33d.

/dup https://github.com/microsoft/PowerToys/issues/23438


Request for a higher order error for bad "type"

Am running v0.15.2 installed inside a Node 13.2.0 alpine Docker container.
Given the following openapi v3 spec:
openapi: 3.0.0
info:
  description: Test types
  version: '0'
  title: Thing service serves things
servers:
  - url: 'https://example.com/test'
paths:
  '/thing':
    get:
      summary: Get thing
      operationId: get_thing
      responses:
        '200':
          description: OK.
          content:
            application/json:
              schema:
                type: object
                properties:
                  thing:
                    type: thing

There is a problem on the last line type: thing.
Current behaviour
When editing this file in the latest Swagger editor at http://editor.swagger.io/ the following error is shown:

And the validator does return an error, which is good. This is the error returned:
errors

  Message :   Property type+format is not well-defined.
  Path    :   paths./things.get.responses.200.content.application/json.schema.properties.things.type
  Line    :   23

There are two problems with this current behaviour of the validator:

In this case, format has not been defined, so it's a confusing error message. My guess is that there is no value we could add for format that would fix this error.

The real error which is that the type of thing is invalid is not being returned, so the validator is not helping the user "get back on track".

Desired behaviour
Would it be possible to generate a new higher order error? In this case something that matches what the swagger editor returns?
errors

  Message :   Property type is not equal to one of the allowed values.
  Path    :   paths./things.get.responses.200.content.application/json.schema.properties.things.type
  Line    :   23

I think that would be helpful because it seems to me that the root cause of the problem is that the type is not one of the allowed values: "array, boolean, integer, number, object, string", rather than there being an issue with format, which is unspecified in this example.
Having a new type error would mean more control available in the config file because type+format errors could be tuned down for files that are using custom format such as "uuid", without stomping on important type errors like this one.

Having an independent error for an invalid type seems like a good idea to me, I will look into this soon!

:tada: This issue has been resolved in version 0.24.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Feature develop merge tf2.4

Intended to be TF 2.4, but downgraded to 2.3 for now to merge with latest develop.

Intended to be TF 2.4, but downgraded to 2.3 for now to merge with latest develop.


Retry apply if conflict is detected

When applying objects to a cluster, there a small chance that a
conflict could arise. Instead of failing instantly, sleep for one
second, and retry up to five times.
Fixes #619
Signed-off-by: bryanl<EMAIL_ADDRESS>

Pull Request Test Coverage Report for Build 1034

110 of 149 (73.83%)  changed or added relevant lines in 6 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.6%) to 71.723%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pkg/cluster/merger.go
5
6
83.33%

pkg/cluster/ksonnet_metadata.go
15
19
78.95%

pkg/cluster/upsert.go
43
49
87.76%

pkg/cluster/apply.go
30
58
51.72%

Files with Coverage Reduction
New Missed Lines
%

pkg/cluster/apply.go
2
43.61%

Totals

Change from base Build 1033:
0.6%

Covered Lines:
10704

Relevant Lines:
14924

💛  - Coveralls


Publications Update

This PR does the following:

Adds a publications page
Adds a results directory with index
Adds an advisory board page
Adds a drop menu to about and results
Some other misc tweaks

Really nice work. Thankyou Scott!!


Should I scale the outputs for SumQE models from 0-1 by 5?

Hi,
I am trying to evaluate my Summ model on your SumQE metrics.
The SumQE paper has tables with values for Q1-5 in the 1-5 range, but the output from the models (with the example in the Readme file in the SumQE repo) is most of the time 0-1 range (once it was 1.03).
Am I missing something or is it just scaling?
Thanks

I believe what is happening is that the final output layer from the original code has no activation function to force the output to be between 0 and 1 (see here), so sometimes the value could be outside of this range. I am not an author of that work, but I would suggest clipping values > 1 to be 1 and < 0 to be 0.

Yep, that makes sense.
Also, to close the loop on my question, I noticed in the training data prep code the human ratings are being divided by 5. So to get back the Likert scale outputs [0-5] we have to multiply the outputs by 5 (after fixing the range of the output to [0, 1] as you said).


Move tests to GitHub Actions

@nicolasstucki is there someone at LAMP or the Scala Center who could tackle this (and other Scala 3 repos under the scala/*) umbrella as part of https://github.com/scala/scala-dev/issues/785 ?

The scala3-* repos are more under the control of LAMP, so I'll let them do it, unless they ask me to. ;-)

I can take care of it


alsa-ucm-conf: apply patch to fix SplitPCM: Device argument may not b…

Description of changes
This patch fixes some audio devices working with alsa-ucm-conf that use SplitPCM.
Upstream fix: https://github.com/alsa-project/alsa-ucm-conf/commit/b68aa52acdd2763fedad5eec0f435fbf43e5ccc6

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Hey @IogaMaster I watched your video about NixOS and I would like to take you up on your offer of a PR review.

LGTM
I've encountered the same problem on one of my devices, and applying this patch fixes it.

Can this be merged, or is there a problem?

Needed to fix issue GoXLR-on-Linux/goxlr-utility/issues/138

Any plans to backport this to 23.11 once it's merged? This issue causes some audio interfaces that were working in 23.05 to not work properly in 23.11.

@errnoh I would love that. AFAIK from the CONTRIBUTING.md only a nixpkgs maintainer can backport. But what I'm wondering is why the "approved by: package-maintainer" label is not set, since I thought @roastiek was the maintainer of alsa-ucm-conf?
I think we have to wait for a nixpkgs committer anyway.

Hi @SuperSandro2000, can you please review this PR.

This PR will cause a whole lot of rebuilds because it's a dependency of alsa-lib, so maybe this should target staging and not master.

@oddlama I tried to follow CONTRIBUTING.md to rebase to staging. Can you check if this is okay?

This should really be one comment otherwise LGTM

@SuperSandro2000 I squashed the commits. Hope it's good now.

please forgive my nix noob question. This fixes and issue I currently have with my goxlr. I was able to get this patch and apply it to the nixpkgs in my nix config but because so much depends on alsa it tried to rebuild like half the packages on my system from source. Is there any way for me to somehow apply to patch to the package but let all the other things on the system build with the old version? https://github.com/arh78750/nixosconfig/blob/d3d6f1ad90dfe5d626f9309a927cb93534af7a80/flake.nix#L19 here is where I'm applying the patch.

@arh78750 Since this is a very simple one-byte patch of a configuration file, I personally just manually edited the file of the broken package in the nix-store by temporarily mounting it rw. In practice you should never ever do that, but in this specific case it's an acceptable workaround because it doesn't affect dependent packages and would otherwise cause too many rebuilds. Just be prepared to redo it when you update your system.

You only need update settings of pipewire or pulseaudio service. Other applications don't read ucm configs directly.
You can change theirs package option with recompiled original againts patches alsa/alsa-ucm-conf.
Or setup ALSA_CONFIG_UCM2 env variable in those services to the new patched config. I think that it worked at leats with pulseaudio two years ago.

Seems unrelated to me. Probably something else was broken on staging.
@ofborg build alsa-ucm-conf
@ofborg eval

@ofborg build alsa-ucm-conf
@ofborg eval

Hey @roastiek, can you review, pls.
Is there anything that needs to be taken care of or can it be merged?

@a-n-n-a-l-e-e thanks for the merge. Is it possible to backport this to 23.11?


[Q] Trouble generating LEFTIST explanations with PYTORCH

Dear Authors and community, thanks for the effort you put in building this library, it's being really useful for me, however i am having trouble trying to generate explanations in the LEFTIST method for pytorch models.
Basically i have this structure:
Shape of X_train: (181177, 30, 1)
Shape of X_test: (45295, 30, 1)
Shape of y_train: (181177,)
Shape of y_test: (45295,)
And the code is the following:
from sklearn.model_selection import train_test_split

# Perform train-test split with stratification
X_train, X_test, y_train, y_test = train_test_split(x_values, y_venda,
                                                    test_size=0.2,
                                                    stratify=y_venda,
                                                    random_state=42)

# Now, X_train, X_test, y_train, y_test contain the split datasets
# X_train and y_train are the training features and labels
# X_test and y_test are the test features and labels

# Print shapes of the resulting datasets
print("Shape of X_train:", X_train.shape)
print("Shape of X_test:", X_test.shape)
print("Shape of y_train:", y_train.shape)
print("Shape of y_test:", y_test.shape)
# Initialize the scaler
scaler_X = MinMaxScaler()
scaler_y = MinMaxScaler()

# Reshape the data to 2D for scaling
X_train_reshaped = X_train.reshape(-1, X_train.shape[1])
X_test_reshaped = X_test.reshape(-1, X_test.shape[1])

# Fit and transform the training data
X_train_scaled = scaler_X.fit_transform(X_train_reshaped)

# Transform the test data
X_test_scaled = scaler_X.transform(X_test_reshaped)

# Reshape back to original shape
X_train_scaled = X_train_scaled.reshape(X_train.shape)
X_test_scaled = X_test_scaled.reshape(X_test.shape)

# Now X_train_scaled, X_test_scaled, y_train_scaled, and y_test_scaled are ready to use
from torch.utils.data import DataLoader, TensorDataset

# Convert arrays to PyTorch tensors
X_train_tensor = torch.tensor(X_train_scaled, dtype=torch.float32)
X_test_tensor = torch.tensor(X_test_scaled, dtype=torch.float32)
y_train_tensor = torch.tensor(y_train, dtype=torch.float32)
y_test_tensor = torch.tensor(y_test, dtype=torch.float32)

X_train_tensor.shape

import numpy as np
import random

# Set the seed for reproducibility
seed = 2
np.random.seed(seed)
random.seed(seed)

# Create DataLoader for training and validation sets
train_dataset = TensorDataset(X_train_tensor, y_train_tensor)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)

val_dataset = TensorDataset(X_test_tensor, y_test_tensor)
val_loader = DataLoader(val_dataset, batch_size=32, shuffle=False)

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Define the simple DNN model
class SimpleDNN(nn.Module):
    def __init__(self, input_shape):
        super(SimpleDNN, self).__init__()
        self.flatten = nn.Flatten()
        self.fc1 = nn.Linear(input_shape, 16)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(16, 1)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        x = self.flatten(x)
        x = self.relu(self.fc1(x))
        x = self.sigmoid(self.fc2(x))
        return x

# Define the model
input_shape = X_train.shape[1] * X_train.shape[2]
model = SimpleDNN(input_shape).to(device)

# Define the optimizer and loss function
optimizer = optim.Adam(model.parameters(), lr=1e-4)
criterion = nn.BCELoss()

# Define the training loop
num_epochs = 500
patience = 50
best_loss = float('inf')
counter = 0

for epoch in range(num_epochs):
    model.train()
    running_loss = 0.0
    correct_train = 0
    total_train = 0

    for X_batch, y_batch in train_loader:
        X_batch, y_batch = X_batch.to(device), y_batch.to(device)  # Move data to GPU
        optimizer.zero_grad()
        outputs = model(X_batch)
        loss = criterion(outputs.squeeze(), y_batch)
        loss.backward()
        optimizer.step()
        running_loss += loss.item() * X_batch.size(0)
        predicted = torch.round(outputs)  # Use round instead of > 0.5
        correct_train += (predicted.squeeze() == y_batch).sum().item()
        total_train += y_batch.size(0)

    # Validation phase
    model.eval()
    val_loss = 0.0
    correct_val = 0
    total_val = 0
    ##y_pred = []

    with torch.no_grad():
        for X_batch, y_batch in val_loader:
            X_batch, y_batch = X_batch.to(device), y_batch.to(device)  # Move data to GPU
            outputs = model(X_batch)
            loss = criterion(outputs.squeeze(), y_batch)
            val_loss += loss.item() * X_batch.size(0)
            #y_pred.extend(outputs.squeeze().cpu().numpy())
            predicted = torch.round(outputs)  # Use round instead of > 0.5
            correct_val += (predicted.squeeze() == y_batch).sum().item()
            total_val += y_batch.size(0)

    val_loss /= len(val_loader.dataset)

    # Calculate accuracies
    train_accuracy = correct_train / total_train
    val_accuracy = correct_val / total_val

    # Print accuracy
    #y_pred_binary = [round(y) for y in y_pred]
    #val_accuracy = accuracy_score(y_test_tensor.cpu(), y_pred_binary)

    print(f"Epoch {epoch+1}/{num_epochs}, Training Loss: {running_loss / len(train_loader.dataset)}, Validation Loss: {val_loss:.4f}, Validation Accuracy: {val_accuracy:.4f}")

    # Early stopping and checkpoint
    if val_loss < best_loss:
        best_loss = val_loss
        best_epoch = epoch+1
        torch.save(model.state_dict(), f"/home/CIN/lram2/BITCOIN_MASTER_DISSERTATION/30-minutes-bitcoin/classification_task_deep-pytorch/fully_conected_mlp/best_DCN_model_epoch.pth")
        print("Model saved!")
        counter = 0
    else:
        counter += 1
        if counter >= patience:
            print("Early stopping triggered")
            break
print(f"best epoch: {best_epoch}")
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Define the simple DNN model
class SimpleDNN(nn.Module):
    def __init__(self, input_shape):
        super(SimpleDNN, self).__init__()
        self.flatten = nn.Flatten()
        self.fc1 = nn.Linear(input_shape, 16)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(16, 1)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        x = self.flatten(x)
        x = self.relu(self.fc1(x))
        x = self.sigmoid(self.fc2(x))
        return x

# Define the model
input_shape = X_train.shape[1] * X_train.shape[2]
model = SimpleDNN(input_shape).to(device)

# Step 3: Load the state dictionary
model.load_state_dict(torch.load("/home/CIN/lram2/BITCOIN_MASTER_DISSERTATION/30-minutes-bitcoin/classification_task_deep-pytorch/fully_conected_mlp/best_DCN_model_epoch.pth"))

# Step 4: Move the model to the appropriate device (if using GPU)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

# Set the model to evaluation mode
model.eval()

# Initialize variables for loss and accuracy
test_loss = 0.0
correct = 0
total = 0

# Create DataLoader for the test set
test_dataset = TensorDataset(torch.tensor(X_test_scaled, dtype=torch.float32), torch.tensor(y_test, dtype=torch.float32))
test_loader = DataLoader(test_dataset, batch_size=32, shuffle=False)

criterion = nn.BCELoss()

with torch.no_grad():
    for inputs, targets in test_loader:
        inputs, targets = inputs.to(device), targets.to(device)
        outputs = model(inputs)
        loss = criterion(outputs.squeeze(), targets)
        test_loss += loss.item()

        predicted = (outputs > 0.5).float()
        correct += (predicted.squeeze() == targets).sum().item()
        total += targets.size(0)

# Calculate average loss and accuracy
average_test_loss = test_loss / len(test_loader)
test_accuracy = correct / total

print(f"Test loss: {average_test_loss:.4f}")
print(f"Test accuracy: {test_accuracy:.4f}")

# Set the model to evaluation mode
model.eval()

# Initialize list to store predictions
predictions = []

with torch.no_grad():
    for inputs, _ in test_loader:
        inputs = inputs.to(device)
        outputs = model(inputs)
        predictions.append(outputs.cpu().numpy())

# Concatenate all predictions into a single array
pred_y = np.concatenate(predictions, axis=0)
pred_y_classes = (pred_y > 0.5).astype(int)  # Convert probabilities to binary predictions

print("Predictions on the test set (probs):", pred_y)
print("Predictions on the test set:", pred_y_classes)

from sklearn.metrics import classification_report, confusion_matrix
# Compute classification report
report = classification_report(y_test, pred_y_classes)
print("Classification Report:\n", report)

# Compute confusion matrix
cm = confusion_matrix(y_test, pred_y_classes)
print("Confusion Matrix:\n", cm)
X_test_scaled[30932].shape
import torch

# Let's assume your new instance is a numpy array with shape (30, 1)
new_instance = X_test_scaled[30945]  # This is your input data

# Convert the new instance to a PyTorch tensor and move it to the appropriate device (CPU/GPU)
new_instance_tensor = torch.tensor(new_instance, dtype=torch.float32).to(device)

# Add a batch dimension since the model expects input in the shape (batch_size, features)
new_instance_tensor = new_instance_tensor.unsqueeze(0)  # Shape changes from (30, 1) to (1, 30, 1)

# Set the model to evaluation mode
model.eval()

# Make the prediction
with torch.no_grad():  # Disable gradient calculation
    output = model(new_instance_tensor)

# The output is a probability, so apply rounding to get a binary prediction
prediction = output.item()

# Print the prediction
print(f"Prediction for the new instance: {prediction}")

explained_instance = x_values[c_indexes[27939]].reshape(1,-1,1)#indices do array com dispositivos eletronicos]
learning_process_name = 'Lime'
transform_name = 'uniform'

from TSInterpret.InterpretabilityModels.leftist.leftist import LEFTIST

model.eval()

leftist = LEFTIST(model.cpu(),(X_train,y_train),mode = "time", backend='PYT',
                  learning_process_name='Lime',transform_name=transform_name, explanation_size=10,nb_neighbors=1000)

label='0'

explanations2 = leftist.explain(explained_instance,0)

leftist.plot(np.array([explained_instance.reshape(-1,1)]),np.array([explanations2[0]]))

All of my explanations appear like this image, where i have no feature importance associated with, however by using tensorflow i get different results:

Hi @marreapato ,
would you mind printing the explanation ?  Just to figure out if the Algorithm implementation or  the plot function is the issue.
Which version of TSInterpret are you using ?

Thanks for the reply @JHoelli

I am using version 0.4.5 and the array of explanations appears as [array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])]

Hi @marreapato,
I think the issue is the output layer of your model. Most approaches in here expect a Softmax Ouput, i.e. two neurons for a binary classification problem.
I did some changes in the PyTorchWrapper of TSInterpret.
Please try the changes by installing and let me know if it works:
 pip install https://github.com/fzi-forschungszentrum-informatik/TSInterpret/archive/refs/heads/main.zip

If that does not work, as fallback, retraining your model with softmaxoutput, two neurons and nn.Crossentropyloss might also be a solution.

@JHoelli Thanks for your reply, i haven't even tried the first solution you provided me with (i can still do it though, in case you need some feedback). I switched my activation function of the output layer and the loss function, and i finally got the feature importance of the timesteps!

Btw i am using the library for my masters research, i am certainly going to cite the paper in my publications related with it, thanks for the effort in giving me a solution for my problem.
It's an applied research with financial time-series data, so if there's any interest in having your names attached with it, or in doing any sort of collaboration, let me know, my institutional email (From a Public University based in Brazil - Federal University of Pernambuco) is<EMAIL_ADDRESS>Off-topic:
Would you also happen to know any online events based on Explainability for time series?


Update webui.nim tooltips show() + setTimeout()

I changed the tooltips for the Nim bindings, making it clearer that a script and js file should be included in the html for proper window response. As per issue #29.
Additionally (as mentioned by another commenter in issue #29), the setTimeout() function controls the general wait times for the backend. I find it important to note that a timeout change will also change the backends response time to the close of the app in case of a browser crash.
Specifically, setting setTimeout(0) should keep the backend running indefinitely in case of a browser crash, which could lead to very unsafe behaviour/multiple backend instances upon restart. (If I understood its behaviour correctly)
Additonally, providing setTimeout with a number greater than 86400 would also lead to the backend waiting forever (see resp. C code). So I added that to the tooltip as well.

Additonally, providing setTimeout with a number greater than 86400 would also lead to the backend waiting forever

Though in the webui_set_timeout function, the time provided is capped to WEBUI_MAX_TIMEOUT
...
#define WEBUI_MAX_TIMEOUT    (60)    // Maximum startup timeout in seconds the user can set
...

void webui_set_timeout(size_t second) {
    ...
    if (second > WEBUI_MAX_TIMEOUT)
        second = WEBUI_MAX_TIMEOUT;

    _webui_core.startup_timeout = second;
}

Thanks for the clarifications, I reversed some of the changes accordingly.

That's okay, but webui.js is essential for everything, single window, multiple windows. The hint Essential for creating multiple windows sounds like we don't need webui.js in a single window.

Haha, yes, I get your point.
However, a single window will correctly spawn even without webui.js, while multiple Windows won't without it.
That makes it a necessity for multi-window sessions and just an "option" for single window sessions. (Wether the single window will work properly is a different matter.)

I got it, now it's clear, so, actually, when you spawn the first window, webui waits for it to connect before spawning the second window, so when webui.js is missing from first window, then the window will never connect, which will make webui waits.

I think you're right, yes. That would make a lot of sense.

Let's keep the text in the correct meaning:
## Please include <script src="webui.js"></script> in the HTML
## for proper window communication.

I understand now your point, but still, we should avoid saying single and multiple windows...

Why do you think that we should avoid also mentioning multiple windows (in addition to the tooltip described above)? Would it cause confusion or similar?
Alternatively I may ask: What would be the benefit of omitting the additonal hint "Essential for creating multiple windows." ?

Because as I said, it's not the normal behavior. webui.js is necessary for communication, while spawning issue can be changed in future, for example a new APIs/settings to make webui does not wait, which will make single and multiple windows spawns all in one shot, even without webui.js.

The change you mentioned has not been implemented yet, idk if there is even a pull request for it lol.
The hint I added to the tooltip would have correctly stated how the current implementation works. The additional hint would not have been a nuisance, since it was quite short, too.
You argue that a future version could fix this issue. Well, since this future version will take approximately several months to ship, I feel like in the meantime, an additional hint wouldn't have been bad.
However, I don't feel the need to argue about this anymore and quite frankly, considering this entire conversation was about a tooltip and the fact that your profile is private, I am almost tempted to think you are trolling me.
At this point I'll just leave it up to the maintainer and consider this discussion done on my part.


mat-select - need Check All & Uncheck All options (Select All & Deselect All)  for multiple selection

Bug, feature request, or proposal:
Request: Need options for Check All and Uncheck All for multiple selection.
What is the expected behavior?
We can have 2 options at the top of the options panel to Check all or Uncheck all options in a single click.
Something like this https://silviomoreto.github.io/bootstrap-select/examples/#selectdeselect-all-options
What is the current behavior?
Currently user has to click on each and every option in order to check or uncheck all the options available in the select control.
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
Angular 5

This use case is very common. I was quite surprise that angular material 2 does not support this....
Anyway hoping to see this feature soon :)

I suppose that one checkbox "Select all" on the top would be more intuitive.
And if items selected partially, then checkbox have indeterminate state

Here is a solution for multi select select all that I wrote, feel free to use it.
https://stackblitz.com/edit/angular-material-select-multi-c6vtux

Hi all!
I came out with a very simple solution, just add disabled property to your options to react when the all options were selected.
Unfortunally the mat-option component doesn't expose the selected property for input :(
<mat-form-field>
<mat-select name="stores" placeholder="Stores" [(ngModel)]="form.stores" multiple required>
  <mat-option #allS  [value]="-1">All</mat-option>
  <mat-option [disabled]="allS.selected" *ngFor="let s of stores | async" [value]="s.id">
    {{ s.name }}
  </mat-option>
</mat-select>
</mat-form-field>

Hope it helps

I use reactive forms, and also needed to implement this.  I ended up using modified version of what @HashSix put out there (thanks!)
If anyone comes along, here is my version using reactive form:
https://stackblitz.com/edit/angular-material-select-multi-c3agwh
In my project i just use checkbox outside of the select itself, and trigger select/deselect from that.

Would be great to have this as a standard part of angular material.
For now, inspired by the solutions from both @HashSix and @vesrah, I've created a simple reusable component that can be added before the the first : https://stackblitz.com/edit/angular-material-select-all

Note this needs to work for groups - for now, I've personally just implemented this something like:
<mat-select placeholder="Stuff"
            [(value)]="selectedStuff"
            multiple>
    <mat-optgroup *ngFor="let group of this.optionGroups"
                    [label]="group.header">
        <mat-option class="select-all" disabled>
            <button mat-button
                    (click)="selectAll(group.children)">
                Select All
            </button>
            <button mat-button
                    (click)="selectAll(group.children, false)">
                Deselect All
            </button>
        </mat-option>
        <mat-option *ngFor="let item of group.children"
                    [value]="item">
            {{item}}
        </mat-option>
    </mat-optgroup>
</mat-select>

Noting in selectedStuff we need to filter out the empty select all options.
What would be nice is if someone could tell me how to style these as simple links & move them up beside the group header?

I did this:
    <span>
        <mat-form-field class="search-max-width" color="warn">
          <mat-select placeholder="Segments" [formControl]="segments" multiple
                      (selectionChange)="toggleSegments()">
            <mat-option (onSelectionChange)="toggleAllSegments($event)" value="all">All</mat-option>
            <mat-option *ngFor="let segment of allSegments" [value]="segment">
              {{segment}}
            </mat-option>
          </mat-select>
        </mat-form-field>
    </span>

in ts => segments = new FormControl([]); and set values conditionally.. But of using angular 6 ;)

@adornala how did you deal with indeterminate state.

<mat-form-field class="select-form">
    <mat-select
    placeholder="Years"
    name="year"
    class="filter-select"
    [(ngModel)]="selectedYears"
    multiple
    #yearSelect="ngModel">
        <mat-option
          (click)="selectAll(yearSelect, years)" [value]="years[0]">
          Seleccionar todos
        </mat-option>
      <mat-option *ngFor="let year of years| slice:1; let i = index;" [value]="year">
        <span *ngIf="year.id > 0">
          {{year.viewValue}}
        </span>
      </mat-option>
    </mat-select>
</mat-form-field>

selectAll(select: NgModel, values) {     if (Object.keys(this.selectedYears).length > 1) {       this.checkAll = false;       select.update.emit([]);    }     else {        this.checkAll = true;        select.update.emit(values);      }   }
This work for me with a checkbox.  Is a extension of @HashSix answer.

I would also like them to add this feature.

+1 from me!

+1 Would love to see this as a configurable option.

1 for mat-selection-list as well

I would like this aswell. In the meantime I added a select-all component that can be used. It is similar to the one by @RobertDiPaolo but it doesn't take the mat-select options and value as inputs. https://stackblitz.com/edit/angular-material-abra-select-all

@jsonod

1 for mat-selection-list as well

This is already/now a feature: https://material.angular.io/components/list/api#MatSelectionList
See selectAll() and deselectAll() methods.
What would be nice is if these two methods also triggered the selectionChange event.

Indeed, to be consistent with mat-selection-list.

allS.selected

with disabled property we can't uncheck selected mat-option in mat-selected

that is weird that so much time after, such an important feature is not yet provided.

Still no update on this matter?

Any updates on such an important feature request?

this is still needed

can it please get an higher priority? 🧐  @jelbourn

4 Years now since the initial request :).
Happy 2022 - still no updated yet?

Hoping this is somewhere in the works?
Has anyone found an implementation outside of angular material for multi-select with select all (and also preferably a search filter)?

any update on this feature ?

Angular material worst library in terms of functionality. You cant use this library in enterprise application.
As compare to react material library is this totally bullshit.

Welcome to angular, prehistoric framework.

The problem with all the workarounds presented so far is that none of them are accessible. I have the feature built out and working, however I realized the 'Select All' and 'Clear All' buttons can't be navigated to via keyboard and came here looking for solutions. The only way you can arrow-up or arrow-down between elements in a mat-select dropdown is if they are  components. Tabbing or Shift + Tabbing closes the dropdown and moves you to the next / previous DOM element.

Almost 2024 and still no updates

when ALL option checked then other options uncheck and when any other option check then ALL option uncheck
<mat-select #ids multiple ngModel name="membershipIds">
<mat-option #allS [value]="-1" (click)="ids.value=allS.selected?[-1]:[]">-- ALL --
<mat-option *ngFor="let membership of memberships"
(click)="ids.value=ids.value[0]===-1?ids.value.slice(1):ids.value"
[value]="membership.id">{{membership.name}}

At the very least, it would be nice if there was an indeterminate input on MatListOption. Without even that, there is no way to implement your own select all in a selection list without either

only using the selected input, and thus missing out on indeterminate when only partial selections are made OR
trying to figure some clunky workaround using mat-checkbox


Use a Microsoft compliant image for docker

Codecov Report

Merging #602 (5c2c36c) into main (8d750cd) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #602   +/-   ##
=======================================
  Coverage   90.17%   90.17%
=======================================
  Files          78       78
  Lines        4549     4549
  Branches      942      942
=======================================
  Hits         4102     4102
  Misses        252      252
  Partials      195      195

Flag
Coverage Δ

unittests
90.17% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d750cd...5c2c36c. Read the comment docs.


Add code splitting example for react

Description
Import loadable-components, a highly maintained and recommended (by react team) library , to code split + lazy load components.
Load in the components when they are used instead of pre-loading them, causing a boost to SEO score, performance etc.
Related Issue
https://github.com/adobe/aem-project-archetype/issues/710
Motivation and Context
Improves load times
Improves SEO performance
Demonstrates a complex issue
How Has This Been Tested?
Tested using a local AEM instance against generated archetype projects
Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] I have signed the Adobe Open Source CLA.
[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

@CezCz @pfauchere @znikolovski @habansal fyi :)

Pending 1 problem with SSR - chunks are not included in the server dist.

Closing, gonna look for another solution then loadable.


[Publication] need to add successful message toast

EPAM AI DIAL chat version
0.16.0
What happened?
Now if to send publication request there is no any successful message. Expected: should be.
TBD the list of the publication actions where the message is useful (unpublish? no need for admin?)
Confidential information

[X] I confirm that do not share any confidential information

I suggest something neutral like:

The request "" has been sent
The request "" has been approved
The request "" has been rejected


Release version 0.0.1

Codecov Report
Patch and project coverage have no change.

Comparison is base (e44fb84) 100.00% compared to head (52fba0c) 100.00%.

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #77   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           53        53
  Lines         1996      1996
  Branches       134       134
=========================================
  Hits          1996      1996

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[Question]: How to use a remote deployed Llama model?

Question Validation

[X] I have searched both the documentation and discord for an answer.

Question
It is only mentioned in the doc how to use a local LLM or Openai.
However, I has a Llama 13B deployed in a remote server, how can I access it from llamaIndex.
I tried llm=OpenAI(api_key=<my api_key>, api_base=<remote server url>), but it will report "The model gpt-3.5-turbo does not exist"

@dosubot I'm trying to bind the llm to a ServiceContext
ServiceContext.from_defaults(
    chunk_size=1024,
    llm=llm,
    embed_model=embeddings
)

but it report ValueError: "AzureOpenAIMultiModal" object has no field "callback_manager"

Use openailike, pass the url of your hosted llm and add any random value into key and model. That should be enough for you to get going!
https://docs.llamaindex.ai/en/stable/examples/llm/localai.html#llamaindex-interaction

@chansonzhang yea, if it is using openai-like API, use the OpenAILike LLM class
Otherwise, if its just your own custom API, you need to implement your own LLM class
https://docs.llamaindex.ai/en/stable/module_guides/models/llms/usage_custom.html#example-using-a-custom-llm-model-advanced

Depending on what you deployed your LLM with, it may also be covered by an existing class
https://docs.llamaindex.ai/en/stable/module_guides/models/llms/modules.html

Use openailike, pass the url of your hosted llm and add any random value into key and model. That should be enough for you to get going! https://docs.llamaindex.ai/en/stable/examples/llm/localai.html#llamaindex-interaction

@anoopshrma Thanks, this works for me.
llm = OpenAILike(
    model='meta-llama/Llama-2-13b-chat-hf',
    api_key=<my api_key>,
    api_base=<remote server url>
)

Depending on what you deployed your LLM with, it may also be covered by an existing class https://docs.llamaindex.ai/en/stable/module_guides/models/llms/modules.html

@logan-markewich  I also tried this, but when I execute pip install llama-index-llms-llama-api, it reports:
ERROR: Could not find a version that satisfies the requirement llamaapi<0.2.0,>=0.1.36 (from llama-index-llms-llama-api) (from versions: none)
ERROR: No matching distribution found for llamaapi<0.2.0,>=0.1.36


attached_network_device (merge to lldp-2 branch)

Make Endpoint attached_network_device optional - though strongly recommended - to allow operation in the case that these details are not available to the BC, cf. IS-04 Node /self interfaces.attached_network_device being optional.
Reasonable?

One concern that has been raised with this is whether for example when running SR-IOV on a NIC, any or all LLDP transmissions from the virtual function interfaces will reach the switch that's part of the IS-06 system. If not, the network controller would not be able to identify where the Endpoint is connected automatically and it must be registered with the attached_network_device info.
The best practice would be to enable LLDP on every network device and endpoint, and mandate that the IS-04 Node includes interfaces.attached_network_device but since it is hard to change IS-04 in this regard, this proposed change, plus a warning in the IS-04 Node test suite (https://github.com/AMWA-TV/nmos-testing/issues/491) could be the best option.

I understand that we would need to make attached_network_device optional for IS-04 consistency but as discussed offline there are cases in BC/NC communication where device info is mandatory. BTW I merged pull request ..Sachin
From: Gareth Sylvester-Bradley<EMAIL_ADDRESS>Reply-To: AMWA-TV/nmos-network-control<EMAIL_ADDRESS>Date: Tuesday, February 18, 2020 at 9:02 PM
To: AMWA-TV/nmos-network-control<EMAIL_ADDRESS>Cc: vsachin33<EMAIL_ADDRESS>Review requested<EMAIL_ADDRESS>Subject: Re: [AMWA-TV/nmos-network-control] attached_network_device (merge to lldp-2 branch) (#66)
@garethsb-sony requested your review on: #66 attached_network_device (merge to lldp-2 branch).
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub, or unsubscribe.


ci: Add pre-commit mixed-line-ending hook

Description
Add pre-commit mixed-line-ending hook with the option to force conversion of line endings to LF (\n)
Motivation and Context
PR https://github.com/terraform-aws-modules/terraform-aws-eks/pull/2973  introduced 6 files with CRLF (\r\n) line endings.
This ensures contributors commit files with LF (\n) line endings.
Breaking Changes
N/A
How Has This Been Tested?


[ERR_UNSUPPORTED_DIR_IMPORT] ES Module factory exports

Note: I am uncertain if this is an issue withtypechain or @typechain/ethers-v5. This issue might also exist with @typechain/web3-v1
Recently upgraded typechain in my project to the versions below and I appreciate the modularized exports as we are looking to export typechain types in our contract's npm package. However there is a slight issue when using ESModules due to incorrect exports that do not explicit use the file extensions.
Used versions:
{
"typechain": "8.1.1",
"@typechain/ethers-v5": "10.2.0",
"@typechain/web3-v1": "6.0.2",
}

Below a short example snippet of the typechain generated index.ts when using openzeppelin.
//index.ts
import type * as openzeppelin from "./@openzeppelin";
export type { openzeppelin };
export * as factories from "./factories";
export type { IERC1155Upgradeable } from "./@openzeppelin/contracts-upgradeable/token/ERC1155/IERC1155Upgradeable";
export { IERC1155Upgradeable__factory } from "./factories/@openzeppelin/contracts-upgradeable/token/ERC1155/IERC1155Upgradeable__factory";

We then export these types in our package so developers can easily use them.
export * as TypechainEthers from "./typechain/ethers/index.js"

There is a slight issue when using ESModules. When using ESModules, js files MUST be imported with explicit file name extensions. However this only applies to importing variables and NOT types in TS files (as these have no runtime value). Therefore in the snippet above, while the contract types are properly exported, the factory exports (both factories and  IERC1155Upgradeable__factory) don't just export types but rather a class that mimics ethers factory but with proper types. Therefore the import breaks because the factory is not exported with explicit reference to the .js file. Here would be an example fix to the above snippet.
//index.ts
import type * as openzeppelin from "./@openzeppelin";
export type { openzeppelin };
export * as factories from "./factories/index.js"; //add "/index.js"
export type { IERC1155Upgradeable } from "./@openzeppelin/contracts-upgradeable/token/ERC1155/IERC1155Upgradeable";
export { IERC1155Upgradeable__factory } from<EMAIL_ADDRESS>//add ".js"

Please let me know if you have an idea where the logic for the export file generation is and I would be open to looking into making the patch.

Fantastic. I will read the Contributing guide and see if I can propose a patch. Seems simple enough.

I am seeing this as well. While versions 8.2.0 of typechain and 11.0.0 of @typechain/target-ethers-v5 fix the addition of a .js suffix, they do not add the index.js to the factories export. I'd like to help here but I'm having a hard time finding the relevant code for the factories export (it does not seem to be in the file linked above).

I recently came back to this and saw the news about 8.2.0. Thanks so much for adding your PR as well to fully make Typechain work with ESM!

I tried it using local linking and it worked! I also published the branch on npm if people want to test it out (though I'd recommend using the official package once it is released or even better using local linking). This was a blocker for my team and it is easier for us to use npm than do local linking.
https://www.npmjs.com/package/@owlprotocol/typechain
https://www.npmjs.com/package/@owlprotocol/typechain-ethers-v5

Note to get this working is not as simple as changing your package.json due to dependencies using the official package names. Instead you will have to use aliasing (https://pnpm.io/aliases). So unless you absolutely need npm package support I would not recommend this and instead use local linking for development (clone @chmanie branch, build, and link https://pnpm.io/cli/link). Or just be patient and wait for the team to hopefull accept the PR and release a new version!
Here are a few steps if you need npm support:

Use pnpm (ideally)
Delete your node_modules and remove pnpm-lock
Replace in package.json "typechain": "8.2.0" with "typechain": "npm<EMAIL_ADDRESS>Only necessary if you have dependencies using typechain. Add a .pnpmfile.cjs file

module.exports = {
    hooks: {
        readPackage(pkg, context) {
             //Global version overrides
            const depsOverrides = {
                "typechain": "<EMAIL_ADDRESS>            };

            Object.entries(depsOverrides).map(([name, version]) => {
                if (pkg.dependencies[name]) pkg.dependencies[name] = version;
                if (pkg.devDependencies[name]) pkg.devDependencies[name] = version;
                if (pkg.peerDependencies[name]) pkg.peerDependencies[name] = version;
            })

},
        afterAllResolved(lockfile) {
            return lockfile;
        }
    }
}

Hey there @leovigna, I'm not sure I understand why import type statements do not need the module suffix too. I'm currently running into typechecking problems due to some of these. If I rewrite the path for each of these type-only imports to have the module suffix, then all of these errors are fixed.
E.g.
ts/src/ethers-contracts/index.ts:10:37 - error TS2307: Cannot find module './StdStorage.sol' or its corresponding type declarations.

10 import type * as stdStorageSol from "./StdStorage.sol";
                                       ~~~~~~~~~~~~~~~~~~

I gues it depends what bundler you use. I use esbuild and since my final output is js files the type imports just get removed anyways.
It is preferable to still have the file extensions for type exports too probably but has no direct impact on ECMAScript since at the end of the day since you are just running js.

Hm, I see. I'll try to add a pull request that does that. I'm personally using tsx to run some scripts with typechain bindings but these pop up as errors if I don't fix them. Thanks!


Update grafana to its latest version

Describe the bug
https://github.com/istio/istio/blob/master/addons/grafana/Dockerfile.grafana#L1 shows that Istio currently uses the version 5.0.4 of grafana, which was released on March 27, 2018. The latest version of grafana is 5.2.2, released on July 24, 2018. Istio should update its grafana version to leverage the fixes included in the latest grafana version.
Expected behavior
A clear and concise description of what you expected to happen.
Steps to reproduce the bug
Steps to reproduce the behavior.
Version
What version of istio and Kubernetes are you using? Use istioctl version and kubectl version
Is Istio Auth enabled or not?
Did you install the stable istio.yaml, istio-auth.yaml.... or if using the Helm chart please provide full command line input.
Environment
Which environment, cloud vendor, OS, etc are you using?
Cluster state
If you're running on Kubernetes, consider following the
instructions
to generate "istio-dump.tar.gz", then attach it here by dragging and dropping
the file onto this issue.

#7945

@lei-tang @sdake Why is this important to go in 1.0.1? Why can't this go in 1.1?

@rkpagadala this is important because the older version does not have the security fixes in the latest version.

An update to this issue: I have created the PR https://github.com/istio/istio/pull/7945 to fix this issue by updating the grafana used by Istio to the latest grafana version 5.2.2.  I have included two screenshots in https://github.com/istio/istio/pull/7945:  the first screenshot shows the upgraded grafana version 5.2.2 and the second screenshot shows the Istio service traffic in the dash board of grafana 5.2.2. The  PR https://github.com/istio/istio/pull/7945 has been merged in the commit dea5796.

@lei-tang Did you test the istio-mesh-dashboard? It requires some changes to work with Grafana 5.2.3 (we did not test 5.2.2):

@louisvernon The tests I did are generating some traffic in Istio and view the Istio service dashboard in grafana. The results are shown in the screenshots attached to #7945. The first screenshot shows the upgraded grafana version 5.2.2 and the second screenshot shows the Istio service traffic in the dash board of grafana 5.2.2.

Closing. Grafana is on 5.2.3 now on all branches now.


Starter web project.json template has error/typo

Upon creating a starter web project I get the following messages:
The value of property "postrestore" must be a string, but was "[ "npm install" ]",
The value of property "prepare" must be a string, but was "[ "grunt bower:install" ]"
This is regarding the following part of the configuration:
    "scripts": {
        "postrestore": [ "npm install" ],
        "prepare": [ "grunt bower:install" ]

message is gone if configuration is changed to:
    "scripts": {
        "postrestore":  "npm install",
        "prepare":  "grunt bower:install"

@maumar as @MartinJohns mentioned it is probably because VS might have cached an old version of JSON schema. The latest schema I can see here [http://json.schemastore.org/project] seems to have an array for each of the script nodes.
"script": {
			"type": [ "string", "array" ],
			"items": {
				"type": "string"
			},
			"description": "A command line script or scripts.\r\rAvailable variables:\r%project:Directory% - The project directory\r%project:Name% - The project name\r%project:Version% - The project version"
		}

Reloading schema should remove the squiggles from the project.json.


Problems with Reddit command.

Recently, the Reddit command PR (#346) was merged with a number of problems being overlooked. This issue is to outline the required fixes and any additional enhancements to the feature.
Current Problems

When navigating using reactions, the embed is edited twice in succession, first making the embed blank and then with the new content. This sometimes results in slow updating and the embed staying blank.

The user-agent in the header dict is currently set to Iceman but in fact it should be set the app/client that is sending the request.
So it needs to be changed to : python3:python-discord/seasonalbot:1.0.0 (by /u/PythonDiscord)

Currently, the API does not use an authenticated request. We should make sure to use an app token from Reddit the same as the Python Bot does also. This bot will need one created by an admin specific to Seasonalbot.
Use the PyDis Bot's Reddit cog implementation here as reference.

The word meant is misspelt as ment(on line 60).

also - this is a minor error but

“meant” is misspelt incorrectly (shown on L60)

Definitely worth mentioning, thanks @Suhail6inkling.

Bot doesn't check if the reddit have any post in it.
Such reddit can be /r/makemyideas
Bot fails silently
I found this on private testing
03/01/20 09:29:29 - bot.seasons.evergreen.error_handler ERROR: Unhandled command error: list index out of range
Traceback (most recent call last):
  File "/home/ag/.local/share/virtualenvs/seasonalbot-0Ba-uPuk/lib/python3.8/site-packages/discord/ext/commands/core.py", line 79, in wrapped
    ret = await coro(*args, **kwargs)
  File "/home/ag/PycharmProjects/bookmark-re-discord/seasonalbot/bot/seasons/evergreen/reddit.py", line 58, in get_reddit
    if posts[1]["data"]["over_18"] is True:
IndexError: list index out of range

It happened today
https://discordapp.com/channels/267624335836053506/607247579608121354/683523350353674313

Instead of checking if the first post is 18+ using if posts[1]["data"]["over_18"] is True: I'd rather filter out all the 18+ posts

yes, that should also work @quanta-kt.

I will be completing this along with #530

Fixed with #530


Switch to POST methods on forecast

What
We're hitting an issue, when requesting more than 13 services to be forecasted by the API. This is not an issue with paas-billing per se, as we believe we're hitting haproxy's header size limit which is 8K per GET request.
We believe this could be sortable by changing the method on the request to POST.

I seem to be able to reproduce this, with 13-ish events.
https://admin.london.cloud.service.gov.uk/calculator?rangeStart=2019-04-01&rangeStop=2019-04-30&items%5B0%5D%5BplanGUID%5D=f4d4b95a-f55e-4593-8d54-3364c25798c4&items%5B0%5D%5BnumberOfNodes%5D=1&items%5B0%5D%5BmemoryInMB%5D=64&items%5B1%5D%5BplanGUID%5D=f4d4b95a-f55e-4593-8d54-3364c25798c4&items%5B1%5D%5BnumberOfNodes%5D=1&items%5B1%5D%5BmemoryInMB%5D=1024&items%5B2%5D%5BplanGUID%5D=c84d1bef-9500-4ce9-88b2-c0bd421bbf8a&items%5B2%5D%5BnumberOfNodes%5D=1&items%5B2%5D%5BmemoryInMB%5D=&items%5B3%5D%5BplanGUID%5D=2f6df103-8216-4bc4-bb38-a6422e03c981&items%5B3%5D%5BnumberOfNodes%5D=0&items%5B3%5D%5BmemoryInMB%5D=&items%5B4%5D%5BplanGUID%5D=63cdac92-9e44-42a6-ba3f-7be3dccf5dc6&items%5B4%5D%5BnumberOfNodes%5D=0&items%5B4%5D%5BmemoryInMB%5D=&items%5B5%5D%5BplanGUID%5D=242a4782-7454-4ff4-93b2-dd88b333cb46&items%5B5%5D%5BnumberOfNodes%5D=0&items%5B5%5D%5BmemoryInMB%5D=&items%5B6%5D%5BplanGUID%5D=f4d4b95a-f55e-4593-8d54-3364c25798c4&items%5B6%5D%5BnumberOfNodes%5D=32&items%5B6%5D%5BmemoryInMB%5D=128&items%5B7%5D%5BplanGUID%5D=f4d4b95a-f55e-4593-8d54-3364c25798c4&items%5B7%5D%5BnumberOfNodes%5D=128&items%5B7%5D%5BmemoryInMB%5D=2048&items%5B8%5D%5BplanGUID%5D=4edc975c-3f07-46f1-bd87-ecb35b76298f&items%5B8%5D%5BnumberOfNodes%5D=0&items%5B8%5D%5BmemoryInMB%5D=&items%5B9%5D%5BplanGUID%5D=b6949ea7-5c98-4c69-b981-4b5318ea8b7c&items%5B9%5D%5BnumberOfNodes%5D=1&items%5B9%5D%5BmemoryInMB%5D=&items%5B10%5D%5BplanGUID%5D=2f6df103-8216-4bc4-bb38-a6422e03c981&items%5B10%5D%5BnumberOfNodes%5D=0&items%5B10%5D%5BmemoryInMB%5D=&items%5B11%5D%5BplanGUID%5D=43b01f78-0c9f-482e-b77f-e28189ccd870&items%5B11%5D%5BnumberOfNodes%5D=0&items%5B11%5D%5BmemoryInMB%5D=&items%5B12%5D%5BplanGUID%5D=b6949ea7-5c98-4c69-b981-4b5318ea8b7c&items%5B12%5D%5BnumberOfNodes%5D=1&items%5B12%5D%5BmemoryInMB%5D=&items%5B13%5D%5BplanGUID%5D=065a7de5-28e8-4de1-8a39-4b4f752e2f2f&items%5B13%5D%5BnumberOfNodes%5D=0&items%5B13%5D%5BstorageInMB%5D=5120

I'll try and fix.

@richardTowers brought up an important caveat to using POST. Right now the calculator links are shareable, but that won't be the case if we switch to using POST. He suggested compressing the data with gzip and base64, which in my example above reduces the size by about 2/3rds. That would allow for 35+ services before this bug emerged.

Fixing this using compression would be awkward. The calculator and the billing would need to be using compression, and that's problematic because we'd need to use javascript in the calculator to compress the query string. This sounds too big for me right now.