Paragraph separator character causes TS Server to get into bad state

From https://github.com/microsoft/vscode/issues/95685
TypeScript Version:  3.9.0-dev.20200420

Search Terms:

invisible character
paragraph
diagnostics / errors

Code
For the JS
var ftext = [
"Und", "dann", "eines"];
var a = 1;

Note there is an invisible paragraph separator character after the opening[

In the file, delete the space after the = on the second line.

Bug:
TS starts reporting bogus errors:

It looks the paragraph character messes up the parsing and causes the error line numbers to be off by one
[Trace  - 22:31:41.88] <semantic> Event received: semanticDiag (0).
Data: {
    "file": "/Users/matb/projects/san/test.js",
    "diagnostics": [
        {
            "start": {
                "line": 2,
                "offset": 8
            },
            "end": {
                "line": 2,
                "offset": 12
            },
            "text": "Cannot find name 'dann'.",
            "code": 2304,
            "category": "error"
        },
        {
            "start": {
                "line": 2,
                "offset": 16
            },
            "end": {
                "line": 2,
                "offset": 21
            },
            "text": "Cannot find name 'eines'.",
            "code": 2304,
            "category": "error"
        }
    ]
}

In JavaScript, Paragraph Separator is a line terminator; however, in the edits described above, VS Code says that edits are occur on line 2.
From TypeScript's perspective, line 2 occurs immediately after the Paragraph Separator. Ultimately, editors need to account for each language's newline settings if they intent to use line/offset encodings for commands, so I don't think we can fix this on the TypeScript side.

See also:

the original issue filed over 5 years ago https://github.com/microsoft/TypeScript/issues/2343
the issue tracked on the LSP side https://github.com/microsoft/language-server-protocol/issues/80 (but that's been closed)

Here's the edit we send to TSServer when deleting the space after the = in a = 1:
[Trace  - 23:20:02.22] <syntax> Sending request: updateOpen (12). Response expected: yes. Current queue length: 0
Arguments: {
    "changedFiles": [
        {
            "fileName": "/Users/matb/projects/san/test.ts",
            "textChanges": [
                {
                    "newText": "",
                    "start": {
                        "line": 2,
                        "offset": 8
                    },
                    "end": {
                        "line": 2,
                        "offset": 9
                    }
                }
            ]
        }
    ],
    "closedFiles": [],
    "openFiles": []

The indexes one based so they look correct to me.
@RyanCavanaugh @DanielRosenwasser Does this also match what you see?

Yeah, that's what I'm seeing as well, but if I understand correctly the edit needs to happen at line 3 because the paragraph separator is a line terminator.

VS Code does not consider the character a line break. Looks like VS does though
@alexdima should be able to say if this is intentional or not. If you have this character in your code and it's not inside a string though, it's probably a bug

Currently, VS Code considers the following characters/sequences to be line terminators: CRLF, CR or LF.
I feel that this is a difficult topic to get wide agreement on, since we need to work with multiple programming languages, each one with its own definition of what a line terminator is. So, if a compiler creates a diagnostic at line 100, it appears that line number is defined according to the programming language's own specification.
From my reading of https://www.unicode.org/reports/tr14/tr14-32.html , Unicode defines a mandatory line break after the following:

LB4. Always break after hard line breaks (BK). In the BK class there are the following:

000C - FORM FEED (FF)
000B - LINE TABULATION (VT) - [OPTIONAL]
2028 - LINE SEPARATOR (LS)
2029 - PARAGRAPH SEPARATOR (PS)

LB5. Treat CR followed by LF, as well as CR, LF, and NL as hard line breaks:

000Dx000A- CARRIAGE RETURN (CR) x LINE FEED (LF)
000D - CARRIAGE RETURN (CR)
000A - LINE FEED (LF)
0085 - NEXT LINE (NEL)

Here I'm looking at the most popular languages used in VS Code for which I could find a specification or where the decision is not pushed out to specific implementations (looking at you, C++).
It looks like the JS spec Table 33 defines the following as line terminators. I suppose TS uses the same:

Interestingly, the C# spec A.1.1 Line terminators defines the same line terminators:

HTML however defines newlines as just CR, LF and CRLF:

Python also defines just CR, LF and CRLF:

PHP also defines just CR, LF and CRLF:

Java defines LF, CR and CRLF:

YAML section 3.1.4 Line Breaks uses the following:

To sum up:

char/seq
Unicode
JS
C#
HTML
Python
PHP
Java
YAML

CRLF
✅
✅
✅
✅
✅
✅
✅
✅

CR
✅
✅
✅
✅
✅
✅
✅
✅

LF
✅
✅
✅
✅
✅
✅
✅
✅

LS
✅
✅
✅

✅

PS
✅
✅
✅

✅

NEL
✅

✅

FF
✅

VT
✅

I am not sure what would be best, since it would be very difficult to make VS Code end-of-line sequence dependent on the file's language and changing things now might break language extensions which follow our definition.
@dbaeumer Is the end-of-line sequence something that is specified in the Language Server Protocol?

Actually, a straight forward way to tackle this would be for us to prompt users and ask for permission to "fix" their files. Most likely, LS, PS or NEL are inserted by accident in files by copy-pasting and they are unwanted there anyways. I've opened https://github.com/microsoft/vscode/issues/96142 to track this.

@alexdima yes it is specified here: https://microsoft.github.io/language-server-protocol/specifications/specification-current/#textDocuments

The LSP support \r, \n and \r\n


support for block comments

Hi there,
I  have just tried your module.
It would be awesome if you could add support for block comments
(just to be clear: comments starting with /* and ending with */ that
could span multiple lines).
Thank you so much for your support!

Hello!
I've just merged in a PR which adds support for c-style block comments. If you want to give it a spin, try installing jsoncparser==1.1.0 and let me know how that goes! There are some limitations currently to the support (I've listed them out in the PR here) but hopefully those won't be too much of an issue and they'll be taken care of as I re-work the backing parser.

Thank you for your fast response!
I have tried it and it works!


Make 99% of API calls return in less than 1s; constant time to number of nodes and pods

@lavalamp would you be interested in taking this (it's from the v1.0 roadmap)?
Also cc'ing @roberthbailey and @satnam6502 for good measure.

I'm looking at slow 'get pods' call #4196 and have a few observations about API server performance. I added a doc about turning on profiling to docs/devel, which should be helpful for trying to find bottlenecks. Sadly AFAICT most of the time is spent in YAML handling, which I guess means that we need to reduce number of etcd entries parsed in normal workflow, try to parallelize some of them, write custom YAML parser or change the way things are stored in etcd.
As first two options have no impact on things outside the master I'm trying to apply them to get pods call. I've already tried parallelizing, but it do not give big improvement.
Also I'm sometimes seeing slow response from server when checking metrics. Most of the times it's instantaneous, but once in a while it takes few seconds. I don't know yet how metrics are stored/computed, but (I guess) it's not a master API call. This could mean that there may be a problem with HTTP server being slow/overcommited from time to time.

There's an issue with this goal, namely there's no machine on which API server should have this <1sec. performance. It is important, as on GCE by default we're running on single threaded CPU, hence there's not a lot of space for parallelization of anything.

Have we measured this and is it actually not meeting this target?
1.a) As Marek said, what is the golden system configuration upon which we will take this measurement? What is its load? 1s is meaningless by itself. I'd rather say "contstant time, with a low constant factor".

Even though it's single-core, parellelizing should still help if there's network access. I think I already did that before I left a month ago in the places it was appropriate, though.
...also it might be worth double checking if we ever allow more than one thread by setting GOMAXPROCS.

@lavalamp No, we haven't measured. It's possible that we already meet the goal. Also, as you noted, the goal here is under-defined, as it doesn't specify a workload. (I beileve @satnam6502 is working on workload generators.) So, part of this issue is to define a workload.

I added parallel decoding of data extracted from etcd for get pods, as it takes 1.5 to 6 sec when 50 pods are present (it's more like 1.2-4 with an empty cluster). It did not help.
Additionally get replicationcontroller is EXTREMELY slow, as per #4536. What is worse than this 15 sec "normal" runtime is that when I was trying to do get during the time when stop is running easily takes 150 sec (yup: 2.5 minutes). I noticed it in the evening, so I didn't have time to investigate it further.
I traced only APIserver part of execution.

Rephrasing my previous comment: get rc running time is far from constant. From <1sec in empty cluster to >15 sec when there's 50 RCs running, and minutes when multiple '''stop rc''' are under way (easily reproducible).
We should agree on some golden master/node configuration. I think it's kind of important, as if we assume that master will be running on single thread, the architecture may be quite different, than if we assume multithreaded one. In addition if we're aiming at multithreaded master I believe that we should add a second thread to the default configuration of the master machine for each provider, to make testing more realistic. Similar thing applies to node machines.
My personal opinion is that we should assume that both master and kubelet are running on multithreaded machines, and modify our default accordingly.
We should agree on some golden master/node configuration. I think it's kind of important, as if we assume that master will be running on single thread, the architecture may be quite different, than if we assume multithreaded one. In addition if we're aiming at multithreaded master I believe that we should add a second thread to the default configuration of the master machine for each provider, to make testing more realistic. Similar thing applies to node machines.
My personal opinion is that we should assume that both master and kubelet are running on multithreaded machines, and modify our default accordingly.

As per @gmarek's comment it seems reasonable to provide a minimum spec of X cores for our SLO of Y. It does seem unreasonable to expect a single core machine to run 50 pods (of any size) in a timely manner.

When I was last doing experiments I used n n1-highmem-16 master and I also tried a n1-highcpu-16 master but still ran into lots of issues -- perhaps because our API server needs to be (more?) multi-threaded?

OK, I see we don't actually use GOMAXPROCS outside of our tests. I'm making a PR to add that in, so at least we'll use more cores if we have them.

 It should also be clear that gets are getting slower with more running pods.

I'm sorry for the format, but GitHub does not allow pdfs:/ All results in seconds.
1 master (highcpu-16), 4 nodes (standard-2), 50 1-pod rc
empty	get rc 50 pods   	get pod 50 pods
0.51	| 4.19 | 0.75
0.48	| 3.53 | 0.71
0.51 | 3.41 | 0.85
0.51 | 3.47 | 0.76
0.5 | 	3.43 | 0.72
0.49 | 4.01 | 0.73
0.5 | 	4.17 | 0.72
0.51 | 3.89 | 0.72
0.49 | 3.76 | 0.83
0.51 | 3.57 | 0.75
1 master (highcpu-4), 10 nodes (standard-1), 200 1-pod rc
get rc emtpy	get rc 200 pods	get pod 200 pods
0.5   | 	1:13.41 | 0.91
0.52 | 	1:09.00 | 0.87
0.51 | 1:10.98 | 0.85
0.5 | 		 | 6.82
0.54 | 	 | 1.68
0.51 | 	 | 1.4
0.5 | 		 | 1.43
0.54 | 	 | 1.72
0.5 | 		 | 1.4
0.48 | 	 | 1.38
0.5 | 		 | 1.37

Prashanth is working on moving replication controller status computation into replication controller, which is going to have the side effect of making list rc not be O(#pods * #rc) anymore.

#4675 will add GOMAXPROCS to the scheduler, which I missed yesterday.

After #4429 lands, the times for replication controllers should drop down to be about the same as the pod times.

I'll paste my comment in this thread then:
When checking out the performance data on the apiserver under soak testing it appears that there is a fair amount of gc work going on:
20.43% kube-apiserver [.] runtime.MSpan_Sweep
3.91% kube-apiserver [.] findrunnable
3.05% kube-apiserver [.] runtime.mallocgc
2.29% kube-apiserver [.] scanblock
1.83% kube-apiserver [.] runtime.sweepone
1.75% kube-apiserver [.] runtime.atomicload
1.73% kube-apiserver [.] runtime.xchg
1.54% kube-apiserver [.] runtime.duffcopy
1.47% kube-apiserver [.] flushptrbuf
1.22% kube-apiserver [.] runtime.memclr
1.19% kube-apiserver [.] runqgrab
Seems related to: golang/go#9265
go version go1.3.3 linux/amd64

More color:
https://github.com/GoogleCloudPlatform/kubernetes/pull/4862 shows a rc fill time ~10 minutes.
A part of this lag is apiserver, but there are other issues in the system that we're tracing.

@timothysc what process are you using to generate those numbers?
I think taking boundpods out will drastically speed adding & removing pods to the system, I think there's a lot of contention for updating that.

@lavalamp - https://github.com/GoogleCloudPlatform/kubernetes/issues/5260 is the load we used for the numbers above.

What is "rc fill time"?

Get or list replicationcontrollers. List replicationcontrollers is currently O(#RCs*#pods).

listing rc's should be better now that #4751 has merged. @gmarek is it easy for you to take those measurements again?

Yup. I'll do it today. Are we still playing with 1 hyperthread master, or have we moved to something truly multithreaded? Are we optimising for cpu time or wall time?

1 master (standard-1), 10 nodes (standard-1), 100 1-pod rc

empty get pod
get rc 100 pods
get pod 100 pods

1.01
22.35
31.48

1.01
21.27
5.68

0.51
19.88
25.41

1.11
10.48
25.56

1.15
12.71
15.74

So get's got much better. Sadly stops are REALLY bad. I guess they are still O(|rc| * |pods|).
1 master (standard-8), 10 nodes (standard-1), 100 1-pod rc

empty get pod
get rc 100 pods
get pod 100 pods

1.13
2.01
2.09

1.17
1.90
1.65

1.01
1.81
1.60

1.13
4.00
1.84

1.14
3.07
1.60

Stops are way better, but still pretty bad. It's still faster to turn down cluster and start it again, than wait for 100 rc to go down.

are these single observations, a mean or 99%ile or what?

As we do these observations, can we check in scripts so that others can repro. the results?
Thanks!
--brendan

Stop polls RCs:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/kubectl/stop.go#L81
Watch should work on RC status now.

Those are single observations. I can write 'scripts', but they are more or less just 'for's of kubectls inside time. Literally single line scripts, but if you believe it's good to have them wrapped in something I can write something more general.

If there's a lot of noise (pods dying/coming up causing the controller to update its replica count) while you're trying the resize or delete (which are the 2 write operations in stop), the writes will conflict and trigger retries. The polling interval of these retries is 100ms. I'm curious to know how much of the stop time is just delete --all on the pods vs waiting for the rc to notice and take action.
We can probably get rid of the overhead of listing pods of an rc every time we modify it in watchcontrollers:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/registry/etcd/etcd.go#L125, although I believe each watch event is translated in a goroutine so they shouldn't add up to O(rc*pods).

On the side: I recall that someone was writing load generator - what's the status of this? If it's still far away I can hack a script which will do some kind of load testing.

@bprashanth The fact that you depend on goroutines explains speedup when running master on 8 hypethread machine. IIUC we still haven't decided what our target configuration is, so I don't know how much parallelism we allow. I'll write a simple script to measure execution times of basic operations.

I created PR #5432 with extremely simple script. It's kind of manual as it outputs time results. I lost some time trying to make variables out of those results, but failed. Somehow I wasn't able to correctly redirect stderr to stdout for timed wait call and put it into a variable.

How often were each of the tests run? I noticed (for tests that are dependent on etcd performance) that if you ran the same experiment again and again the results change as etcd accumulates state between invocations.

Is this a good thing, or a bad thing (i.e. what do you mean by 'accumulates states'). It depends on which ones you're asking about - roughly with few second pauses.

If I get a moment (which I may not since I need to change my focus to
networking/plug-ins) I will try and describe in more detail what I observed
(based on state accumulating due to events so a subsequent run of the sam
experiment may have different results from an earlier invocation). However,
I am not sure and I would need time to carefully check what is going on.
On Tue, Mar 24, 2015 at 8:48 AM, gmarek<EMAIL_ADDRESS>wrote:

Is this a good thing, or a bad thing (i.e. what do you mean by
'accumulates states'). It depends on which ones you're asking about -
roughly with few second pauses.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/4521#issuecomment-85569062
.

Status update:
@wojtek-t and me are working on three changes to meet this goal:

Faster DeepCopy
Speed up etcd client library by using ugorji codec to unmarshal etcd responses
Limit number of inflight "list pods" requests
According to our manual tests, with these changes we will meet 1s goal for all requests, except for listing pods, where 99tile is around few seconds.

@fgrzadkowski could we add a check of the /metric counters in e2e for regression detection?

Which counters do you mean? We already check latency in density tests.

@timothysc don't think we should fail any other e2e because of latent requests. one can invoke the same method from any e2e to log slow ones (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/test/e2e/util.go#L1006).

@bprashanth I'm ok with that for now.  We can setup prometheus alerts on queries.

Is it worth it to build an in memory pod cache fed by etcd directly at this point, or is limiting pod list requests sufficient?   We don't really gain anything by going to etcd for pods every few seconds, and we can reduce latency by the rtt to etcd for the same effective memory cost.

One additional idea we discussed is increasing the interval between relists, to something more like an hour rather than 5-8 minutes. I think it's fine to wait to see if the other changes are sufficient for our 1.0 goals, but this change may be necessary for our next scalability goal (1000 nodes or whatever). @lavalamp was OK with this change as long as this interval is configurable and all of the e2e tests set it low enough to force a relist during each test, so we are sure the functionality to do a level-based restore works.

We've option enabled list intervals on the tests we are interested in.  As mentioned in #8119 some lists are more expensive then others.
/cc @rrati

Status update:
@wojtek-t and me are working on three changes to meet this goal:
Faster DeepCopy
Speed up etcd client library by using ugorji codec to unmarshal etcd responses
Limit number of inflight "list pods" requests
According to our manual tests, with these changes we will
meet 1s goal for all requests, except for listing pods, where 99tile is around few seconds.

Can you give an update on these? It looks like first one is dependent on #8860/#8872, and last one is dependent on #6770? Is there a PR or issue for the second one?

I can elaborate a little on the relist situation, since I'm currently working on it:

Half of the relist problem (#6770) is addressed by https://github.com/GoogleCloudPlatform/kubernetes/pull/8574
The second half is split between:

piping the timeout in the above pr as a command line arg to the apiserver, setting it really high in prod, and lower during tests. This will not completely solve the relist problem because of the following point.
https://github.com/GoogleCloudPlatform/kubernetes/issues/8700. This should give us control over how often we relist.

We dropped the high latency limit to 5s and things aren't too bad (https://github.com/GoogleCloudPlatform/kubernetes/pull/8781)

@davidopp There is not PR or issue for the high CPU usage of etcd client library. Recently we've observed that despite CPU not being saturated we still have high latency (99tile up to 3-4s). It seems it's not CPU bound, so for now I've focused on understanding the reasons behind it.

at @fgrzadkowski we see the same thing.

@fgrzadkowski we don't need to saturate cpu to starve the watch goroutines. If you have a process with n goutines all taking locks and one goroutine that somtimes gets an event on a channel, the go scheduler seems to de-prioritize the latter (no hard evidence, a conclusion drawn just from experimentation, see the test in the first comment on https://github.com/GoogleCloudPlatform/kubernetes/issues/8700).

We don't really gain anything by going to etcd for pods every few seconds, and we can reduce latency by the rtt to etcd for the same effective memory cost.

+1
I'm also a +1 to the interval shift.

@bprashanth I don't follow. I'm not talking about watch go routines, but regular http handlers for requests like list or get.

For the last few days @wojtek-t and me were trying to understand why observe non-saturated CPU but still have high request latency (around 3s for 99tile of list pods). We found few interesting facts.
The whole list requests consists of two parts:

generate in-memory list object
write response

Surprisingly the second part is longer than the first one and can take seconds! There three slow parts:

encoding list to json
indenting json
writing json to ResponseWriter

Each of those steps takes hundreds of milliseconds (up to seconds)
At the same time all of the long requests are coming from density test which lists all pods every 5 or 10 s, which generates up to 10M of data each. This also explains why the CPU is not saturated - each of those heavy requests is handled in one thread, which increases high percentiles, but doesn't load the CPU. Effectively they outweigh normal requests (i.e. those not listing all pods).
To fix 1. we can use ugorji codec.
To fix 2. we can just delete this code and do the indentation on the client side (i.e. in kubectl) as systems (e.g. kubelet) do not care about indentation. This also reduces maximum size of the response to ~6M.
We have no idea why 3. is so slow. Surprisingly it's the slowest out of three and can take seconds. Any suggestion how to debug this? Is it expected? As this is network related cc-ing @thockin

Do we need to tune the output buffer of the http response writer?  Do we have any wrappers around the response writer besides the HttpLog?

On May 29, 2015, at 11:11 AM, Filip Grzadkowski<EMAIL_ADDRESS>wrote:
For the last few days @wojtek-t and me were trying to understand why observe non-saturated CPU but still have high request latency (around 3s for 99tile of list pods). We found few interesting facts.
The whole list requests consists of two parts:
generate in-memory list object
write response
Surprisingly the second part is longer than the first one and can take seconds! There three slow parts:

encoding list to json
indenting json
writing json to ResponseWriter

Each of those steps takes hundreds of milliseconds (up to seconds)
At the same time all of the long requests are coming from density test which lists all pods every 5 or 10 s, which generates up to 10M of data each. This also explains why the CPU is not saturated - each of those heavy requests is handled in one thread, which increases high percentiles, but doesn't load the CPU. Effectively they outweigh normal requests (i.e. those not listing all pods).
To fix 1. we can use ugorji codec.
To fix 2. we can just delete this code and do the indentation on the client side (i.e. in kubectl) as systems (e.g. kubelet) do not care about indentation. This also reduces maximum size of the response to ~6M.
We have no idea why 3. is so slow. Surprisingly it's the slowest out of three and can take seconds. Any suggestion how to debug this? Is it expected? As this is network related cc-ing @thockin
—
Reply to this email directly or view it on GitHub.

@filbranden We don't see pod listing as the long pole in our tests, do you have the metrics values that you are looking at?

@timothysc I'm not really looking at metrics values right now... Are you sure you mean me here?

sorry for some reason my autofill is hitting you.. @fgrzadkowski ^

No worries, I'm glad it's not me losing my mind :-)

@fgrzadkowski I was talking about the reason we have concurrent lists when we don't saturate cpu, not the reason concurrent lists are slow.

Re. indentation in the client: Please see the discussion in the original issue -- #2243. Having the ability to pretty-print the json in a browser is useful. It doesn't need to be the default, but let's not lose that. Either a URL parameter or, preferably, using User-Agent to disable indentation.

Even for curl, indentation is useful, so if we want to disable it, we should do that in our client library.

Status update:
Current e2e scalability tests for 100 node cluster with 4 core master in majority of runs have latency metrics similar to this:
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.99 Latency:2.420198s}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.9 Latency:2.266635s}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.5 Latency:1.32378s}
INFO: Top latency metric: {Verb:POST Resource:pods Quantile:0.99 Latency:481.504ms}
INFO: Top latency metric: {Verb:POST Resource:bindings Quantile:0.99 Latency:414.718ms}

99tile of list of pods is almost never above 3s. Other requests are always below 1s.
I also manually tested this setup with 8 core master and metrics are below 1.5s.
Today I've also reenabled load tests (#8873), that @piosz wrote some time ago but has been disabled due to e2e instability. We will observe it closely and add expectation on latency metrics tomorrow.

In this case, the 99%ile is less than 2x the median.  So thread load
balance is good.   Problem is that average case is too slow.
On Jun 2, 2015 4:47 AM, "Filip Grzadkowski"<EMAIL_ADDRESS>wrote:

Status update:
Current e2e scalability tests for 100 node cluster with 4 core master in
majority of runs have latency metrics similar to this:
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.99 Latency:2.420198s}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.9 Latency:2.266635s}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.5 Latency:1.32378s}
INFO: Top latency metric: {Verb:POST Resource:pods Quantile:0.99 Latency:481.504ms}
INFO: Top latency metric: {Verb:POST Resource:bindings Quantile:0.99 Latency:414.718ms}
99tile of list of pods is almost never above 3s. Other requests are always
below 1s.
I also manually tested this setup with 8 core master and metrics are below
1.5s.
Today I've also reenabled load tests (#8873
https://github.com/GoogleCloudPlatform/kubernetes/pull/8873), that
@piosz https://github.com/piosz wrote some time ago but has been
disabled due to e2e instability. We will observe it closely and add
expectation on latency metrics tomorrow.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/4521#issuecomment-107926162
.

Can you quantify "CPU not saturated"?
On Jun 2, 2015 5:08 AM, "Eric Tune"<EMAIL_ADDRESS>wrote:

In this case, the 99%ile is less than 2x the median.  So thread load
balance is good.   Problem is that average case is too slow.
On Jun 2, 2015 4:47 AM, "Filip Grzadkowski"<EMAIL_ADDRESS>wrote:

Status update:
Current e2e scalability tests for 100 node cluster with 4 core master in
majority of runs have latency metrics similar to this:
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.99 Latency:2.420198s}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.9 Latency:2.266635s}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.5 Latency:1.32378s}
INFO: Top latency metric: {Verb:POST Resource:pods Quantile:0.99 Latency:481.504ms}
INFO: Top latency metric: {Verb:POST Resource:bindings Quantile:0.99 Latency:414.718ms}
99tile of list of pods is almost never above 3s. Other requests are
always below 1s.
I also manually tested this setup with 8 core master and metrics are
below 1.5s.
Today I've also reenabled load tests (#8873
https://github.com/GoogleCloudPlatform/kubernetes/pull/8873), that
@piosz https://github.com/piosz wrote some time ago but has been
disabled due to e2e instability. We will observe it closely and add
expectation on latency metrics tomorrow.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/4521#issuecomment-107926162
.

@erictune Most of the time it's around 80% utilization according to htop.

Can you grab GC metrics for length of time spent in GC and the collection rate?  I think it's GOGCVERBOSE=1 or similar (or you can trigger a pprof heap dump, which will show the collection time at the bottom in a comment).  We might be stuck waiting on the GC lock to be released for that 20%

On Jun 2, 2015, at 8:32 AM, Filip Grzadkowski<EMAIL_ADDRESS>wrote:
@erictune Most of the time it's around 80% utilization according to htop.
—
Reply to this email directly or view it on GitHub.

@fgrzadkowski wrote "For the last few days @wojtek-t and me were trying to understand why observe non-saturated CPU but still have high request latency (around 3s for 99tile of list pods)."
I think you are asking why the 99%ile is 2x higher than the mean, despite cpu not being 100%.
That seems consistent with what you might expect from queuing theory.
If you use:
http://www.supositorio.com/rcalc/rcalclite.htm
and then pick:
M/M/C
then set:

c = 8  (hyperthreads)
lambda = 6.5 (requests per second, assumed)
mu = 1 second of cpu time per request (implied by 80% utilization, curious what QPS you measured)

then you get:

80% utilization (which you stated)
1.32 mean request latency (you stated 1.32s median LIST latency, mean would be useful)
which is consistent with your results.

Then, if you set:
Probabilites (time in system <= Seconds) to 2.26 (which is your 90%ile latency),
the calculator predicts that 2.26 is your 82%ile latency.
That seems pretty consistent too.
From this, I conclude that there is no need to invoke garbage collection or thread starvation to explain the tail latencies you are seeing.

@fgrzadkowski There was a request to also test on g1-small since that's the smallest master users might run in practice. There's no expectation that the performance will be good -- we just want to make sure it is not unusable.

@piosz have enabled load test which creates :

300 rc with 5 pods each
25 rc with 30 pods each
3 rc with 250 pods each
All creations are spread across all first 10 minuts. After that we randomly resize/delete/recreate all RCs.

Metrics from the first run:
INFO: Top latency metric: {Verb:POST Resource:pods Quantile:0.99 Latency:1.50546s}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.99 Latency:1.360146s}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.9 Latency:1.022879s}
INFO: Top latency metric: {Verb:PUT Resource:pods Quantile:0.99 Latency:481.39ms}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.5 Latency:400.466ms}

Thanks for the update! Can you indicate which PRs related to this issue are still out for review (other than the CoreOS one), if any?

CoreOS one is the only PR in-flight. Number in my previous comment where taken from jenkins.

CoreOS one is the only PR in-flight. Number in my previous comment where taken from jenkins.

Great, thanks! So the CoreOS PR will improve the above numbers further? (I assume "taken from jenkins" means only using what is already in the codebase -- but wanted to make sure I am understanding correctly.)

@davidopp Yes. I'll send a PR to update go-etcd client library version today.

@fgrzadkowski : Once you're confident we're meeting the goal described in the title of this issue, please move the issue to milestone "1.0-post". (We don't need to wait on the etcd PRs to merge if we're already meeting the goal.)

I enabled load test. Currently it plays the following scenario:

[0-10min] Run 300 RCs with 5 replicas each, 25 RCs with 30 pods each, 3 RC with 250 pods each
[10-13min] Scale all RCs randomly and list pods within RC
[13-16min] Repeat 2
[16-26min] Delete all RC
The test is configured in a way to keep ~5 creation/deletions per second, which IMO sounds reasonable. The test checks latency metrics after all of the phases to see if we meet 1s criteria. This means that some of requests were performed when the cluster was not full.

First run gave really good results (with 8 core master):
load test:
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.99 Latency:707.259ms}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.9 Latency:382.948ms}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.5 Latency:239.901ms}
INFO: Top latency metric: {Verb:DELETE Resource:pods Quantile:0.99 Latency:193.937ms}
INFO: Top latency metric: {Verb:PUT Resource:nodes Quantile:0.99 Latency:161.933ms}

density 30 pods per node:
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.99 Latency:1.60586s}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.9 Latency:969.726ms}
INFO: Top latency metric: {Verb:PUT Resource:nodes Quantile:0.99 Latency:513.258ms}
INFO: Top latency metric: {Verb:POST Resource:pods Quantile:0.99 Latency:494.578ms}
INFO: Top latency metric: {Verb:PUT Resource:pods Quantile:0.99 Latency:491.616ms}

I think we should pay more attention to the load test which plays more realistic scenario.
Tomorrow I will set thresholds to 1s for load test and 2s for density test.
@davidopp @brendanburns @wojtek-t Assuming that next runs will give similar results I suggest we announce victory. Does that sound reasonable to you?

@fgrzadkowski just fyi, with the load test we resize rcs quickly so there's a chance you will run into https://github.com/GoogleCloudPlatform/kubernetes/issues/9147

Assuming that next runs will give similar results I suggest we announce victory. Does that sound reasonable to you?

Yes.

@davidopp Results with 4 core master:
Density with 3 pods per node
INFO: Top latency metric: {Verb:POST Resource:pods Quantile:0.99 Latency:589.097ms}
INFO: Top latency metric: {Verb:PUT Resource:pods Quantile:0.99 Latency:430.241ms}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.99 Latency:372.226ms}
INFO: Top latency metric: {Verb:POST Resource:bindings Quantile:0.99 Latency:359.243ms}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.9 Latency:302.09ms}

Density with 30 pods per node
INFO: WARNING Top latency metric: {Verb:LIST Resource:pods Quantile:0.99 Latency:2.656382s}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.9 Latency:1.269396s}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.5 Latency:854.817ms}
INFO: Top latency metric: {Verb:POST Resource:pods Quantile:0.99 Latency:481.31ms}
INFO: Top latency metric: {Verb:PUT Resource:nodes Quantile:0.99 Latency:370.808ms}

Load test
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.99 Latency:575.679ms}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.9 Latency:372.226ms}
INFO: Top latency metric: {Verb:DELETE Resource:pods Quantile:0.99 Latency:190.719ms}
INFO: Top latency metric: {Verb:PUT Resource:nodes Quantile:0.99 Latency:152.142ms}
INFO: Top latency metric: {Verb:PUT Resource:replicationcontrollers Quantile:0.99 Latency:151.097ms}

NOTE: For load tests some requests were performed when we were creating/deleting RCs, so the cluster was not full (full spectrum from 0 to 30 pods per node)
@davidopp I suggest we keep 4 core master, but increase the threshold for density to 3 seconds. WDYT?

Sure, sounds good.

And thanks for collecting the additional data. That doesn't look too terrible vs. the 8-core data (i.e. it's not 2x the latency).

After merging #9862 the latency improved
https://github.com/GoogleCloudPlatform/kubernetes/pull/9862#issuecomment-113504442
says
Results from density test with 30 pods per node in 100 node cluster with n1-standard-4 master:
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.99 Latency:1.605515s}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.9 Latency:1.270142s}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.5 Latency:529.251ms}
INFO: Top latency metric: {Verb:POST Resource:bindings Quantile:0.99 Latency:444.361ms}
INFO: Top latency metric: {Verb:PUT Resource:pods Quantile:0.99 Latency:426.816ms}

This is way better (~25%) than the usual run for density test.


Deploy the STAC API backend database

Overview
Deploy the pgstac database via AWS RDS. Try Amazon Aurora PostgreSQL Serverless or, if that doesn't work, a straight Postgres database.
We will need to create a Postgres 13 database with the PostGIS extension:

pgstac Dockerfile
RDS PostGIS

We will then need to run the pypgstac migrate command on the database, for example:
PGUSER=username PGPASSWORD=password PGHOST=localhost PGDATABASE=postgis PGPORT=5439 pypgstac migrate

The above command works for our local PostGIS database (deployed using https://github.com/stac-utils/stac-fastapi/blob/master/docker-compose.yml#L64 with image: postgis/postgis:13-3.3-alpine).
Also see https://stac-utils.github.io/pgstac/pypgstac/ for documentation on migrations.
Tasks

[x] Determine how to adapt pgstac to run in AWS RDS.
[ ] Deploy a Postgres database to AWS RDS via CloudFormation
[ ] Automated installation of the PostGIS extension
[ ] Automate running the migrate command

Double-check that PostGIS must be installed manually before running the migrate script.

(TL;DR: We can't use pgstac with a serverless database because Aurora serverless doesn't currently support newer versions of Postgres, but I'm preserving the deployment/connection steps for a serverless Postgres database below, in case they become useful in the future.)
Unfortunately a serverless database won't work for our purposes (as I'll explain below), but I did create the following CloudFormation template and successfully deployed a serverless database:
Resources:
  DatabaseCluster:
    Type: AWS::RDS::DBCluster
    Properties:
      DatabaseName: StacApiBackend
      Engine: aurora-postgresql
      EngineMode: serverless
      MasterUsername: postgres
      MasterUserPassword: password
      Port: 5432
      VpcSecurityGroupIds:
        - !GetAtt DatabaseSecurityGroup.GroupId

  DatabaseSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: 'Security group for the STAC API backend database'
      SecurityGroupIngress:
        - SourceSecurityGroupName: default
          IpProtocol: tcp
          FromPort: 5432
          ToPort: 5432

The database security group inbound rule allows connections from the default security group (the default VPC security group), so you can follow these steps to connect to the database:

Create an EC2 instance in the same VPC as your database
Make sure the instance has the default security group attached
SSH into the instance, then run PGDATABASE=StacApiBackend PGUSER=postgres PGPASSWORD=sandboxpass123 PGHOST=<host> PGPORT=5432 psql where <host> is the database cluster Endpoint (obtained from the AWS Console).

However, when we attempt to run pypgstac migrate on our database from within our EC2 instance, the command fails with Exception: PGStac requires PostgreSQL 13+. We run select version(); from psql and see that the database is running PostgreSQL 10.18. We can specify the version with the EngineVersion property of the database cluster, but when we check the available versions with:
aws rds describe-db-engine-versions --engine aurora-postgresql --query "DBEngineVersions[].EngineVersion" --filters Name=engine-mode,Values=serverless

we see that the only two supported versions are 10.18 and 11.13. If we remove the --filters option, we see versions up to 14.5. So it would appear that we cannot use pgstac with a serverless database, because serverless doesn't support Postgres 13+.

Next step: determine if we should use a different engine mode with Aurora Postgres (remember to check the versions this time) or just use a normal Postgres database. Are there advantages to either option?

We'll probably want a single make target for deployment, PostGIS install/upgrade, and PGStac migration (install or upgrade).


Fix a typo (I think) in git clone progress text

Why:
Closes #20290

What's being changed (if available, include any code snippets, screenshots, or gifs):
I compared the git clone progress text with the one in my terminal when I ran git clone on my fork of Spoon-Knife, in the way the instructions indicate. It appears that the "> remote ... remote ... remove: Total 10 (delta 1), reused 10 (delta 1)" contains a typo, with remove instead of remote.
I fixed this in case it is a typo, by replacing remove with remote in the relevant line it appears.

Check off the following:

[x] I have reviewed my changes in staging (look for the "Automatically generated comment" and click the links in the "Preview" column to view your latest changes).
[x] For content changes, I have completed the self-review checklist.

Writer impact (This section is for GitHub staff members only):

[ ] This pull request impacts the contribution experience

[ ] I have added the 'writer impact' label
[ ] I have added a description and/or a video demo of the changes below (e.g. a "before and after video")

@hat-owl Thanks so much for opening a PR and linking it to your issue! I'll get this triaged for review :zap:


fuzzing input in result, resolves #76

Will implement the more complex matching.

Now for example -mr 'FUZZ *[a-f0-9]+ FUZ2 t' using multiple wordlists works.


Merging backend from experimental regions branch

Congratulations on this huge step forward! :tada:

Thanks Paulo! This was a massive merge, and uncovered a lot of small bugs in the backend. Now onward to merging the frontend stages!


hook stat

I'll use fishhook hook stat to get the call all the time  thank you

It's unclear what you're looking for. Could you file a new issue where you describe what you'd like to do, what's the expected behavior of fishhook and what's the actual behavior?
Thanks!

I want to intercept   “stat” Underlying c language .  Do you know how to intercept


using named constructor to FrameTiming in scheduler_test.dart

Description
This is part of flutter/engine#20229, for compatibility of the new FrameTiming field, and should be merged after flutter/engine#20269 is rolled.
Related Issues
flutter/flutter#62689
Tests
The PR itself a modification of the unit test
Checklist
Before you create this PR, confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] I signed the CLA.
[x] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[x] I read the Tree Hygiene wiki page, which explains my responsibilities.
[x] I updated/added relevant documentation (doc comments with ///).
[x] All existing and new tests are passing.
[x] The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Did any tests fail when you ran them? Please read Handling breaking changes.

[x] No, no existing tests failed, so this is not a breaking change.

We probably also need to verify if there are any reference of the default constructor of FrameTiming in Google3 tests.


⚠ Grafana has degraded performance

In 3157e59, Grafana ($SERVER_BASE/grafana/) experienced degraded performance:

HTTP code: 500
Response time: 5207 ms

Resolved: Grafana performance has improved in 8583b95 after 17 minutes.


WIP: Introduce an etcd operator leader status field

This patch explores adding an etcd operator status field which reports
leader member information. Exposing this would allow, for example, smarter
decision-making by the MCO regarding reboot ordering by providing a hint
to help minimize disruption via excessive leader changes.

See the discussion in https://github.com/openshift/machine-config-operator/issues/1897 for more details about one way this could be useful.

P.S., this is for now to help promote discussion and experimentation — we haven't actually done the work/measurement to prove the cited use case has a benefit with expanding the API for

@ironcladlou what about something like
status struct{
  conditions
  []EtcdMembers
}

EtcdMembers struct{
  name string
  node string
  status string // learning,unhealthy,not-a-member,healthy
  memberType string // leader,follower (whatever this is)
  fsyncP99InLastMinute string
  peerLatencyP99InLastMinute string

  // or whatever it is you want
}

OK so I'm about at the point in https://github.com/openshift/machine-config-operator/issues/1897 work where I'm blocking on this, because I really want to test OS update things after we're only updating etcd followers first.
I starting looking at hacking something into the MCD for this but it's ugly.

If we don't land this my tentative thoughts are:

Patch the MCD to detect if it's on control plane, if so start watching the local etcd and find out if the current node is a leader
If it is a leader add mco.openshift.io/etcdleader: "" label to node, if not remove label if exists

In playing with this what I'm stumbling over is getting the right creds set up to talk to etcd, and I don't know if there's a non-polling way to watch for leader changes.
The rest of the logic for managing upgrade order would be in the MCC and proceed basically the same.

You know what may be far simpler (and avoid the "etcd writes its state to kube which writes to etcd" issue) is having the etcd pod write out something extremely simple like /run/etcd/status.json or whatever, then the MCD pod on the control plane can use inotify to watch it.  Or we can set up some sort of local read-only communication channel.
Anyways for now I wrote some hacky code that reuses the certs from the apiserver in the MCD
https://github.com/openshift/machine-config-operator/pull/1946

Or, rather than trying to extend the API here we could use a node annotation.


Use iproute2 instead of ifconfig

@xlii-chl said : The package net-tools and the included ifconfig seems to be slowly abandoned in favor of iproute2.
It should be great to stick to the most commonly used packages.

Quick note before going to sleep :)
ip route get <IP_ADDRESS> | sed -n 's/.* src \([^[:space:]]*\) .*/\1/p'
ip route get 2001:DB8::1 | sed -n 's/.* src \([^[:space:]]*\) .*/\1/p'

Also available in JSON :
ip -json route get <IP_ADDRESS>
# [{...,"prefsrc":"<IP_ADDRESS>",...}]

The <IP_ADDRESS>/24 (RFC5737 TEST-NET-2) and 2001:db8::/32 (RFC3849) are reserved addresses so it should be safe to use them as is.

What do you think about this ChatGPT version ?
ip addr show | awk '/inet /{print $2}' | cut -d'/' -f1 | grep -v '<IP_ADDRESS>'

Well, if it works on your side, that's good but on mine, it outputs the dockers' adresses:
# ip addr show | awk '/inet /{print $2}' | cut -d'/' -f1 | grep -v '<IP_ADDRESS>'
<IP_ADDRESS>
<IP_ADDRESS>
<IP_ADDRESS>
<IP_ADDRESS>
# ip route get <IP_ADDRESS> | sed -n 's/.* src \([^[:space:]]*\) .*/\1/p'
<IP_ADDRESS>

On another computer with some virtualization, the VM network address is before the "main" one:
$ ip addr show | awk '/inet /{print $2}' | cut -d'/' -f1 | grep -v '<IP_ADDRESS>'
<IP_ADDRESS>
<IP_ADDRESS>
$ ip route get <IP_ADDRESS> | sed -n 's/.* src \([^[:space:]]*\) .*/\1/p'
<IP_ADDRESS>

(note: you can replace <IP_ADDRESS> by <IP_ADDRESS> or <IP_ADDRESS> if you feel it makes more sense.)

Seems to work...


How to install?

Hi, i just wanted to ask, if theres any Guide on how to install the WirelessCarPlay?
Thanks in advance!


freeze store in dev mode

Which @ngneat/elf-* package(s) are relevant/releated to the feature request?
store
Description
Akita has this awesome feature which prevents a lot of bugs: freezing the store in dev mode
This is to implement a similar feature for the elf.
Proposed solution
it will be a matter of setting __DEV__ variable just like it's done in Akita and including the following code here:
const { deepFreeze } = this.getConfig();
if (__DEV__ && deepFreeze) {
  value = deepFreeze(value);
}

Alternatives considered
Prod mode

This could be a breaking change if we decide that the default behavior should require calling enableElfProdMode() for production mode.

The alternative is to have elf act in prod mode by default and create a function enableElfDevMode().

I honestly would prefer enableElfProdMode() even though this would make it a breaking change.
Tree shaking

We could include the default implementation of the deepFreezeFn to be bundled together with the Store just like Akita does it. This would increase the bundle size a bit.

Alternatively, we could require a user to provide the deepFreeze function in the store configuration. This way it would be shaken away for these people who are not interested in this functionality:
import { deepFreeze } from '@ngneat/elf';

const { state, config } = createState(withProps<AuthProps>({ user: null }));
const authStore = new Store({ state, name: 'auth', config, deepFreeze });

Do you want to create a pull request?
Yes

It would probably be better to add a global custom preStoreUpdate hook:
preStoreUpdate(newState) {
  return deepFreeze(newState)
}

Not sure where preStoreUpdate function comes from. We could introduce a global object: elfHooks:
export interface ElfHooks {
  preStoreUpdate?: <S>(newState: S, storeName: string) => S;
}

export const elfHooks: ElfHooks = {};

This way, the consumer could use it like so:
import { elfHooks } from '@ngneat/elf';

elfHooks.preStoreUpdate = (newState, storeName) => {
  return deepFreeze(newState);
};

Does that sound all right?

Yes, that's exactly what I meant.

I'm working on this feature and trying to make it ergonomic for the developer. One thing I'd like to achieve is to specify the type of the store so that the developer gets intellisense inside of the preStoreUpdate function.
However, I'm running into an error that I'm having a hard time understanding. Do you have an idea what's happening here?

interface ElfHooks<State> {
  preStoreUpdate?: (prevState: State, newState: State, storeName: string) => State;
}

const elfHooks: ElfHooks<State> = {
  preStoreUpdate(state) {
    return { filter: 'elf' };
  }
};

type State = { filter: string };

How you access it in the store?

through elfHooksRegistry. When I said that we don't export it, I meant we don't export it to the public API. we can export the const and access it in the store

Please see the PR for details. Let me know if anything looks off. If not, I'll add docs


Can't use CSS (and its related languages) as an entrypoint

If you had index.scss in resources/scripts, and tried to do @vite('index.scss'), it won't work. This is not laravel-vite's fault, unfortunately. I believe it's related to this Vite issue.
The way you handle this normally in Vite is through a style tag:
<link rel="stylesheet" href="/resources/scripts/index.css">
<!-- I believe you can even do this -->
<link rel="stylesheet" lang="scss" href="/resources/scripts/index.scss">

I don't think this can be solved unless Vite was somehow hooked up as middleware or even compiled Blade templates (as in, resolved their URL references). Excluding the bug getting fixed upstream, of course.
The only thing I can think of is like... virtually compiling an HTML template with these assets in them, and then parsing for the emitted URLs (or using @include functionality)? It would be goofy.

Okay, so I've went for what you suggested (patching the manifest) and I did actually get something not too hacky working:
import { createHash } from "crypto"
import fs from "fs/promises"
import path from "path"
import { InputOption, OutputAsset } from "rollup"
import { Manifest, ManifestChunk, normalizePath, Plugin, ResolvedConfig } from "vite"

export function manifestPlugin(): Plugin {
  const manifest = new Map<string, ManifestChunk>()

  let config: ResolvedConfig

  return {
    name: "vite:laravel:manifest",
    apply: "build",
    enforce: "post",

    configResolved(resolved) {
      config = resolved
    },

    async generateBundle(_, bundle, isWrite) {
      if (!isWrite) return

      let entrypoints = normalizeInputOption(config.build.rollupOptions.input)
      if (!entrypoints) return

      entrypoints = entrypoints.map(entry =>
        normalizePath(path.relative(config.root, entry))
      )

      const values = Object.values(bundle)
      const assets = values.filter(c => c.type === "asset") as OutputAsset[]

      // -- CSS

      const cssEntrypoints = entrypoints.filter(entry => isStylesheet(entry))
      const cssAssets = assets.filter(asset => isStylesheet(asset.name!))

      if (config.build.cssCodeSplit) {
        // add CSS entrypoints to manifest
        for (const chunk of cssAssets) {
          if (!chunk.name) continue
          const name = removeExtension(chunk.name!)
          for (const entry of cssEntrypoints) {
            if (removeExtension(path.basename(entry)) === name) {
              manifest.set(entry, { file: chunk.fileName, src: entry })
            }
          }
        }
      } else {
        // Vite emits a single CSS file in this mode, named `style.css`
        const chunk = assets.find(asset => asset.name === "style.css")
        if (chunk) manifest.set(chunk.name!, { file: chunk.fileName, src: chunk.name! })
      }

      // -- Remaining Assets

      const remaining = entrypoints.filter(entry => isAssetEntrypoint(entry))

      for (const entry of remaining) {
        const fullPath = path.join(config.root, entry)
        const source = await fs.readFile(fullPath)
        const hash = getAssetHash(source)

        const ext = path.extname(entry)
        const name = removeExtension(entry)
        const fileName = path.posix.join(
          config.build.assetsDir,
          `${path.basename(name)}.${hash}${ext}`
        )

        manifest.set(entry, { file: fileName, src: entry })

        if (!bundle[fileName]) {
          this.emitFile({ name: entry, fileName, source, type: "asset" })
        }
      }
    },

    async writeBundle(_opts, bundle) {
      if (!bundle["manifest.json"]) return

      const manifestPath = path.resolve(config.root, config.build.outDir, "manifest.json")
      const viteManifest = JSON.parse(await fs.readFile(manifestPath, "utf8")) as Manifest

      for (const [key, value] of Object.entries(viteManifest)) {
        manifest.set(key, value)
      }

      await fs.writeFile(
        manifestPath,
        JSON.stringify(Object.fromEntries(manifest), null, 2)
      )
    }
  }
}

function normalizeInputOption(input?: InputOption) {
  if (!input) return null
  if (typeof input === "string") return [input]
  if (typeof input === "object") {
    if (Array.isArray(input)) {
      if (input.length === 0) return null
      return input
    }
    // must be a map object
    const keys = Object.keys(input)
    if (keys.length === 0) return null
    return keys
  }

  return null
}

function removeExtension(filename: string) {
  return filename.replace(/\.[^.]*$/, "")
}

function isStylesheet(filename: string) {
  return /\.(css|less|sass|scss|styl|stylus|pcss|postcss)$/.test(filename)
}

function isAssetEntrypoint(filename: string) {
  if (isStylesheet(filename)) return false
  return !/\.(html|jsx?|tsx?)$/.test(filename)
}
function getAssetHash(content: Buffer) {
  return createHash("sha256").update(content).digest("hex").slice(0, 8)
}

And a snippet of the manifest:
{
  "resources/scripts/wiki.scss": {
    "file": "assets/wiki.9b1650e2.css",
    "src": "resources/scripts/wiki.scss"
  },
  "resources/scripts/auth.scss": {
    "file": "assets/auth.faf72c64.css",
    "src": "resources/scripts/auth.scss"
  },
  "resources/scripts/index.scss": {
    "file": "assets/index.33b199bf.css",
    "src": "resources/scripts/index.scss"
  },
  "resources/scripts/auth-confirm.ts": {
    "file": "assets/auth-confirm.84453c6a.js",
    "src": "resources/scripts/auth-confirm.ts",
    "isEntry": true,
    "imports": [
      "_vendor.c819c281.js",
      "_toasts.14a23b13.js",
      "_FormError.04a43d00.js"
    ],
    "css": [
      "assets/auth-confirm.c1633232.css",
      "assets/auth.faf72c64.css"
    ]
  }
}

Unfortunately, I can't think of a way around using that writeBundle trick. I could maybe reorder the plugins, but that's probably even hackier.
I'll make a PR soon, it seems!


Implement CachedModelDependencyContainer, replaces BindingContainer

Usage for something like this would follow:
Model:
    Bindable<int> FieldBindable;  // Fields are ok
    Bindable<int> PropertyBindable { get; set; }  // Properties are also ok
    int NonBindable;  // Not ok - all members must be bindables

Resolver:
    [Resolved]
    Model ResolvedModel { get; private set; }  // As per the usual dependency injection

    [Resolved(typeof(Model))]
    Bindable<int> ResolvedFieldBindable { get; set; }  // Can resolve model's individual properties

Container:
    override CreateChildDependencies(parent)
        => new CachedModelDependencyContainer<Model>(parent);

I've left functionality for both modes of operation - injecting the model or injecting its bindables.
It is safe to reference ResolvedModel's bindables directly, since they're bound to Model rather than being a direct copy of it.
This would allow us to keep Multiplayer an OsuScreen, while overriding dependencies to provide the capabilities that CMC would have given.

Supersedes https://github.com/ppy/osu-framework/pull/2081


SQL Server Driver

Is there are recommended SQL Server Driver for  SQL Azure? See a few out there, none of them look maintained. (realize this could be because the application of node better aligns with mongodb or even other oss relational db choices)

Actually, a few of these look promising - https://github.com/joyent/node/wiki/modules#wiki-db-mssql (I have not tried any yet). Still curious to see the position of this project as it relates to SQL Azure. Thanks

Sql Azure is something we definetely would like to support. The timelines are, however, still unclear. We'll keep you guys posted as we have more information.


[SPARK-20477][SparkR][DOC]: Document R bisecting k-means in R programming guide

What changes were proposed in this pull request?
Add hyper link in the SparkR programming guide.
How was this patch tested?
Build doc and manually check the doc link.

Test build #76276 has started for PR 17805 at commit 540bf7a.

Test build #76276 has finished for PR 17805 at commit 540bf7a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76276/
Test PASSed.

cc @felixcheung This is a similar documentation change.

merged to master/2.2


Late initialization error

While using package in crashlytics I'm getting "LateInitializationError: Field 'baseList' has already been initialized" error. And another question is why you've added "_initCountries" to "didChangeDependencies", not to "initState"?

Hello, thanks for pointing out. I'll update the code.

Should be fixed in v0.1.3.
Please open this again if bug still occurs.
Thank you! 🎉


feat: SSR support

PR Checklist
Please check if your PR fulfills the following requirements:

[ ] Tests for the changes have been added (for bug fixes/features)
[ ] Docs have been added/updated (for bug fixes/features)

PR Type
What kind of change does this PR introduce?

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no API changes)
[ ] Build related changes
[ ] CI-related changes
[ ] Documentation content changes
[ ] Other... Please describe:

Issue Number

Issue Number: #114
Does this PR introduce a breaking change?

[ ] Yes
[x] No

TODO:

[ ] add docs
[ ] check if any migrations or changes in schematics are needed
[ ] update the deploy pipeline to deploy the prerendered version of NgDoc documentation to the AWS for better SEO

:tada: This PR is included in version 17.2.0 :tada:
The release is available on:

v17.2.0
GitHub release

Your semantic-release bot :package::rocket:


fix(pags): InvertedRegexTester against multivalued attributes

Checklist

[x] the individual contributor license agreement is signed
[x] commit message follows commit guidelines
[x] tests are included
[ ] documentation is changed or added

Description of change

InvertedRegexTester doesn't provide a logical test result against a multivalued attribute.
The Inverted response should be done against the final result and not for each value test (this case work only on a monovalued attribute). The problem is on the foreach attribute values, when it find a not match it returns true without testing all other values, and so it return always true expect if a single matching result appear.

And so any comment/review ? Can I merge it ?


python310Packages.pysigma: 0.9.4 -> 0.9.5

Changelog: https://github.com/SigmaHQ/pySigma/releases/tag/v0.9.5
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 222152 run on x86_64-linux 1

38 packages built:

python310Packages.pysigma
python310Packages.pysigma-backend-elasticsearch
python310Packages.pysigma-backend-elasticsearch.dist
python310Packages.pysigma-backend-insightidr
python310Packages.pysigma-backend-insightidr.dist
python310Packages.pysigma-backend-opensearch
python310Packages.pysigma-backend-opensearch.dist
python310Packages.pysigma-backend-qradar
python310Packages.pysigma-backend-qradar.dist
python310Packages.pysigma-backend-splunk
python310Packages.pysigma-backend-splunk.dist
python310Packages.pysigma-pipeline-crowdstrike
python310Packages.pysigma-pipeline-crowdstrike.dist
python310Packages.pysigma-pipeline-sysmon
python310Packages.pysigma-pipeline-sysmon.dist
python310Packages.pysigma-pipeline-windows
python310Packages.pysigma-pipeline-windows.dist
python310Packages.pysigma.dist
python311Packages.pysigma
python311Packages.pysigma-backend-elasticsearch
python311Packages.pysigma-backend-elasticsearch.dist
python311Packages.pysigma-backend-insightidr
python311Packages.pysigma-backend-insightidr.dist
python311Packages.pysigma-backend-opensearch
python311Packages.pysigma-backend-opensearch.dist
python311Packages.pysigma-backend-qradar
python311Packages.pysigma-backend-qradar.dist
python311Packages.pysigma-backend-splunk
python311Packages.pysigma-backend-splunk.dist
python311Packages.pysigma-pipeline-crowdstrike
python311Packages.pysigma-pipeline-crowdstrike.dist
python311Packages.pysigma-pipeline-sysmon
python311Packages.pysigma-pipeline-sysmon.dist
python311Packages.pysigma-pipeline-windows
python311Packages.pysigma-pipeline-windows.dist
python311Packages.pysigma.dist
sigma-cli
sigma-cli.dist


fix: only return RG name output after group has been created

Description
fix: only return RG name output after group has been created
Check the relevant boxes:

[x] Bug fix (nonbreaking change that fixes an issue)
[ ] New feature (nonbreaking change that adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Examples or tests (addition or updates of examples or tests)
[ ] Documentation update
[ ] CI related update (pipeline, etc.)

Checklist

[ ] If relevant, a test for the change has been added or updated as part of this PR.
[ ] If relevant, documentation for the change has been added or updated as part of this PR.

Merge

Merge using "Squash and merge".
Make sure to use a relevant conventional commit message that is based on the PR contents. The commit message determines whether a new version of the modules needs to be released, and if so, which semver number to use).

:tada: This PR is included in version 1.0.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


upload photo does not work

Hi, I'm testing the upload() function, even though it returned "Done", the photo was not uploaded.
However if I added the flag story=True, it works for the instagram story.

Sounds weird.
Can you provide other info via instapy terminal output to check instapy-cli and python version.
Can be helpful to know how you use it, so please provide your code or an example very similar to it.

If the problem continue please provide some other infos such as the account creation date, errors, instapy-cli and python version and a sample code to inspect the problem.
Thanks.
🥂


How to configure api key?

I'm using latest 2.2.1 version.
Regex search for api?key gives nothing.
This documentation is not right:
https://github.com/egeloen/IvoryGoogleMapBundle/blob/master/Resources/doc/service/distance_matrix.md#api-key
This as well:
https://github.com/egeloen/IvoryGoogleMapBundle/blob/master/Resources/doc/map.md#api-key
How should it be done?

First, thanks for taking taime to report it but unfortunatelly, there is nothing to fix...
Let me explain your issue, when refering to https://github.com/egeloen/IvoryGoogleMapBundle/blob/master/Resources/doc/service/distance_matrix.md#api-key you're refering to dev-master doc whereas if you're using the 2.2.1 version, you should use: https://github.com/egeloen/IvoryGoogleMapBundle/blob/2.2.1/Resources/doc/usage/services/distance_matrix/distance_matrix.md
Unfortunatelly for you, the api key support has only been added on master, so if you want to take benefit of it, you will need to upgrade.

Thanks for help. Just one more question - how much instability can I expect on dev-master?

Dependending what do you mean by stable :) If by stable, you mean it works, yes it works :) but if you mean no BC breaks I can't say I will not introduce BC breaks before releasing the next major release... I'm currently reworking the services. It will still work the same way but I will maybe break the way to create a service in a near future...

Closing as the issue has been resolved :)


'smolok spark submit' should fallback to default --master=spark://localhost:7077 if no --master option is present

This will be useful for local development, where you use localhost master address anyway.

I added one more think to the mechanism [1]
[1] - https://github.com/smolok/smolok/commit/67203068f11d4fe293d7716e284eedf26e861042#diff-519b3731c0b3c26bafa462d3efdcc120

Thanks!


How to add Spanish dictionary

Hello, I want to know if there is a way of adding a Spanish dictionary and if is possible how can I do it?

Looking forward to it! (for French in my case)

hello excellent repo. I would like to know if you have advanced in this matter?

The Spanish dictionary is already available in the latest version of this extension :)


(maint) Use callback to report on resolver exceptions

Previously, resolver exceptions were collected in the session, and were only
cleared the next time route_to was called. To ensure the exceptions have the
same lifetime as the call to route_to and no more, pass an error_handler
callback to the route_to method. The callback will be called once per
exception that occurs during the resolution process.

CLA signed by all contributors.

I did not test this, but it looks legit

jenkins please test this with servertests

One of the things I wanted to ensure was that all errors are being collected. In my testing, it is. But I'm wondering if it would be worthwhile to add a test for this?
    it "raises when there are no more routes" do
      Puppet[:ca_server] = 'foo.example.com'
      Puppet[:use_srv_records] = true
      allow_any_instance_of(Puppet::Network::Resolver).to receive(:each_srv_record).and_yield('bar.example.srv', 8140)

      stub_request(:any, /example.com/).to_return(status: 503)
      allow_any_instance_of(Net::HTTP).to receive(:start).and_raise(Errno::EHOSTUNREACH)
      session = client.create_session

      expect {
        session.route_to(:ca)
      }.to raise_error(Puppet::HTTP::RouteError, 'No more routes to ca')
    end

➜  puppet git:(use_resolver_callback) ✗ be rspec spec/unit/http/session_spec.rb:158
Run options:
  include {:locations=>{"./spec/unit/http/session_spec.rb"=>[158]}}
  exclude {:broken=>true, :benchmark=>true}

From: /Users/melissa/workdir/puppet/lib/puppet/http/session.rb @ line 80 Puppet::HTTP::Session#route_to:

    75:         Puppet.debug("Resolved service '#{name}' to #{service.url}")
    76:         return service
    77:       end
    78:     end
    79:
 => 80:     require 'pry'; binding.pry
    81:     resolution_exceptions.each { |e| Puppet.log_exception(e) }
    82:     raise Puppet::HTTP::RouteError, "No more routes to #{name}"
    83:   end
    84:
    85:   #

[1] pry(#<Puppet::HTTP::Session>)> resolution_exceptions
=> [#<Puppet::HTTP::ConnectionError: Request to https://bar.example.srv:8140/puppet-ca/v1 failed after 0.0 seconds: No route to host>,
 #<Puppet::HTTP::ConnectionError: Request to https://foo.example.com:8140/puppet-ca/v1 failed after 0.0 seconds: No route to host>]

The only thing I didn't do here was add an expectation that there are exactly two connection errors in resolution_exceptions that are logged. I'm not sure if that's an important thing to test though.


The "classifiers" imported in Main.py does not exist.

Hello, can you provide Disjoint-CNN network source code? The "classifiers" you mentioned in Main.py "from classifiers import DCNN_2L" does not exist and cannot run code.

Thank you for raising this issue. I apologize for the confusion caused by the missing "classifiers" in the Main.py file. We have addressed this issue by adding the classifiers to the repository. You can now find the Disjoint-CNN network source code, along with other models such as FCN, ResNet, and MC-CNN, for multivariate time series classification.
Please feel free to explore the repository and let us know if you have any further questions or need assistance.


can not compile successfully

when I run CLI "./gradlew distDocker", it can not run successfully, the error is as below:

./gradlew distDocker

The Task.leftShift(Closure) method has been deprecated and is scheduled to be removed in Gradle 5.0. Please use Task.doLast(Action) instead.
at docker_7r8ejptnt4f4tnx46yncyk4ag.run(/home/zhy/iot/incubator-openwhisk-package-alarms/gradle/docker.gradle:43)
FAILURE: Build failed with an exception.

Where:
Build file '/home/zhy/iot/incubator-openwhisk-package-alarms/tests/build.gradle' line: 28

What went wrong:
A problem occurred evaluating project ':tests'.

Could not get unknown property 'sourceSets' for project ':whisktests' of type org.gradle.api.Project.

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED
Total time: 1.51 secs

You need to set the OPENWHISK_HOME env variable and run gradlew distdocker as gradlew :distdocker

@jasonpet  I set the  OPENWHISK_HOME env variable in file /etc/profile,  but the error is still there:

./gradlew :distdocker
Starting a Gradle Daemon (subsequent builds will be faster)
The Task.leftShift(Closure) method has been deprecated and is scheduled to be removed in Gradle 5.0. Please use Task.doLast(Action) instead.
at docker_7r8ejptnt4f4tnx46yncyk4ag.run(/home/zhy/iot/incubator-openwhisk-package-alarms/gradle/docker.gradle:43)

FAILURE: Build failed with an exception.

Where:
Build file '/home/zhy/iot/incubator-openwhisk-package-alarms/tests/build.gradle' line: 28

What went wrong:
A problem occurred evaluating project ':tests'.

Could not get unknown property 'sourceSets' for project ':whisktests' of type org.gradle.api.Project.

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED
Total time: 11.0 secs

I download the openwhisk code to  /home/zhy/incubator-openwhisk
and add "OPENWHISK_HOME=/home/zhy/incubator-openwhisk" at the end of the file /etc/profile
"source /etc/porfile"
then I run "./gradlew distDocker", the error is the same as before.

@jasonpet
I delete the line  "compile project(':whisktests').sourceSets.test.output" in the file "incubator-openwhisk-package-alarms/tests/build.gradle"    and  build the docker image  "catalog_alarms " successfully.
then I try to run sample helloworld with alarm trigger . After I create the action trigger and rule  successfully,then run the cli "wsk -i activation pull", there is no result to output to the screen

wsk -i action list | grep hello
/whisk.system/hello                                                    private nodejs:6

wsk -i trigger list
triggers
/whisk.system/every-20-seconds                                         private

wsk -i rule list
rules
/whisk.system/invoke-periodically                                      private

How do I check the reason for no result to print out to the screen?

I assume you meant activation poll.  Did you use the --feed parameter when creating your 20 second trigger?  Did you start the "catalog_alarms" docker container after building it?

Please see https://github.com/apache/incubator-openwhisk-package-alarms/issues/51 if you need help with installing the alarms actions or starting up the alarms container.  This issue can be closed.  The compilation issue is due to the fact that your OPENWHISK_HOME variable is not set correctly.  If you are not trying to run tests then removing the line you mentioned above is fine.

@jasonpet thanks

wsk trigger create every-20-seconds --feed /whisk.system/alarms/alarm --param cron "*/20 * * * * *" -i
ok: invoked /whisk.system/alarms/alarm with id c6e0085cbc6e4c22966861487ff396ac
{
"activationId": "c6e0085cbc6e4c22966861487ff396ac",
"annotations": [
{
"key": "limits",
"value": {
"logs": 10,
"memory": 256,
"timeout": 60000
}
},
{
"key": "path",
"value": "whisk.system/alarms/alarm"
}
],
"duration": 1007,
"end":<PHONE_NUMBER>530,
"logs": [],
"name": "alarm",
"namespace": "whisk.system",
"publish": false,
"response": {
"result": {},
"status": "success",
"success": true
},
"start":<PHONE_NUMBER>523,
"subject": "whisk.system",
"version": "0.0.6"
}
ok: created trigger every-20-seconds

I run this script to start up the  alarms container

cat docker_run.sh

WSK_HOME=/openwhisk
DB_PROTOCOL=awk -F "=" '/db_protocol/ {print $2}' $WSK_HOME/ansible/db_local.ini
DB_HOST=awk -F "=" '/db_host/ {print $2}' $WSK_HOME/ansible/db_local.ini
DB_PORT=awk -F "=" '/db_port/ {print $2}' $WSK_HOME/ansible/db_local.ini
DB_USER=awk -F "=" '/db_username/ {print $2}' $WSK_HOME/ansible/db_local.ini
DB_PWD=awk -F "=" '/db_password/ {print $2}' $WSK_HOME/ansible/db_local.ini
DB_PREFIX=test
docker run -d -p 32691:8080 -v $HOME/wsklogs/alarms:/logs
-e PORT=8080 -e ROUTER_HOST=<IP_ADDRESS>
-e DB_PREFIX=${DB_PREFIX} -e DB_USERNAME=${DB_USER}
-e DB_PASSWORD=${DB_PWD} -e DB_HOST=${DB_HOST}:${DB_PORT}
-e DB_PROTOCOL=${DB_PROTOCOL} whisk/catalog_alarms

docker ps | grep alarm
243523e5ad17        whisk/nodejs6action:latest   "/bin/sh -c 'node --e"   About a minute ago   Up About a minute                                                                                wsk0_36_whisksystemalarm
bf1286ae2427        whisk/catalog_alarms         "/bin/bash -c 'node /"   45 hours ago         Up 45 hours         <IP_ADDRESS>:32691->8080/tcp                                                                                agitated_engelbart

@csantanapr - can you please close this?


DSD-889: Adding a11y docs for navigation components

Fixes JIRA ticket DSD-889
This PR does the following:

Adds table of contents and accessibility documentation for the Breadcrumbs, Link, and Pagination components.

How has this been tested?
Storybook.
Accessibility concerns or updates

More docs.

Checklist:

[ ] I have updated the Storybook documentation accordingly.
[ ] I have added relevant accessibility documentation for this pull request.
[ ] All new and existing tests passed.

Front End Review:

[ ] View the example in Storybook

Tugboat has finished building the preview for this pull request!
Link:

https://pr955-3oo8dlwzj0q3zed9jqjo0m3ncc8ex2b8.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/62587a5e492bf2bdb4f2f553

Tugboat has finished building the preview for this pull request!
Link:

https://pr955-3oo8dlwzj0q3zed9jqjo0m3ncc8ex2b8.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/62587a5e492bf2bdb4f2f553


Org Name not populating on Nav Bar

When signing up for an org account, got a message back saying "Something went wrong"
However, was able to sign in.  Most functionality seems to be present; however, the account name does not update on Nav Bar.
Switched to a different account that is a full org, was able to change the name and have that be properly reflected in the Nav Bar.

Error appeared with signing up: Firebase Error: Missing or insufficient permissions.

Updating the firestore.rules at line 42 to allow create: if validUser() && (request.resource.data.role == 'user' || request.resource.data.role == 'organization') && request.resource.data.public == false Prevents the error from appearing during sign up. But the missing name is still there unless the organization becomes verified.


#30 - Added script to bulk import notifications from UI into Monte Carlo YAML template

We can create Notifications from UI directly as well as YAML. However, for accounts that already have a lot of notifications created from UI, it can be tedious to have to manually create the YAML for each notification. This PR aims to provide a script that, when executed with the MCD_DEFAULT_API_ID and MCD_DEFAULT_API_TOKEN will automatically generate a Monte Carlo compatible YAML file, which can then be deployed using the montecarlo monitors apply command.
Prerequisites to run the code -
Export MCD_DEFAULT_API_ID and MCD_DEFAULT_API_TOKEN values in the ~/.zshrc or ~/.bashrc files for Mac or Linux respectively. Alternately for Windows, these two credentials can be added to the environment variables.

@lgavish @bheltzel @glencwillis @melissa-hale Request your review comments on this PR


Editing Deployment image names via "Edit Config" results in failure

When editing a deployment's image name, via "Edit Config" errors occur. However, when editing it via "Edit YAML", no errors occur.
Error that appears in UI:
Deployment.apps "fleet-agent" is invalid: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms: Required value: must have at least one node selector term

Changing this one field appears to create a large diff as well when using "Edit Config". One field is offset too:

@Sean-McQ Check with @rak-phillip - I think he looked at a similar issue elsewhere (I could be wrong)

There are additional reports about this happening. Internal reference: SURE-3676.

From the internal reference, there's a comment in place tied to preference that may be the culprit.

Was unable to reproduce and after a brief conversation with @mantis-toboggan-md  it looks like this was previously resolved in 2.6.3 under https://github.com/rancher/dashboard/pull/4319

Reproduction
Rancher version: 2.6.3
Rancher cluster type: single-node docker install
Downstream cluster type: rke2 ec2 node driver
Downstream K8s version: 1.21.9

Workloads -> Deployments -> Create
Name: test
Container name: test
Image: nginx
Node Scheduling: Run pods on node(s) matching scheduling rules
-Priority: Preferred
-Key: node-role.kubernetes.io/worker
-Operator: is set

Test Environment:
Rancher version: v2.6-head 4df2214
Rancher cluster type:  HA
Docker version: 20.10
Downstream cluster type: rke1 ec2 node driver
Downstream K8s version: 1.23.4
Testing:
Edit Config for:

deployment with Node Scheduling -> Run pods on nodes(s) matching scheduling rules -> Preferred (affinity)
deployment with Node Scheduling -> Run pods on nodes(s) matching scheduling rules -> Required (affinity)
deployment with Node Scheduling -> Run pods on specific node(s) -> Select node (nodeName)

All deployments are updated successfully with pods recreated


Broken <def[hit_entities]> in 1.14.4

/version git-Paper-187 (MC: 1.14.4) (Implementing API version 1.14.4-R0.1-SNAPSHOT)
/version denizen version: 1.1.0-SNAPSHOT (build 4122-DEV), core version: 1.50-SNAPSHOT (Build 372)
Debug: https://one.denizenscript.com/paste/58326
Script: https://one.denizenscript.com/haste/58327
The tag <def[hit_entities]> is broken in 1.14.4
It works in in 1.13.2 tested with same denizen version

This looks like you just didn't hit anything? The tag returned an empty list.

Video and referenced debug:
https://youtu.be/8hBysVJfHeg
https://one.denizenscript.com/paste/58389
After the Video I also updatetd both server to build 4136 you can see the debug vom the shooting there, same result...
https://one.denizenscript.com/paste/58390

You are very clearly not hitting your targets (notice how they don't react at all?)
I don't think item_projectile can hit targets in the first place actually. You'd need an actual weapon projectile, like snowball or arrow.


Optionally swap redis for logstash persistent queue

Now that logstash supports persistent queues, operators might not need to use redis--especially because the redis queue used here is a single point of failure. This might be as simple as configuring the current ingestor job to output directly to elastic. If so, would you all be interested in a patch with updated templates that drop the queue and parser jobs and configure the ingestor to use persistent queues and output to elastic?
cc @wjwoodson @cnelson

Added in https://github.com/cloudfoundry-community/logsearch-boshrelease/commit/f3c7b61b2fc4a1910af1e33932053333d58c7512^^.
I have moved redis job to logsearch-for-cloudfoundry repo since it's no longer needed in current repo. Will keep this issue opened for some time in case I missed something.


impl Index and IndexMut

Codecov Report
Patch coverage: 60.00% and project coverage change: -0.16 :warning:

Comparison is base (2c3864f) 35.40% compared to head (74cf2af) 35.24%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master      #20      +/-   ##
==========================================
- Coverage   35.40%   35.24%   -0.16%
==========================================
  Files           8        9       +1
  Lines        1045     1044       -1
==========================================
- Hits          370      368       -2
- Misses        675      676       +1

Impacted Files
Coverage Δ

src/index.rs
0.00% <0.00%> (ø)

src/lib.rs
14.55% <ø> (+0.01%)
:arrow_up:

src/map.rs
88.97% <100.00%> (+1.11%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

@Zefick the coverage dropped, we won't be able to merge this way. Can you add more tests to cover the functionality better?

@rultor merge

@rultor merge

@yegor256 OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge

@yegor256 Done! FYI, the full log is here (took me 3min)


Promisified Prediction API.

The purpose of this PR is to show how to promisify the gcloud-node API while still maintaining support for callbacks. See #551
npm test succeeds for me. I did not try the system tests (but I can, depending on how serious we are about this pull request).
Summary of changes
Breaking API changes

Service#request, the Prediction API methods, and the ServiceObject methods now only return a single errback value (which is more idiomatic, as opposed to also passing random positional success arguments to the callback). The positional argument (e.g. apiResponse) which was previously passed to the callback as a separate argument is now attached as a property of the single err argument which is now the only value returned in the event of an error/rejection. Now, in the event of an error/rejection, err is all you need to inspect.

Backwards compatible API changes

Service#request now returns a promise, but makes use of .asCallback to maintain support for callbacks.
The Prediction API methods now return promises, which make use of .asCallack to maintain support for callbacks.
The ServiceObject methods now return promises, which make use of .asCallback to maintain support for callbacks.
Added dependency on bluebird

Other

Refactored Prediction API and ServiceObject unit tests, which frequently mock Service#request, to take into account the new promisified methods.
Made a second version of a number of the refactored tests. One version interacts with the API via promises, and the other via callbacks, to show that the API now supports both styles.

Note: Many methods return multiple values. If you use callbacks, the multiple values will still be passed to your callback as multiple positional arguments. But if you use Promises, .then will receive an array containing the return values, so make use of .spread instead.
Note 2: My first commit did not do anything about streams, which still need to be addressed.
Note 3: I'm not sure I completely promisified ServiceObject, as I haven't spent enough time studying the code. In fact I'm sure I didn't promisify it completely, because I would have expected to have needed to refactor more tests than I did.

Awesome, thanks for doing this!

@jmdobry thanks for taking the time to put this together! I started to implement Promises via https://github.com/GoogleCloudPlatform/gcloud-node/pull/1142, but with the recent decision to continue callback support I was a little thrown off/set back.
@stephenplusplus and I have been having a lot of discussions about promises and how we should go about implementing them into gcloud-node. One thing we decided on was not to use Bluebird, but instead a lightweight polyfill, although .spread() does make Bluebird very appealing.

Thanks again for the PR @jmdobry. As @callmehiphop said, he's working on adding support in #1142 -- we'll be sure to come back and reference the changes here as we go.


Create a default configuration for debugging tests.

Requesting to reopen https://github.com/microsoft/vscode-python/issues/17150. The issue is still present in v2022.4.1, and as the ticket is locked I cannot comment on it directly.

@scottshambaugh Can you elaborate? if you have "request": "test" only then you will see the message:
'This configuration can only be used by the test debugging commands. `"request": "test"` is deprecated use "purpose" instead.'

When creating the default configuration use this template:
{
            "name": "Debug Tests",
            "type": "python",
            "request": "launch",
            "purpose": ["debug-test"],
            "justMyCode": true
}

Yes I've been able to figure out what I should be using. My request is the exact same as the linked issue, ie that the error message be changed to something more helpful:

I don't mind changing my launch.json file. However, in order to comply with the error message and "use 'purpose' instead," I need to know the precise syntax to use. The phrase "use purpose" doesn't tell me if that should be the key or the value in the json - should it be "purpose": "test" or should it be "request": "purpose"? Actually, neither one of these work. Both throw an error.

Is this new syntax ("purpose": ["debug-test"]) documented somewhere? That was the big issue for me...the error message told me I had something misconfigured but did not let me know what the valid options are, or point me to some documentation where the new options are listed.

So instead of the error message being "This configuration can only be used by the test debugging commands. '"request": "test"' is deprecated use "purpose" instead.", I would ask that it be changed to something like "This configuration can only be used by the test debugging commands. '"request": "test"' is deprecated, please keep as "request": "launch" and add "purpose": ["debug-test"] instead."


tmux-xpanes: init at 4.1.0

Built on platform(s)

[ ] NixOS
[ ] macOS
[x] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Assured whether relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

:heavy_check_mark:

:heavy_check_mark:

@GrahamcOfBorg build tmux-xpanes

@grahamc: Hi! Could you take a look at this PR if you get round to it? Thanks! :)

@GrahamcOfBorg build tmux-xpanes


vrtnu subtitles aren't downloaded

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.01.21. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2018.01.21

Before submitting an issue make sure you have:

[x] At least skimmed through the README, most notably the FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[x] Feature request (request for a new functionality)
[ ] Question
[ ] Other

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

Single video: https://www.vrt.be/vrtnu/a-z/eigen-kweek/seizoen-2/coloradokevers/
Single video: https://www.vrt.be/vrtnu/a-z/blue-planet-ii--nederlandstalig-/2017/blue-planet-ii--nederlandstalig--s2017a4-big-blue/

Description of your issue, suggested solution and other information
Downloading video's works perfect. But subtitles are available with the video's on the vrt website. I've tried every possible command for downloading these subtitles but only the video gets downloaded.
Hopefully this is possible to implement.
The python file where the vrtnu download is implemented is called canvas.py

Could you link to the duplicate report, so that we can subscribe there? Thanks!

Search: webvtt.


DBZ-3992, DBZ-4273: Improve capture instance instrospection logic

This pull request contains fixes for DBZ-3992 and DBZ-4273. Although the two issues are unrelated, fixing them requires changing the same code, so it makes sense to combine their fixes together.
The root cause of DBZ-3992 (SQL Server fails to read CDC events if there is a schema change ahead) is that the connector tries to use all available capture instances for a table, even though the newest one doesn't contain the changes that fall into the currently processed LSN interval (only applicable if max.iteration.transactions is configured).
CI  |                                                               LSNs
------------------------------------------------------------------------->
ci1 |  A                                                            NULL
    |  |----------------------------------------------------------------
    |
ci2 |           B                                                   NULL
    |           |-------------------------------------------------------
    |
ci3 |                                  C                            NULL
    |                                  |--------------------------------
    |                ^-----------^
    |                X           Y

ci1, ci2, ci3 : capture instances
X             : "from" LSN
Y             : "to" LSN
[X, Y]        : current LSN interval with size of "max.iteration.transactions"

Here we expect only ci1 and ci2 capture instances to be selected but we get all three. The solution is to consider the capture instance's start LSN in the GET_LIST_OF_CDC_ENABLED_TABLES query.
Additionally (DBZ-4273), the connector doesn't expect multiple capture instances with the same start LSN for the same schema. In this case, the behavior of the connector is undefined.
Key changes
Do not use cdc.change_tables.end_lsn
According to the documentation, it's always NULL. The code still uses SqlServerChangeTable#stopLsn and its accessors in order to artificially set the stop LSN of the current capture instance to the start LSN of the future one. It would make sense to rework this logic but this is out of the scope of this pull request.
Use cdc.change_tables instead of sys.sp_cdc_help_change_data_capture
Despite the documentation suggesting to use the stored procedure, there are reasons to use the table:

The stored procedure doesn't allow filtering capture instances by start LSN.
There doesn't seem to be an easy way to implement a query like SELECT * FROM (EXEC stored_procedure).

Additionally, the call to the stored procedure is much more expensive than the SQL query. This difference is insignificant given that schema changes are rare but is worth mentioning.
Deduplicate capture instances by create_date
There are multiple places in the code where capture instances are compared by the start LSN. Adding comparison by the creation date is possible but would require changes in multiple places and result in the oldest one being discarded anyway. It's easier to filter it out at the query level.
Testing considerations

The changes for DBZ-3992 are covered by new tests. Additionally, now the entire test suite passes with MAX_TRANSACTIONS_PER_ITERATION set to 1.
Automated testing of DBZ-4273 is challenging. First, the setup procedure is time-consuming since it depends on periodic background jobs owned by the SQL Server. Second, the issue is not always reproducible, so a passing test won't mean that the issue doesn't exist. Automated testing might be reconsidered if the data structures for managing capture instances are redesigned. E.g., instead of HashSet<SqlServerChangeTable> a more specialized structure with ordering guarantees is used.

Thanks a lot, @morozov! This looks great overall. A few comments inline; mostly about cosmetics, but the unused parameter is a bit troublesome. Could you also rebase this, so as to resolve the merge conflict?

According to the documentation, it's always NULL. The code still uses SqlServerChangeTable#stopLsn and its accessors in order to artificially set the stop LSN of the current capture instance to the start LSN of the future one. It would make sense to rework this logic but this is out of the scope of this pull request

Ok, that makes sense. Could you log a follow-up issue though for removing this eventually?
Thanks again!

Could you log a follow-up issue though for removing this eventually?

Logged as DBZ-4358.

And applied, thanks again, @morozov!