feat(v2): Use AWS CRT as the default SDK client

#1092
Description of changes:
This PR changes the default SDK HTTP client from UrlConnectionClient to the AWS CRT client. The CRT client is a lightweight client which should reduce the cold start and latency of lambda functions.
Checklist

[ ] Meet tenets criteria
[ ] Update tests
[ ] Update docs
[ ] PR title follows conventional commit semantics

Breaking change checklist

RFC issue #:

[ ] Migration process documented
[ ] Implement warnings (if it can live side by side)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Thanks @jreijn, looks great! Thanks to the new sync API, it's not so much change, that's a good news.
Question/ask: could you do a load test on the parameter sample (maybe the easiest to do) before and after CRT? Feel free to say no and I'll do it later.

Build is failing, problem in DynamoDBWriter in batch module.

@jeromevdl thanks! Yes the change is quite simple now. I'll look into the issues. I've just setup a dev linux box to see if I can run the tests locally as it seems to fail on my MacBook book due to some docker issue.

@jeromevdl thanks! Yes the change is quite simple now. I'll look into the issues. I've just setup a dev linux box to see if I can run the tests locally as it seems to fail on my MacBook book due to some docker issue.

It's the test in the CDK example, which is autogenerated and doesn't really test anything. I'd like to remove it in #1542

Thanks @jreijn, looks great! Thanks to the new sync API, it's not so much change, that's a good news.
Question/ask: could you do a load test on the parameter sample (maybe the easiest to do) before and after CRT? Feel free to say no and I'll do it later.

@jeromevdl Do we already have something in place for doing the load test? Or should I just run something on my own?

@jeromevdl I've ran a load test with artillery on the parameters module and the results are mentioned below.
I've used the parameters example with 1 slight improvement (tiered compilation) against the current v1 (main) branch with the connection client and this PR branch with the aws crt client. It's using 512MB. I've created two reports:

Duration
Memory

I used CW Logs insights with the following 2 queries:
filter @type="REPORT"
| fields greatest(@initDuration, 0) + @duration as duration, ispresent(@initDuration) as coldStart
| stats count(*) as count, pct(duration, 50) as p50, pct(duration, 90) as p90, pct(duration, 99) as p99, max(duration) as max by coldStart

filter @type = "REPORT"
| stats max(@memorySize / 1000 / 1000) as provisionedMemoryMB,
  min(@maxMemoryUsed / 1000 / 1000) as smallestMemoryRequestMB,
  avg(@maxMemoryUsed / 1000 / 1000) as avgMemoryUsedMB,
  max(@maxMemoryUsed / 1000 / 1000) as maxMemoryUsedMB,
  provisionedMemoryMB - maxMemoryUsedMB as overProvisionedMB

Duration
For the main branch (v1) I get the following results:

coldStart
count
p50
p90
p99
max

0
17909
3.1752
4.9216
31.5072
12609.12

1
91
3670.57
6515.75
7931.03
7931.03

For the v2 crt branch I get the following results:

coldStart
count
p50
p90
p99
max

0
17902
3.125
4.7294
24.0299
12774.03

1
98
4634.31
8409.65
9343.73
9343.73

Memory
v1

provisionedMemoryMB
smallestMemoryRequestMB
avgMemoryUsedMB
maxMemoryUsedMB
overProvisionedMB

512
107
117.3101
183
329

v2 with crt

provisionedMemoryMB
smallestMemoryRequestMB
avgMemoryUsedMB
maxMemoryUsedMB
overProvisionedMB

512
117
128.9981
188
324

I expected different results and I noticed that the example was not functioning due to some changes in the v2 branch which I fixed in a commit. There are still some messages during a coldstart for the LambdaJsonLayout of the log4j configuration and a mention about a missing version, which I will try to look at somewhere over the next couple of days.
I also see that the parameters example is using an amazon API call to https://checkip.amazonaws.com. Not sure if that impacts the results, but will try to analyse further.

Thanks for the tests. I didn't expect such results... It's disappointing. @msailes @scottgerring any thoughts? Do you think the asynchronous version would be better?

I don't think this is expected, one to ask Zoe Wang, when she's next working.

I've also tested with the sync crt client in the v1 branch and I see similar behavior. It makes me wonder about the impact compared to the url connection client. In the announcement post on the aws blog for the crt client we see the comparison to the netty client which is a more heavy weight client. Looking forward to some insights from the team that worked on the aws crt client.

@jreijn thanks for the extensive testing!
I think we must be missing something.  We will loop Zoe in once she's back on deck after the break. Failing that we probably need to  hook up Codeguru to it and get some profiling data out.

@scottgerring great suggestion for code guru. I'll see if I can find some over the next week to hook it up and find out what I can see about it.

It seems like this PR will also resolve #1544 !

I've used this as an opportunity to have a play with CodeGuru.
V2 current branch - No CRT Client - Java 11 Runtime

I'll post the CRT version once it's bubbled through

It seems like this PR will also resolve #1544 !

Yes. I wasn't sure if the issue was known so solved it in this PR for now.

It seems like this PR will also resolve #1544 !

Yes. I wasn't sure if the issue was known so solved it in this PR for now.

it wasn't - I found it getting parameters ready for codeguru testing, then realized you'd already resolved it when I pulled in your changes!

I'm running some profiling using this change on top of your branch now - I think this should be more interesting! I've moved all of the SDK calls into the request handler itself, and removed the HTTP website call.
https://github.com/aws-powertools/powertools-lambda-java/commit/bb1aa76998bfe356b3aefff481166bdd04ba9db8
A big part of the CRT advantage is the cold-start time; i'm not sure how we can measure this effectively. I don't think CodeGuru is very helpful, as it randomly samples after some time.

CRT client, chore-perf-test-crt branch
Again, in retrospect I don't think this is actually that useful. We care about the impact of cold start. But I am pasting it here for completeness and in case someone else notices something!

Thanks for your tests guys, I agree we should have an independent test (not querying a 3rd party website/API). Not sure how to read these diagrams... Let's see what Zoe has to say. @scottgerring did you ping her already?

Thanks for your tests guys, I agree we should have an independent test (not querying a 3rd party website/API). Not sure how to read these diagrams... Let's see what Zoe has to say. @scottgerring did you ping her already?

I have, but I can see she's still marked OOO, so we should expect some latency there.
For a "better test" we can use the branch I linked the commit to above, taken from Jeroen's branch --> https://github.com/aws-powertools/powertools-lambda-java/compare/v2...chore-perf-test-crt

@jreijn / @jeromevdl in the meantime, I suggest we do something like this:
1/ agree on a more indicative test. How do you both feel about this commit I pushed on the branch I referenced? It removes the call to the external website, and initializes the CRT client in the handler itself (so it happens per-request).
2/ Re-use Jeroen's existing cloudwatch cloudstart/warm-start based analysis process above with this test for v2 and the crt branches. I think this actually captures exactly what we want to see, but calling out to the external website muddies the waters.
At this point we'll have an idea on what the performance impact is for both cold/warm scenarios.
What do you both think?

Hey @jreijn sorry for the lag on this. I wanted to let you know that Zoe's looking at this and we'll loop back shortly.
It seems fairly likely the issue is the test we're using.

Thanks @scottgerring . I'll keep an eye on the issue. I'm still enjoying some time off so if something changes I'll chime in again.

@jreijn / @jeromevdl in the meantime, I suggest we do something like this:
1/ agree on a more indicative test. How do you both feel about this commit I pushed on the branch I referenced? It removes the call to the external website, and initializes the CRT client in the handler itself (so it happens per-request).
2/ Re-use Jeroen's existing cloudwatch cloudstart/warm-start based analysis process above with this test for v2 and the crt branches. I think this actually captures exactly what we want to see, but calling out to the external website muddies the waters.
At this point we'll have an idea on what the performance impact is for both cold/warm scenarios.
What do you both think?

@scottgerring i just noticed I missed this comment.
With regards to:
1 I fully agree. The remote call makes the test less precise due to the latency from the remote call.
2 sounds like a plan. If you want me to incorporate the changes and try again let me know!

Hi @jreijn, I'm on the AWS SDK for Java team 🙂. We performed tests to compare HTTP clients coldstart latency on Lambda and from our tests, switching to AWS CRT sync HTTP client from UrlConnectionHttpClient reduced 35% startup latency. AWS CRT sync has slight higher startup latency (6%) compared to AWS CRT async HTTP client and we'll look into that. The Lambda memory setting was 512MB and we were just sending one single ListBuckets request with S3 SDK. LMK if you have any questions and I'm interested to see the new test result!

Thanks @zoewangg ! I'll try to make some time today or tomorrow to take it for another test run and see if we can see similar results. Was there a specific java version you used in your tests?

I was using Java 8.

@scottgerring i hope you had some wonderful time with the family. I don't mind spending some more time, but it's good to share experiences. I did a test 2 days ago where I incorporated most of your changes (from the branch) and ran a new test. The results did not differ that much and I'm planning to do another run tomorrow before I post the results here just to check for any mistakes I might have made. I'll send an update tomorrow!

Hey @jreijn sounds good (although a bit surprised about the results). Looking forward to seeing the numbers.

I did a retest today to see the difference. I've incorporated some your changes @scottgerring into this PR for clarity. I'm a bit surprised by the memory results.
Duration v2-with-crt-client (AWS SDK 2.22.0)

coldStart
count
p50
p90
p99
max

0
2950
1.7087
2.3787
7.0289
122.65

1
50
4555.15
4711.33
4802.67
4802.67

Memory usage (AWS SDK 2.22.0)

provisionedMemoryMB
smallestMemoryRequestMB
avgMemoryUsedMB
maxMemoryUsedMB
overProvisionedMB

512
173
177.8377
179
333

Duration (AWS SDK 2.23.9)

coldStart
count
p50
p90
p99
max

0
2745
1.5186
2.1981
6.2845
42.02

1
31
3069.25
3247.63
3306.67
3306.67

Memory (AWS SDK 2.23.9)

provisionedMemoryMB
smallestMemoryRequestMB
avgMemoryUsedMB
maxMemoryUsedMB
overProvisionedMB

512
169
173.5416
175
337

I'll try to create a heap dump of the JVM to analyse the what's actually in the heap and post an update later.

Hey @jreijn great - thanks for the dive deep!

I did a retest today to see the difference.

So .... we can see the remote call latency is down, but the overall picture hasn't really changed - the CRT is still slower both with and without cold starts. Memory usage seems similar.
We'll reach out to the team again on our side, it still feels like we are missing something!
Would you be able to give us the exact lambda configuration you are using - region, runtime, size?

I've analyzed a heap dump of about (28 MB). The amount of live objects is very small. Not sure how this relates to the 169 MB reported by Lambda itself. @scottgerring @jeromevdl any ideas?

I believe the lambda-reported usage will be the "OS view" and include the java runtime as well, not just the JVM-allocated heap. I would guess this accounts for the discrepancy.

@jreijn , @jeromevdl : I think i've gotten to the bottom of it! It's the size of the output JAR; using the CRT client by default includes a pile of sos for every platform it supports, which results in a function JAR that is 2.3x the size of the non-CRT jar.
Detailed numbers here,  testing bits here (i'll clean this up a bit so others can use it)
Stripping back (removing all the shared objects for unneeded platforms) the JAR to the minimum reduces cold start time below the URLHttpClient:

CRT-Stripped Avg: 4.45.    (lambda size = 15.9 MB)
CRT Normal Avg: 6.19,      (lambda size = 32.73 MB)
URL Http Client: 6.49       (lambda size = 13.915 MB)

CRT-normal and URL client are very close to the same.
I note that in the maven repo, there are jar variants for all the different combos of arch/runtime. Turns out - you can use the "classifier" to get the right one - details here.
I haven't played with this yet, but it seems like there will be an onus on us to document this clearly in Powertools as part of the "getting started" guide - otherwise simply including the CRT as the default means that v2 may be slower than v1 because of all the unneeded libraries.

@scottgerring thanks for your great tests.
Looking at the jar sizes in maven, it's clear that the generic crt is quite big compared to the others 17 Mb vs ~2 Mb for the specific versions. Did you test one of these specific versions ?
If I'm not mistaken, we need either:
<dependency>
    <groupId>software.amazon.awssdk.crt</groupId>
    <artifactId>aws-crt</artifactId>
    <version>0.29.10</version>
    <classifier>linux-aarch_64</classifier> <!-- arm64 -->
</dependency>

Or:
<dependency>
    <groupId>software.amazon.awssdk.crt</groupId>
    <artifactId>aws-crt</artifactId>
    <version>0.29.10</version>
    <classifier>linux-x86_64</classifier>  <!-- x86_64 -->
</dependency>

Problem is that we can't set this in Powertools, so we need to document it... In PT, we can use
<dependency>
    <groupId>software.amazon.awssdk.crt</groupId>
    <artifactId>aws-crt</artifactId>
    <version>0.29.10</version>
    <scope>provided</scope>
</dependency>

If I'm not mistaken, we need either:

no directly; i recreated a specific variant by hand by stripping the "full" JAR, before I realised you can use the classifier. We need to test that and make sure it works. I think the only trick is going to be ensuring that maven does what we think it will!

I think the only trick is going to be ensuring that maven does what we think it will!

Maven will do what it's supposed to do if you specify the classifier... Test it but it should work.
Wondering if we could simplify this somehow, rather than just documenting... We'll build a powertools-maven-plugin that reads in your IaC to evaluate the architecture, the target java runtime and choose the best dependencies ;)

@scottgerring wow nice find! I was curious if the package size had any impact after reading this post, but I did not get to it yet.
In my local branch I made some changes and marked the aws-crt-client in the PT parameters module to scope provided  ike @jeromevdl suggested and included the aws-crt-client and the aws-crt library the with classifier in the example project, but I was not very pleased by the developer experience.
This is mainly because the following observations:

you need to explicitly set the aws http client in your project (parameters example) by adding the dependencies . Of course we can document this, but it's not a seamless experience with just adding a single dependency (pt-module) to a project and setting the aspectj plugin.
you need to make sure you keep the version of the aws-crt library in combination with classifier in sync with the version used by the aws-crt-client. In your project you will need to exclude the aws-crt transitive dependency from the aws-crt-client like and include the correct aws-crt dependency (runtime specific):

        <dependency>
            <groupId>software.amazon.awssdk.crt</groupId>
            <artifactId>aws-crt</artifactId>
            <version>0.29.9</version>
            <classifier>linux-x86_64</classifier>
        </dependency>

        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>aws-crt-client</artifactId>
            <version>2.23.21</version>
            <exclusions>
                <exclusion>
                    <groupId>software.amazon.awssdk.crt</groupId>
                    <artifactId>aws-crt</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

otherwise you can/will end up with both versions on the class path (I noticed this after the sam build).

If your local environment is different from the target runtime environment this will become a bit of an hassle and you will probably need multiple maven profiles to be able to locally test your function. (I have a Macbook) and the lambda is targeted to run on linux.

It makes me wonder if we should use the CRT client as the default client, as overall the developer experience will be less. Perhaps it's better to leave the url connection client as the default as it offers quite a good performance over dev experience . We could add documentation on how to use the CRT client for the local/target runtime for the users that want to try and squeeze the most out of their function, but it will be an explicit decision made by the developer to take on the extra burden of the local development vs remote and the dependency synch.
Now for some good news.
I did another run with the classifier and I see some very promising results compared to the previous benchmark(s).
The following table is for the v2 crt client with classifier and latest version of the sdk (2.23.21) and crt library (0.29.9) :

coldStart
count
p50
p90
p99
avg
min
max

0
2967
1.5493
2.1589
5.5797
1.7932
1.07
40.57

1
33
3170.56
3310.47
3433.73
3097.1491
2507.71
3433.73

Thanks @jreijn. I agree the devX is quite bad and we probably should not do this by default but document it so that people who want better performance can do it...
@scottgerring @msailes wdyt?

Yeah, the developer experience isn't great. I didn't even consider this:

probably need multiple maven profiles to be able to locally test your function. (I have a Macbook)

... plus you often want flexibility to mix ARM64/AMD64 on the AWS side.
Considering our tenets 1/ Keep it lean and 2/ Work backwards from the community ...  I think it is clear we should document how to use the CRT client (including what we've learnt here in terms of JAR-size minimization) and 2/ not use the CRT by default.
Open question: Is UrlConnectionHttpClient the best choice for us? Looking at this flowchart it seems like it is - minimize latency over throughput, minimize bundle size - but I think now is a good opportunity to make sure we're all on the same page @jreijn @jeromevdl @msailes !

I agree that we should leave CRT for now. At best we document how to use it as an optimization.

@jreijn - FYI - we have passed over our collective findings to the CRT team, who mentioned they have been thinking about this. We will loop back when we have any further interesting information to share!
In the meantime, would you like to work on the documentation improvements? I appreciate this isn't as glamorous as code, and we can certainly pick it up on our side if not :)

@scottgerring great! Yes, I was already looking into adding some documentation. I will probably do that in a separate PR if that's ok.


incorrect/confusing X.L.IndependentScreens documentation

The code shows an example (twice)
keyBindings conf = let m = modMask conf in fromList $

where m should be modm, or it should be rewritten as
keyBindings conf@(XConfig {modMask = modm}) = fromList $

I'd like to submit a patch to address this. I'm looking at the relevant example in X.L.IndependentScreens:
-- > keyBindings conf = let m = modMask conf in fromList $
-- >     {- lots of other keybindings -}
-- >     [((m .|. modm, k), windows $ f i)
-- >         | (i, k) <- zip (XMonad.workspaces conf) [xK_1 .. xK_9]
-- >         , (f, m) <- [(W.greedyView, 0), (W.shift, shiftMask)]]

Am I right to think that if I change the let m to let modm on the first line, the (m .|. modm) two lines below will use the m from the (f, m) below? In other words, literally the following patch should suffice:
diff --git a/XMonad/Layout/IndependentScreens.hs b/XMonad/Layout/IndependentScreens.hs
index d88c767..2ef230b 100644
--- a/XMonad/Layout/IndependentScreens.hs
+++ b/XMonad/Layout/IndependentScreens.hs
@@ -54,7 +54,7 @@ import XMonad.Hooks.DynamicLog
 -- to specific workspace names.  In the default configuration, only
 -- the keybindings for changing workspace do this:
 --
--- > keyBindings conf = let m = modMask conf in fromList $
+-- > keyBindings conf = let modm = modMask conf in fromList $
 -- >     {- lots of other keybindings -}
 -- >     [((m .|. modm, k), windows $ f i)
 -- >         | (i, k) <- zip (XMonad.workspaces conf) [xK_1 .. xK_9]
@@ -62,7 +62,7 @@ import XMonad.Hooks.DynamicLog
 --
 -- This should change to
 --
--- > keyBindings conf = let m = modMask conf in fromList $
+-- > keyBindings conf = let modm = modMask conf in fromList $
 -- >     {- lots of other keybindings -}
 -- >     [((m .|. modm, k), windows $ onCurrentScreen f i)
 -- >         | (i, k) <- zip (workspaces' conf) [xK_1 .. xK_9]

If someone can confirm that I'm not missing something about that m, I'll turn this into a PR. Thanks.

That change looks right to me.
On Fri, Nov 27, 2020, 11:07 Abhijit Menon-Sen<EMAIL_ADDRESS>wrote:

I'd like to submit a patch to address this. I'm looking at the relevant
example in X.L.IndependentScreens:
-- > keyBindings conf = let m = modMask conf in fromList $-- >     {- lots of other keybindings -}-- >     [((m .|. modm, k), windows $ f i)-- >         | (i, k) <- zip (XMonad.workspaces conf) [xK_1 .. xK_9]-- >         , (f, m) <- [(W.greedyView, 0), (W.shift, shiftMask)]]
Am I right to think that if I change the let m to let modm on the first
line, the (m .|. modm) two lines below will use the m from the (f, m)
below? In other words, literally the following patch should suffice:
diff --git a/XMonad/Layout/IndependentScreens.hs b/XMonad/Layout/IndependentScreens.hs
index d88c767..2ef230b 100644--- a/XMonad/Layout/IndependentScreens.hs+++ b/XMonad/Layout/IndependentScreens.hs@@ -54,7 +54,7 @@ import XMonad.Hooks.DynamicLog
-- to specific workspace names.  In the default configuration, only
-- the keybindings for changing workspace do this:
----- > keyBindings conf = let m = modMask conf in fromList $+-- > keyBindings conf = let modm = modMask conf in fromList $
-- >     {- lots of other keybindings -}
-- >     [((m .|. modm, k), windows $ f i)
-- >         | (i, k) <- zip (XMonad.workspaces conf) [xK_1 .. xK_9]@@ -62,7 +62,7 @@ import XMonad.Hooks.DynamicLog
-- This should change to
----- > keyBindings conf = let m = modMask conf in fromList $+-- > keyBindings conf = let modm = modMask conf in fromList $
-- >     {- lots of other keybindings -}
-- >     [((m .|. modm, k), windows $ onCurrentScreen f i)
-- >         | (i, k) <- zip (workspaces' conf) [xK_1 .. xK_9]
If someone can confirm that I'm not missing something about that m, I'll
turn this into a PR. Thanks.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/xmonad/xmonad-contrib/issues/111#issuecomment-734899917,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAPRFIFSTS4MH4ZFOEH22UDSR7FDVANCNFSM4CU2BPJA
.

Thanks for confirming. I've submitted a PR.


[FLINK-3166] [runtime] The first program in ObjectReuseITCase has the wrong expected result, and it succeeds

TestEnvironment now honors configuration of object reuse
Fixed reduce transformations to allow the user to modify and return either input

This is neither clean nor complete but I wanted to get some feedback before continuing.
ObjectReuseITCase was forcing object reuse, this was fixed by updating TestEnvironment.
ReduceCombineDriver and ReduceDriver both requests the next record and call the user's reduce() and so can handle whichever of the two inputs objects the user might write to.
In the second test case DataSourceTask (which requests the next record) calls ChainedAllReduceDriver.collect (which calls the user's reduce()). There is no feedback so DataSourceTask must cycle through n+1 reusable objects for unknown n.

A few comments:

the change to the TestEnvironment is unnecessary; ObjectReuse is enabled/disabled in the JavaProgramTestBase. it doesn't fix anything, removing the call to enabledObjectReuse in the test already did the trick.
The ObjectReuseITCase is supposed to verify that objects are reused. Now, the results are the same regardless of whether they are reused (except in case 3), making it little more then a reduce test. (There is also little point of running this test with ObjectReuse disabled, which is why it forced it)
A little bit above your changes in Reduce(Combine)Driver is a comment as to why we only need 2 objects when reusing objects, this will need to be adjusted.
I like the changes in DataSourceTask/Reduce(Combine)Driver. They are a seemingly simple solution for a common problem. In fact, But i am a bit apprehensive about them, because i have a hard time imagining that no one else could think of them; there might be a bigger issue with changes that i can't see at the moment.

I haven't found a prohibition on returning the second input to a user-defined reduce function. I think fixing this in code is preferable to documenting the problem or throwing an error, especially since the problem only exists with object reuse enabled.
Object reuse is configured in JavaProgramTestBase but must be set in the ExecutionEnvironmentFactory by TestEnvironment. I don't see that any of the many tests subclassing JavaProgramTestBase were testing with object reuse enabled.
It is rather difficult to test for object reuse. ObjectReuseITCase simply captured a few isolated instances of strange behavior (with the original tests 3 and 4 being perfect copies). I would look to duplicate the first two tests in the appropriate JavaProgramTestBase subclass and delete ObjectReuseITCase.

"Object reuse is configured in JavaProgramTestBase but must be set in the ExecutionEnvironmentFactory by TestEnvironment. I don't see that any of the many tests subclassing JavaProgramTestBase were testing with object reuse enabled."
Does this (L.112 JavaProgramTestBase)
// prepare the test environment
TestEnvironment env = new TestEnvironment(this.executor, this.parallelism);
env.getConfig().enableObjectReuse();
env.setAsContext();

not enable (and as such force) object reuse for every test that extends JavaProgramTestBase?

TestEnvironment was only setting the parallelism when creating and establishing the ExecutionEnvironmentFactory:
	public void setAsContext() {
		ExecutionEnvironmentFactory factory = new ExecutionEnvironmentFactory() {
			@Override
			public ExecutionEnvironment createExecutionEnvironment() {
				lastEnv = new TestEnvironment(executor, getParallelism());
				return lastEnv;
			}
		};

		initializeContextEnvironment(factory);
	}

ooohhh...that's a problem. alrighty, your solution for that looks good. actually, would you mind moving that into a separate PR so we can get it in faster? this should be fixed ASAP.

Good catch. I think all of these changes look actually good.
Minor comment in-line, otherwise +1

Merging this...


Compiler error in BERT MLM pretraining

Hi, I'm trying to run the BERT MLM example (examples/language-modeling/run_mlm.py) on an on-premise POD16 and am hitting a compiler error. I'm trying to debug on my side but I thought I'd make an issue here to see you can spot anything obvious.
Tech stack:
OS: CentOS 7
Python version: 3.6.8
Poplar version: 2.4.0
optimum-graphcore commit: 32eee51
transformers version: 4.18.0
Full stacktrace:

Graph compilation:  18%|████████████████████████████████████████████████████████▋                                                                                | 18/100 [01:02<03:23]2022-06-08T17:10:24.574645Z popart:popart 11633.11633 E: snap::Tensor Accum___model.bert.embeddings.word_embeddings.weight of unexpected Type. Poplar tensor type : half. Expected (Ir) tensor type : float. This for tensor Accum___model.bert.embeddings.word_embeddings.weight
[0] popart::popx::PopTensors::insert(std::string, snap::Tensor const&)
[1] popart::popx::InitTensorCreator::initTensor(popart::popx::IrLowering&) const
[17:10:25.174] [poptorch::python] [critical] poptorch.poptorch_core.Error: In poptorch/python/poptorch.cpp:1319: 'popart_exception': snap::Tensor Accum___model.bert.embeddings.word_embeddings.weight of unexpected Type. Poplar tensor type : half. Expected (Ir) tensor type : float. This for tensor Accum___model.bert.embeddings.word_embeddings.weight
Error raised in:
[0] popart::popx::PopTensors::insert(std::string, snap::Tensor const&)
[1] popart::popx::InitTensorCreator::initTensor(popart::popx::IrLowering&) const
[2] popart::Session::prepareDevice: Poplar compilation
[3] Compiler::compileAndPrepareDevice
[4] LowerToPopart::compile
Traceback (most recent call last):
File "examples/language-modeling/run_mlm.py", line 604, in
main()
File "examples/language-modeling/run_mlm.py", line 553, in main
train_result = trainer.train(resume_from_checkpoint=checkpoint)
File "/home/oscark/PycharmProjects/huggingface--optimum-graphcore/optimum/graphcore/trainer.py", line 887, in train
self._compile_model(model, next(iter(train_dataloader)), log=True)
File "/home/oscark/PycharmProjects/huggingface--optimum-graphcore/optimum/graphcore/trainer.py", line 355, in _compile_model
model.compile(**sample_batch)
File "/home/oscark/venvs/graphcore24/lib64/python3.6/site-packages/poptorch/_poplar_executor.py", line 622, in compile
self._compile(in_tensors)
File "/home/oscark/venvs/graphcore24/lib64/python3.6/site-packages/poptorch/_poplar_executor.py", line 505, in _compile
*trace_args)
poptorch.poptorch_core.Error: In poptorch/python/poptorch.cpp:1319: 'popart_exception': snap::Tensor Accum___model.bert.embeddings.word_embeddings.weight of unexpected Type. Poplar tensor type : half. Expected (Ir) tensor type : float. This for tensor Accum___model.bert.embeddings.word_embeddings.weight
Error raised in:
[0] popart::popx::PopTensors::insert(std::string, snap::Tensor const&)
[1] popart::popx::InitTensorCreator::initTensor(popart::popx::IrLowering&) const
[2] popart::Session::prepareDevice: Poplar compilation
[3] Compiler::compileAndPrepareDevice
[4] LowerToPopart::compile
06/08/2022 17:10:25 - critical - poptorch::python - poptorch.poptorch_core.Error: In poptorch/python/poptorch.cpp:1319: 'popart_exception': snap::Tensor Accum___model.bert.embeddings.word_embeddings.weight of unexpected Type. Poplar tensor type : half. Expected (Ir) tensor type : float. This for tensor Accum___model.bert.embeddings.word_embeddings.weight
Error raised in:
[0] popart::popx::PopTensors::insert(std::string, snap::Tensor const&)
[1] popart::popx::InitTensorCreator::initTensor(popart::popx::IrLowering&) const
[2] popart::Session::prepareDevice: Poplar compilation
[3] Compiler::compileAndPrepareDevice
[4] LowerToPopart::compile
Traceback (most recent call last):
File "examples/language-modeling/run_mlm.py", line 604, in
main()
File "examples/language-modeling/run_mlm.py", line 553, in main
train_result = trainer.train(resume_from_checkpoint=checkpoint)
File "/home/oscark/PycharmProjects/huggingface--optimum-graphcore/optimum/graphcore/trainer.py", line 887, in train
self._compile_model(model, next(iter(train_dataloader)), log=True)
File "/home/oscark/PycharmProjects/huggingface--optimum-graphcore/optimum/graphcore/trainer.py", line 355, in _compile_model
model.compile(**sample_batch)
File "/home/oscark/venvs/graphcore24/lib64/python3.6/site-packages/poptorch/_poplar_executor.py", line 622, in compile
self._compile(in_tensors)
File "/home/oscark/venvs/graphcore24/lib64/python3.6/site-packages/poptorch/_poplar_executor.py", line 505, in _compile
*trace_args)
poptorch.poptorch_core.Error: In poptorch/python/poptorch.cpp:1319: 'popart_exception': snap::Tensor Accum___model.bert.embeddings.word_embeddings.weight of unexpected Type. Poplar tensor type : half. Expected (Ir) tensor type : float. This for tensor Accum___model.bert.embeddings.word_embeddings.weight
Error raised in:
[0] popart::popx::PopTensors::insert(std::string, snap::Tensor const&)
[1] popart::popx::InitTensorCreator::initTensor(popart::popx::IrLowering&) const
[2] popart::Session::prepareDevice: Poplar compilation
[3] Compiler::compileAndPrepareDevice
[4] LowerToPopart::compile

Hi Oscar!
Which command are you using to run this?
I'm not able to reproduce this with BERT-Base.
Also I see you are using Poplar SDK2.4. Optimum only supports the latest SDK - 2.5. I recommend you upgrade to be fully compatible. :)

The poplar SDK version is probably the issue here, thanks for spotting. I'll upgrade, try again and let you know how that goes.

Closing the issue, please feel free to reopen it if needed.


Scale bar in the demo

The demo currently features no scale bar (unless I missed it somehow). Could one be added? You can make imperial units the default to really show that this is an American map style.

GL JS has a ready-made ScaleControl that we can enable. Here’s where we start adding controls to the map:
https://github.com/ZeLonewolf/openstreetmap-americana/blob/e0cd6a6d773581311d605022fdf111bec806c4e2/src/americana.js#L74-L78
ScaleOptions includes an option to switch between metric and imperial (customary) units. We could tie that to a URL parameter so the user can choose more familiar units. It could default to metric unless navigator.language is en or en-US (American English) or has US as the region code. This would be consistent with how the map defaults to the user’s preferred language, not necessarily American English.


Rounding of sensor values

In your weather tile examples, the temperatures (for example) appear as floats.
How would I round the values to the closest integer?
Trying a Math.round("&sensor.dark_sky_daily_low_temperature_" + id + ".state") results in a NAN, which leads me to believe that the generated strings, are evaluated some time later on after their creation in the tiles...
Maybe an anonymous function, but how would I use it in the context of the  TYPES.WEATHER_LIST tile?
Thanks!

Sorry, duplicate of #84


Fix error in 'pl.lapp.process_options_string'

Fix error "attempt to concatenate a nil value (local 'vtype')" by
checking making sure vtype is not nil.
I encountered this error when using ldoc which uses penlight to compile awesome WM. The error was as follows:
Scanning dependencies of target ldoc
[ 97%] Generating API documentation
lua: /usr/share/lua/5.3/pl/lapp.lua:317: attempt to concatenate a nil value (local 'vtype')
stack traceback:
        /usr/share/lua/5.3/pl/lapp.lua:317: in function 'pl.lapp.process_options_string'
        (...tail calls...)
        /usr/bin/ldoc:79: in main chunk
        [C]: in ?
make[2]: *** [CMakeFiles/ldoc.dir/build.make:204: doc/index.html] Error 1
make[1]: *** [CMakeFiles/Makefile2:389: CMakeFiles/ldoc.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

I'm not sure this is the best solution to this error and I don't even know what this function does but this minor change enabled me to finish building the awesome wm package and I thought I should share it with you.

Coverage remained the same at 83.324% when pulling 5ff83d69041325299ad565c220207d49d20ea293 on doronbehar:patch-1 into 0ba9c24000d3cad1d1d81a3dd233584b0bbd2950 on stevedonovan:master.


preparing TodoTemplate Project to get started (#1240)

this closes #1240

I'm gonna do whatever you've requested on my own. I've to merge this PR.


MS Bot to Teams Integration issues

Bot OnSend Activity is returning twice and it results error: activity must include non empty 'text' field or at least 1 attachment

Since you have this opened on SO, I'm going to close this and provide support there.

Hi Richardson, The SDK Version we are using are v4 and attached the error we are getting in Azure as warnings and below is log report FYR.. In mainDialog: displayHelpOptions step Innnnnnn onSendActivities Innnnnnn onSendActivities Dispatch Intent: Qna IT intents In QnaMaker dialog - confirm Innnnnnn onSendActivities Innnnnnn QnA if Innnnnnn onSendActivities Regex not found. Sending answer Innnnnnn onSendActivities INSIDE SENDACTIVITY Hi! How may I help to make your day better? Innnnnnn MainDialog intro step Innnnnnn onSendActivities Innnnnnn onSendActivities


Generate Label Uncertainty with GLEnet:  I reported an error when running step1.1:

Generate Label Uncertainty with GLEnet:
I reported an error when running step1.1:
2023-02-17 10:37:27,087   INFO  Start training /exp20_gen(fold_0)
epochs:   0%|                                           | 0/400 [00:05<?, ?it/s]
Traceback (most recent call last):                      | 0/111 [00:00<?, ?it/s]
File "train.py", line 247, in
main()
File "train.py", line 202, in main
train_model(
File "/home/dell/project/pcdet/GLENet-main/cvae_uncertainty/train_utils/train_utils.py", line 119, in train_model
accumulated_iter = train_one_epoch(
File "/home/dell/project/pcdet/GLENet-main/cvae_uncertainty/train_utils/train_utils.py", line 70, in train_one_epoch
loss.backward()
File "/home/dell/anaconda3/envs/glenet/lib/python3.8/site-packages/torch/tensor.py", line 245, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/dell/anaconda3/envs/glenet/lib/python3.8/site-packages/torch/autograd/init.py", line 145, in backward
Variable._execution_engine.run_backward(
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)
Traceback (most recent call last):
File "train.py", line 247, in
main()
File "train.py", line 202, in main
train_model(
File "/home/dell/project/pcdet/GLENet-main/cvae_uncertainty/train_utils/train_utils.py", line 119, in train_model
accumulated_iter = train_one_epoch(
File "/home/dell/project/pcdet/GLENet-main/cvae_uncertainty/train_utils/train_utils.py", line 70, in train_one_epoch
loss.backward()
File "/home/dell/anaconda3/envs/glenet/lib/python3.8/site-packages/torch/tensor.py", line 245, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/dell/anaconda3/envs/glenet/lib/python3.8/site-packages/torch/autograd/init.py", line 145, in backward
Variable._execution_engine.run_backward(
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)
Killing subprocess 3121397
Killing subprocess 3121398
Traceback (most recent call last):
File "/home/dell/anaconda3/envs/glenet/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/dell/anaconda3/envs/glenet/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/dell/anaconda3/envs/glenet/lib/python3.8/site-packages/torch/distributed/launch.py", line 340, in
main()
File "/home/dell/anaconda3/envs/glenet/lib/python3.8/site-packages/torch/distributed/launch.py", line 326, in main
sigkill_handler(signal.SIGTERM, None)  # not coming back
File "/home/dell/anaconda3/envs/glenet/lib/python3.8/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/dell/anaconda3/envs/glenet/bin/python', '-u', 'train.py', '--local_rank=1', '--launcher', 'pytorch', '--cfg_file', 'cfgs/exp20_gen.yaml', '--tcp_port', '18889', '--max_ckpt_save_num', '10', '--workers', '1', '--extra_tag', 'fold_0']' returned non-zero exit status 1.

Do you have two GPUs on your machine? Note I use two GPUs for training the GLENet.
Have you installed the right version Pytorch?

My computer has two GPUs, which can train other models. There is also a missing file when executing step 1.1: cfgs/${exp_id}_ gen_ ori.yaml

Sorry, I have fixed the bug. The cfgs/exp20_gen.yaml should be cfgs/exp20_gen_ori.yaml. You can pull the latest change.

Please make sure you execute step 1.1 successfully. You can check whether the checkpoints are well saved in fold_*.

Sorry, I didn't succeed in step 1.1 training just now. Now I can train normally. After the step 1.1 training is completed, I believe that step 1.2 should be able to run normally.

Please make sure you execute step 1.1 successfully. You can check whether the checkpoints are well saved in each directory like fold_*.

Sorry, I didn't succeed in step 1.1 training just now. Now I can train normally. After the step 1.1 training is completed, I believe that step 1.2 should be able to run normally.  @Eaphan

Thank you very much. Now step1.0-step1.3 can be successfully executed. @Eaphan

@Eaphan @6Superman6
I too have the same issues. Need your kind help.
Although I can use kitti_infos_train.pkl & kitti_dbinfos_train.pkl offered by @Eaphan but I want to study and tune model.py in cvae_uncertainty so that I must generate kitti_infos_train.pkl & kitti_dbinfos_train.pkl of my own.

I run ln -s data/kitti cvae_uncertainty, which doesn't work. So when running step 1.2: GLENet Prediction I have:

Traceback (most recent call last):
File "test.py", line 203, in
main()
File "test.py", line 189, in main
dist=dist_test, workers=args.workers, logger=logger, training=False
File "/home/lab715/Code/GLENet/cvae_uncertainty/dataset.py", line 1047, in build_dataloader
logger=logger,
File "/home/lab715/Code/GLENet/cvae_uncertainty/dataset.py", line 136, in init
with open(db_infos_path, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'kitti/kitti_dbinfos_train.pkl'
So I guessed and moved kitti/kitti_dbinfos_train.pkl into cvae_uncertainty/kitti (I built kitti under cvae_uncertainty).

Running step 1.2 again then I have:

Traceback (most recent call last):
File "test.py", line 203, in
main()
File "test.py", line 199, in main
eval_single_ckpt(model, test_loader, args, eval_output_dir, logger, epoch_id, dist_test=dist_test)
File "test.py", line 59, in eval_single_ckpt
model.load_params_from_file(filename=args.ckpt, logger=logger, to_cpu=dist_test)
File "/home/lab715/Code/GLENet/cvae_uncertainty/model.py", line 371, in load_params_from_file
raise FileNotFoundError

Seems I don't have training results in cvae_uncertainty/output/exp20_gen/fold_x_ckpt, so step 1.1 would be that for me:
FOLD_IDX: 0
FOLD_IDX: 1
FOLD_IDX: 2
FOLD_IDX: 3
FOLD_IDX: 4
FOLD_IDX: 5
FOLD_IDX: 6
FOLD_IDX: 7
FOLD_IDX: 8
FOLD_IDX: 9

How can I do to have the training results in fold_x_ckpt?
Thanks.

ln -s data/kitti cvae_uncertainty, which doesn't work

Why? Do you prepare the data of KITTI and create the directory well?

我也有同样的问题。需要你的帮助。虽然我可以使用kitti_infos_train.pkl & kitti_dbinfos_train.pkl提供的，但我想在cvae_uncertainty中学习和调整 model.py，以便我必须生成我自己的kitti_infos_train.pkl &kitti_dbinfos_train.pkl。

我运行 ln -s data/kitti cvae_uncertainty，它不起作用。因此，在运行步骤 1.2：GLENet 预测时，我有：

回溯（最近一次调用）：文件“test.py”，第 203 行，在 main（） 文件“test.py”，第 189 行，在 main dist=dist_test，workers=args.workers，logger=logger，training=False 文件“/home/lab715/Code/GLENet/cvae_uncertainty/dataset.py”，第 1047 行，在 build_dataloader logger=logger，文件“/home/lab715/Code/GLENet/cvae_uncertainty/dataset.py”，第 136 行，init with open（db_infos_path， 'rb'） as f： FileNotFoundError： [Errno 2] 没有这样的文件或目录： 'kitti/kitti_dbinfos_train.pkl'
所以我猜测并将 kitti/kitti_dbinfos_train.pkl 移动到 cvae_uncertainty/kitti 中（我在 cvae_uncertainty 下构建了 kitti）。

再次运行步骤 1.2，然后我有：

回溯（最近一次调用最后一次）：文件“test.py”，第 203 行，在 main（） 文件“test.py”，第 199 行，在 main eval_single_ckpt（model， test_loader， args， eval_output_dir， logger， epoch_id， dist_test=dist_test） 文件“test.py”，第 59 行，在 eval_single_ckpt model.load_params_from_file（filename=args.ckpt， logger=logger， to_cpu=dist_test）文件“/home/lab715/Code/GLENet/cvae_uncertainty/model.py”，第 371 行，load_params_from_file引发 FileNotFoundError

似乎我没有 cvae_uncertainty/output/exp20_gen/fold_x/ckpt 的训练结果，所以步骤 1.1 对我来说就是这样： FOLD_IDX：0 FOLD_IDX：1 FOLD_IDX：2 FOLD_IDX：3 FOLD_IDX：4 FOLD_IDX：5 FOLD_IDX：6 FOLD_IDX：7 FOLD_IDX：8 FOLD_IDX：9

我怎样才能获得 fold_x/ckpt 的训练结果？ 谢谢。

@Eaphan @6Superman6 I too have the same issues. Need your kind help. Although I can use kitti_infos_train.pkl & kitti_dbinfos_train.pkl offered by @Eaphan but I want to study and tune model.py in cvae_uncertainty so that I must generate kitti_infos_train.pkl & kitti_dbinfos_train.pkl of my own.

I run ln -s data/kitti cvae_uncertainty, which doesn't work. So when running step 1.2: GLENet Prediction I have:

Traceback (most recent call last): File "test.py", line 203, in main() File "test.py", line 189, in main dist=dist_test, workers=args.workers, logger=logger, training=False File "/home/lab715/Code/GLENet/cvae_uncertainty/dataset.py", line 1047, in build_dataloader logger=logger, File "/home/lab715/Code/GLENet/cvae_uncertainty/dataset.py", line 136, in init with open(db_infos_path, 'rb') as f: FileNotFoundError: [Errno 2] No such file or directory: 'kitti/kitti_dbinfos_train.pkl'
So I guessed and moved kitti/kitti_dbinfos_train.pkl into cvae_uncertainty/kitti (I built kitti under cvae_uncertainty).

Running step 1.2 again then I have:

Traceback (most recent call last): File "test.py", line 203, in main() File "test.py", line 199, in main eval_single_ckpt(model, test_loader, args, eval_output_dir, logger, epoch_id, dist_test=dist_test) File "test.py", line 59, in eval_single_ckpt model.load_params_from_file(filename=args.ckpt, logger=logger, to_cpu=dist_test) File "/home/lab715/Code/GLENet/cvae_uncertainty/model.py", line 371, in load_params_from_file raise FileNotFoundError

Seems I don't have training results in cvae_uncertainty/output/exp20_gen/fold_x/ckpt, so step 1.1 would be that for me:
FOLD_IDX: 0
FOLD_IDX: 1
FOLD_IDX: 2
FOLD_IDX: 3
FOLD_IDX: 4
FOLD_IDX: 5
FOLD_IDX: 6
FOLD_IDX: 7
FOLD_IDX: 8
FOLD_IDX: 9

How can I do to have the training results in fold_x/ckpt? Thanks.

I have the same issues as you, have you solved your problem yet?

You should complete step 1.1 before you start step 1.2.

作者您好:
我發現可以透過指令修改 for iter in seq 0 9至 seq 0 0，如此只做 FOLD_IDX: 0，一樣能訓練並生成標籤。
想請問 FOLD_IDX: 0到  FOLD_IDX: 9分別對應KITTI中不同的部分與樣本嗎？
抑或 FOLD_IDX: 0與FOLD_IDX: 9代表一樣的模型與樣本，只是每次運算都有其隨機性？
我能只訓練FOLD_IDX : 0並生成標籤，代表遍歷了KITTI資料集所有樣本？
謝謝
[image: Mailtrack]
https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&
Sender
notified by
Mailtrack
https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&
24/04/09
下午08:05:16
YF Zhang @.***> 於 2024年3月25日 週一 下午5:33寫道：

You should complete step 1.1 before you start step 1.2.
—
Reply to this email directly, view it on GitHub
https://github.com/Eaphan/GLENet/issues/3#issuecomment-2017577166, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AZ35VPRRK7NLIXSIANUOSL3YZ7VQJAVCNFSM6AAAAAAU65EO36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJXGU3TOMJWGY
.
You are receiving this because you commented.Message ID:
@.***>

Updated.
I found that I can modify label training instructions from "for iter in
seq 0 9" to "seq 0 0". I can train with single FOLD_IDX 0，but still
can't generate the uncertainty label (still lacking evaluation data from
FOLD_IDX 1 to 9) FOLD_IDX 0 and FOLD_IDX 9, are they corresponding to
different parts of KITTI, respectively? Should I train the label
uncertainty with the entire FOLD_IDX 0, 1, 2, ... to 9?
[image: Mailtrack]
https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&
Sender
notified by
Mailtrack
https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&
24/04/09
下午08:24:11
DL251陳秀吉 @.***> 於 2024年4月9日 週二 下午8:08寫道：

作者您好:
我發現可以透過指令修改 for iter in seq 0 9至 seq 0 0，如此只做 FOLD_IDX: 0，一樣能訓練並生成標籤。
想請問 FOLD_IDX: 0到  FOLD_IDX: 9分別對應KITTI中不同的部分與樣本嗎？
抑或 FOLD_IDX: 0與FOLD_IDX: 9代表一樣的模型與樣本，只是每次運算都有其隨機性？
我能只訓練FOLD_IDX : 0並生成標籤，代表遍歷了KITTI資料集所有樣本？
謝謝
[image: Mailtrack]
https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11& Sender
notified by
Mailtrack
https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11& 24/04/09
下午08:05:16
YF Zhang @.***> 於 2024年3月25日 週一 下午5:33寫道：

You should complete step 1.1 before you start step 1.2.
—
Reply to this email directly, view it on GitHub
https://github.com/Eaphan/GLENet/issues/3#issuecomment-2017577166, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AZ35VPRRK7NLIXSIANUOSL3YZ7VQJAVCNFSM6AAAAAAU65EO36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJXGU3TOMJWGY
.
You are receiving this because you commented.Message ID:
@.***>


Remove corrupted lines in xvg files

Hi,
when I want to analyze free energy differences while simulations are still running, the last line of the xvg files is often corrupted (not fully written yet) and alchemlyb fails to do the analysis. Alchemical analysis has a feature that repairs those files, so I usually run that first and then run alchemlyb on the repaired xvg files. Is it possible to move that feature into alchemlyb?
https://github.com/MobleyLab/alchemical-analysis/blob/master/alchemical_analysis/utils/corruptxvg.py

Is this meant as an issue for alchemlyb?

Oh yes, sorry, wrong place, will put it there!


'NoneType' object has no attribute 'WhichOneof'

Hi,
I'm trying to produce a data model diagram but am running into this:
♪  jsonator git:(master) ✗ sysl data -o data-model.png -j Project ./sysl/jsonator.sysl
'NoneType' object has no attribute 'WhichOneof'

My sysl file is in a sysl directory and I'm running the command one dir above, in the parent folder of the project. See https://github.com/nicholasf/jsonator/blob/master/sysl/jsonator.sysl
I can produce sequence and integration diagrams without an issue.

First of all, It seems like you might be using Sysl1, written in python, which is now deprecated
You can install sysl2 with go get -u github.com/anz-bank/sysl/cmd/sysl

Currently there's an  issue open which seems to be connected to this one #474
Investigating further into this at the moment

You were right - I was on the python version. Just getting the golang one setup now so maybe there's nothing to fix.

So, using the go version now, I'm running into another issue that seems restricted to data model diagrams.
I can still generate integration and sequence diagrams with the commands I used in the python version.
However, the data model diagram that is generated resembles a standard 'Welcome to PlantUML!' image.
This isn't an urgent issue by any means.
♪  jsonator git:(master) ✗ which sysl
/Users/nicholasf/go/bin/sysl
♪  jsonator git:(master) ✗ sysl data -o data-model.png -j Project sysl/jsonator.sysl
WARN[0000] root and .sysl are undefined, sysl will be used instead
WARN[0000] Field "data" (type "Data") refers to type Sink in app "Consumer"
♪  jsonator git:(master) ✗ l data-model.png
-rwxr-xr-x@ 1 nicholasf  staff    11K  8 Jan 10:44 data-model.png
♪  jsonator git:(master) ✗
♪  jsonator git:(master) ✗
sysl sd -o "sequence.png" -s "Sink <- Receive" sysl/jsonator.sysl
WARN[0000] root and .sysl are undefined, sysl will be used instead
WARN[0000] Field "data" (type "Data") refers to type Sink in app "Consumer"
♪  jsonator git:(master) ✗ l integration.png
-rwxr-xr-x  1 nicholasf  staff   3.6K  8 Jan 10:41 integration.png


[GSSOC'23]GUI Calculator with dark and light theme support

Project Description
In this project , iam going to make gui calculator .
This Calculator has a theme change support (i.e) We can Change to dark mode or light mode .
Solution:
This can done using tkinter module in python .
I have  finished this project  . Please assign this issue under gssoc 2023 , iam very intrested to contribute this repo.

@LakshmanGamers go ahead.

Sure 😃

I request you assign gssoc 23 label to this issue m


File -> "recent projects" items become available only when VSCode is loaded completely

Issue Type: Bug
With the new version clicking on "File" -> "recent projects" items become available only when VSCode is loaded completely.
I used to click VSCode icon, and then instantly select a project I need. Now I need to do it twice.  Or wait for 3~5 seconds before the native menu is loaded. Has something changed that could justify this  behavior? Am I the only one who is annoyed by this change on MacOS?


Add back the ಠ_ಠ face

I like to judge people who don't use evergreen browsers and I'd hope that the code checking for support does as well.

⚡ 🔥


New step not getting focus and global editor not updated

Great job on this project!
Not sure if this is intended behavior but after dragging a new step into the workflow I expected the newly added step to have focus and the appropriate step editor rendered. Instead, the focus stays on the global editor and doesn't reflect the underlying definition change.
Steps to Repro (version 0.1.6):

Start with an empty definition in the fullscreen example (the global editor is shown)
Add a step from the toolbox (focus remains on the global editor showing the empty definition even though onDefinitionChanged is called on the designer)
Change focus to the newly added step and then back to the overall process (globalEditorProvider is called and the current state of the definition is now rendered)

I think this is a good idea. It should increase the speed of the designer's usage. I've just release 0.1.7, this version contains that feature.

0.1.7 works great, thanks!


VerifyIDToken

Hello im triying to validate a token bun i'm getting this error:
decodedToken.Uid undefined (type *firebase.Token has no field or method Uid, but does have UID)

use this
decodedToken.UID()

use this
decodedToken.UID()


Choisir le clavier sur lequel on veut mettre le raccourci.

Choisir le clavier sur lequel on veut mettre le raccourci.

Please translate the text into English, if possible.

@PIORIGIN i believe you want multi-keyboard support for KBM.  correct?

if so, https://github.com/microsoft/PowerToys/issues/1460 was the tracking issue, TLDR, can't do it without a low level driver.


Edit scope pls. Birthday is removed.

Link to all legal scopes:
https://developer.spotify.com/documentation/general/guides/scopes/#scopes
or let us choose scopes, this would be the best.
Cos current request gives Illegal scope cos of birthday.

You can choose scopes, that was included in one of the lastest updates (v1.4.7). That still being included is part of SpotifyAPI.NET and only recently got removed from the Web API, hence it being a problem.
SpotifyAPI.NET should remove it soon however, just deselect it in the Advanced Settings of the Spotify Service


Added options to ontodoc for adding redirections to generated html file

Description:
Closes #552
Type of change:

[ ] Bug fix.
[x] New feature.
[ ] Documentation update.

Checklist:

This checklist can be used as a help for the reviewer.

[ ] Is the code easy to read and understand?
[ ] Are comments for humans to read, not computers to disregard?
[ ] Does a new feature has an accompanying new test (in the CI or unit testing schemes)?
[ ] Has the documentation been updated as necessary?
[ ] Does this close the issue?
[ ] Is the change limited to the issue?
[ ] Are errors handled for all outcomes?
[ ] Does the new feature provide new restrictions on dependencies, and if so is this documented?

Comments:

Codecov Report

Merging #553 (0a070df) into master (5d89725) will decrease coverage by 0.40%.
The diff coverage is 51.02%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##           master     #553      +/-   ##
==========================================
- Coverage   65.85%   65.46%   -0.40%
==========================================
  Files          16       16
  Lines        3099     3124      +25
==========================================
+ Hits         2041     2045       +4
- Misses       1058     1079      +21

Impacted Files
Coverage Δ

ontopy/utils.py
58.71% <44.18%> (-3.54%)
:arrow_down:

ontopy/__init__.py
80.00% <100.00%> (ø)

ontopy/graph.py
61.63% <100.00%> (ø)

ontopy/ontodoc.py
67.51% <100.00%> (ø)

ontopy/ontology.py
70.27% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Added support for using the sourcemap.sourceRoot

When ingesting sourcemaps, sometimes the file location has moved, and this is usually indicated using the sourceRoot property which shows where the original base directory was. By checking for that first before assuming the sourcemap path was at the directory of the sourcemap, we guarantee that the original file locations are preserved correctly.

+1


[Snyk] Security upgrade org.apache.jena:jena-dboe-base from 4.9.0-SNAPSHOT to 4.9.0

This PR was automatically created by Snyk using the credentials of a real user.Snyk has created this PR to fix one or more vulnerable packages in the `maven` dependencies of this project.
Changes included in this PR

Changes to the following files to upgrade the vulnerable dependencies to a fixed version:

jena-db/jena-dboe-transaction/pom.xml

Vulnerabilities that will be fixed
With an upgrade:

Severity
Priority Score (*)
Issue
Upgrade
Breaking Change
Exploit Maturity

579/1000  Why? Has a fix available, CVSS 7.3
Improper Neutralization of Special Elements used in an Expression Language Statement ('Expression Language Injection') SNYK-JAVA-ORGAPACHEJENA-6043134
org.apache.jena:jena-dboe-base:  4.9.0-SNAPSHOT -> 4.9.0
No
No Known Exploit

(*) Note that the real score may have changed since the PR was raised.
Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
🛠 Adjust project settings
📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

Checkmarx One – Scan Summary & Details – 3ef4d0de-3aeb-49d9-8408-04781c064fb0
New Issues

Severity
Issue
Source File / Package
Checkmarx Insight

CVE-2023-36478
Maven-org.eclipse.jetty:jetty-http-10.0.15
Vulnerable Package

CVE-2023-43646
Npm-get-func-name-2.0.0
Vulnerable Package

CVE-2023-45133
<EMAIL_ADDRESS>Vulnerable Package

CVE-2023-36479
Maven-org.eclipse.jetty:jetty-servlets-10.0.15
Vulnerable Package

CVE-2023-40167
Maven-org.eclipse.jetty:jetty-http-10.0.15
Vulnerable Package

CVE-2023-42503
Maven-org.apache.commons:commons-compress-1.23.0
Vulnerable Package

CVE-2023-44270
Npm-postcss-8.4.21
Vulnerable Package


Added .gitignore and cleaned up

Missing .gitignore file was added for C++, and debug directories containing executable and local settings were removed as part of the cleanup.

OK


Update hourly.pp

if you change group to 0 as you has in other classes, we can use this plugin on FreeBSD as well

Pull Request (PR) description
Change group from root to 0 to use this in freeBSD as well
This Pull Request (PR) fixes the following issues

this will be cancelled because of wrong solution


how to get the current running  aux jobs

hello ,
How can I get the current running aux jos?
thanks
Sorin

Hi @sorin00 ,
Can you please try using the below API?
https://github.com/Commvault/cvpysdk/blob/master/cvpysdk/job.py#L652
commcell_object.job_controller.active_jobs(job_filter='AUXCOPY')
Thanks,

the function is correct , but is not working with that filter .
do you know where do I find the right parameter for job_filter or job type ?

The below link has details on all the acceptable job filter values. All the values are case sensitive.
https://documentation.commvault.com/commvault/v11/article?p=105923.htm

but the AUXCOPY is not working .
can we do something ?
can we have some alternative filter ?

We might have to look into the logs for this. Can you please create a commvault support ticket for this?
You can email to<EMAIL_ADDRESS>

but can you confirm that is not working ?

Also, did you try adding greater lookupTime. By default, we look for job details for the last hour.
https://github.com/Commvault/cvpysdk/blob/master/cvpysdk/job.py#L652
Ideally, it should work. To check why it is not working for you we need to check logs on your setup.

k,
thanks


Unable to store localStorage session key when app opens in two tabs

Strange bug that happened on my application that when I logged in the token key is not stored in the localStorage at all.
To reproduce the issue:

Use ember-simple-auth version 1.0.0
Setup localStorage as session store in config/environment.js.
Open the application login route in 2 tabs.
Then login using any of those tabs.

The error got after login: Uncaught SyntaxError: Unexpected token u (in Chrome), SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data (in Firefox)
Under browser localStorage debug: ember_simple_auth:session    undefined
Please look into this issue. Thanks!

If you want to use localStorage store with ESA 1.0 you don't configure that in 1.0 but instead define the store in app/session-stores/application.js (and extend the LocalStorageStore in that file). Configuring the store in config/environment.js worked pre 1.0.

I'm not quite sure I understood what you wrote. Could you give me an example of how to configure that? I'm using ESA 1.0. Thanks so much!!

You cannot configure the session store to use in config/environment.js. See the README for more info on choosing the store to use for the application: https://github.com/simplabs/ember-simple-auth#session-stores

I see. I was using ESA since pre 1.0, and was not aware of this change. Thanks!


user docs: General update of docs.

I did a pass through our docs, and a few things stood out

We should add info about keyboard shortcuts into the relevant articles. E.g. the article about lightbox should include info about the v shortcut.
I put up a diff between what our left sidebar would ideally be, and what it currently is (after #6878 is merged), at https://github.com/zulip/zulip/commit/32f6a768e3ba1a3501dfaa94ab70e7cf0340fdc9. It would be great to start writing articles and otherwise make the article changes needed to support the left sidebar links.
Some of the changes in the diff are easier than others (e.g. are mostly moving existing content around, rather than creating new content), and should be done first.

@synicalsyntax fyi

WIP

I've noticed a few broken links or inconsistencies in the docs, leading to some possibly unclear parts of the documentation.

[ ] https://chat.zulip.org/help/getting-started-with-zulip. At the bottom of this page (under Next Steps), the "Add an avatar" and "Configure your notifications" links link directly to the settings, unlike many of the other links in the documentation, which link you to the page that shows how to do those things.
[ ] https://chat.zulip.org/help/change-your-password. For number 2 under "If you forgot your password", the terminology on the Zulip homepage has been changed to "Forgot your password?" from "Forgot password?"
[ ] https://chat.zulip.org/help/signing-in. There are more than 3 ways to sign in that aren't documented here.
[ ] https://chat.zulip.org/help/analytics.  This page should be a little bit more clear on how to get to the statistics; it's fairly easy to miss the one (broken) link at the top of the page.
[ ] https://chat.zulip.org/help/zulip-on-windows. This and the page for Android should be deleted as they seem to be from older versions of those apps. The link for the Windows download is broken on this page, but the Android one links to the correct Google Play Store listing.
[ ] https://chat.zulip.org/help/about-streams-and-topics. On this page, for the links to customize stream settings, there are many items missing, so something should be added to lead users to the other customization options.
[ ] https://chat.zulip.org/help/change-your-organization-settings. This one is extremely minor, but an image for a chevron is present next to all the other instances of the word "chevron" in the docs (same with "gear"), but it's missing here.

@rishig You told me that you had a different contributor in mind for taking up this task, so you should probably assign this to that individual before this issue gets claimed by someone else.

@zulipbot claim
Are there docs somewhere that say how to compile the documentation. I'd like to check links etc as I work on this.

I'll just compile and run the entire zulip server, I guess.

https://github.com/zulip/zulip/pull/8324


[Service Bus] Limits on ServiceBusMessageBatch

Issue
There is a limit of 4500 messages that can be sent on the link
To repro

put a byte of buffer in the body for each message
keep adding messages with the tryAdd method
A total of 33K messages will be added
Upon sending the batch, service throws an error

If stopped adding messages at 4500, send works perfectly fine

Thanks to @MiYanni for the find!
TODO

Reach out to the service team on the limit
Ideal solution: link should have the max_number_of_messages that can be sent along with the max_size, which could be used to restrict adding messages when tried to add more with tryAdd()
Add a test upon clarifying

Do we know if Event Hubs has a similar limitation?

Just tested event-hubs, it doesn't seem to have the same limit.
I was able to add the messages until I maxed out w.r.t max_batch_size and could send the batch.

For now, we will be updating tryAddMessage() to return false if the 4500 limit is reached.
We will be logging a feature request with the service to either remove this limit or provide this information on the link so that they are free to increase the limit and we dont need to ship an update

Related: https://github.com/Azure/azure-service-bus/issues/397

Instead of adding this hard limit client-side, we are instead going to place information in the documentation for tryAdd, since the limit for Basic and Standard tier is 4500 and the limit for Premium is ~15885. It would be best to link to Service Bus documentation on these limits instead of hard-coding into our reference documentation. However, we'd have to wait until they publish that information.

👋 @HarshaNalluru I'm cleaning up old issues, mind moving this to the appropriate milestone when you get a chance?

No service doc available about this limit yet. Moving into backlog.


[RESTEASY-1721] ResourceMethodInvoker checks if @Cont…

…ext HttpServletResponse was written to.

Replaced by https://github.com/resteasy/Resteasy/pull/2753.


Use stream transporter when sending emails in local development

Detailed Description
#158 replaces the local stream transporter with the SES transporter. While SES will work in local development as configured, it might still be desirable to use the stream transporter for local development and testing to avoid dependence on an external service.
The necessity and priority of this issue is totally up to the discretion of the developers. I'm creating this issue separately from #158 because this requires a decision on if and how we handle configuration that is conditional on the environment. I think that decision is best left to application developers who are more familiar with NestJS and this application's approach to managing configuration.

@kshepard What's your opinion on this?

I've been discussing this in #158. Ops would prefer preserving this functionality for sake of consistency with other projects but we're trying to figure out the best way to handle configuring it. We're likely to coordinate with DistrictBuilder developers in an upcoming sprint.

I agree with the rationale here.

We ended up preserving this functionality, though the solution could use some improvement. I'm queuing this up for our next sprint. Ops will coordinate with one of the DB devs to pair up and make the code pretty ✨
I've also updated the description to turn this issue into a refactoring issue.

I paired up with @kshepard on this one. We figured out that this object we were logging was defined with an any type in an upstream module. We decided to refactor logging and property checking to a separate function to increase visibility of how we're handling the potentially different object properties.


Drop CompilerCache

Fixes #835

To validate this pull request against other sbt modules, please visit this link.

I'm happy to rebase #839 on top of this, so we can review what's left over.  Does that work for you?

Sounds good.

Happy to merge this?


list-formatting - style property cannot have any of the following characters ( : & ; ! .

Category

[X] Question
[ ] Typo
[ ] Additional article idea

Question

I can see couple answers that the style properties cannot support any of the following characters: (, :, &, ;, ! exception and that is the use of the rgba css function:

Scenario 01: background-image URL property is not supported because URL contains ( character.
"style": {                 "display": "inline-block",                 "width": "8px",                 "height": "8px",                 "background-image": {                     "operator": "?",                     "operands": [                         {                             "operator": "==",                             "operands": [                                "@currentField",                                "expectedValue"                             ]                          },                         "url('/site/Shared Documents/imageName.jpg')",                         ""                         ]                 }             }
Scenario 02: calc() method is not supported because calc contains ( character.
{   "$schema": "https://developer.microsoft.com/json-schemas/sp/column-formatting.schema.json",   "elmType": "div",   "children": [     {       "elmType": "span",       "txtContent": "@currentField",       "style": {         "padding": "0 8px",         "white-space": "nowrap"       }     }   ],   "attributes": {     "class": "sp-field-dataBars"   },   "style": {     "padding": "0",     "height": "calc(100% + 22px)",     "width": "100%"   },   "debugMode": true }

Refer the URL

https://github.com/SharePoint/sp-dev-docs/issues/1161#issuecomment-404240884
https://github.com/SharePoint/sp-dev-docs/issues/3696#issuecomment-478178019

However, the character ( is supported in style proprieties - refer color property
{   "$schema": "https://developer.microsoft.com/json-schemas/sp/v2/column-formatting.schema.json",   "elmType": "div",   "debugMode": true,   "txtContent": "@currentField",   "style": {     "color": "=if([$DueDate] <= @now, '#ff0000', '')"   } }
https://docs.microsoft.com/en-us/sharepoint/dev/declarative-customization/column-formatting#apply-formatting-based-on-date-ranges

Is background-image style property support in column formatting? as this property calls URL() which contains ( character.
background-image: url("paper.gif");

If no, I will update the document accordingly - https://docs.microsoft.com/en-us/sharepoint/dev/declarative-customization/column-formatting#detailed-syntax-reference
@ shagra-ms @thechriskent  &PopWarner
Thanks for your contribution! Sharing is caring.

Issue #1161 outlines this issue accordingly, and even states that using url() is not supported (by @shagra-ms).
@Shagra-ms already noted that he will updated the documentation for this. There is no action necessary.

@bcameron1231 - do you have any idea why the character ( is supported in style proprieties - refer color property
{ "$schema": "https://developer.microsoft.com/json-schemas/sp/v2/column-formatting.schema.json", "elmType": "div", "debugMode": true, "txtContent": "@currentField", "style": { "color": "=if([$DueDate] <= @now, '#ff0000', '')" } }

That's because the character is used in an Excel Style conditional expression =if(), which is supported in column formatting.


Add mechanism to notify users of backwards-incompatible updates

We need a good way to notify users running upgrade-zulip of any backwards-compatible updates (e.g. to configuration).

This can be done in several ways but the one which I found most appropriate is:-
We can add a 'pop-up screen' as soon as a user opens this app before updating or while in progress both.
From there, the user can proceed to see the updating process or if he/she has some other important  work to do at that particular time, he/she can find these updates in the 'settings' options provided in this chat app.

Just to be clear, this issue is about notifying the system administrator of a Zulip server, not about notifying the individual users of that Zulip server.

Our "upgrade notes" system is pretty effective here, but it would be nice if we could offer a way to display/prompt about these in the upgrade script, so leaving this open.


perf(deque): use @deque.iter[2]() for @deque.each[i]()

Continuation of https://github.com/moonbitlang/core/pull/1354.
Summary
Improvements to iteration methods:

pub fn each[A](self : T[A], f : (A) -> Unit) -> Unit: Updated the method to use a direct iteration over the collection instead of indexing.
pub fn eachi[A](self : T[A], f : (Int, A) -> Unit) -> Unit: Updated the method to use a direct iteration with index over the collection instead of indexing.

Pull Request Test Coverage Report for Build 4304
Details

2 of 2 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.006%) to 81.248%

Totals

Change from base Build 4303:
-0.006%

Covered Lines:
4623

Relevant Lines:
5690

💛  - Coveralls


Bump PMD to version 7.0.0

I've been playing around with upgrading RuneLite to Java 21. Part of that work would require upgrading PMD to version 7.0.0, which was a big enough change on its own that felt like it needed its own PR.

I've looked into this before, but decided to not update because they made the immutable field check no longer work with Lombok (I guess due to make the new Java record types?). But since we use Lombok so heavily it is just a loss of functionality. I'd probably rather just remove the immutable field check than go around and annotate everything as PMD ignore though.
I am not familiar with this pmd-compat6 dep, do you know what that is for?

I've looked into this before, but decided to not update because they made the immutable field check no longer work with Lombok (I guess due to the new Java record types?). But since we use Lombok so heavily it is just a loss of functionality. I'd probably rather just remove the immutable field check than go around and annotate everything as PMD ignore though.

PMD's states their reasoning here. It's an unfortunate change considering use cases like Lombok. I'd be happy to modify this PR to remove that rule if there is still interest in the upgrade, though.

I am not familiar with this pmd-compat6 dep, do you know what that is for?

This was added specifically to support maven-pmd-plugin until they update to version 7.0.0 themselves. There's already  a PR to do that here, https://github.com/apache/maven-pmd-plugin/pull/144, so the need for pmd-compat6 will go away quickly.

I guess I don't mind the update / I am sort of indifferent to it. I see you mentioned moving to Java 21, but to do that we first have to update the launcher to 21 and then push launcher updates out. Updating the launcher version is a pretty major thing, is somewhat irreversible, and tends to uncover obscure Java bugs and also break our code in places we are depend on 11 specific functionality. It is not something I want to sink a bunch of time in unless I have a compelling reason to update.

https://github.com/pmd/pmd/pull/4474 looks like they made it ignore fields annotated with @Getter or @Setter in 7.0.0. Do you know why you are still having to annotate those fields as ignored?

I see you mentioned moving to Java 21, but to do that we first have to update the launcher to 21 and then push launcher updates out. Updating the launcher version is a pretty major thing, is somewhat irreversible, and tends to uncover obscure Java bugs and also break our code in places we are depend on 11 specific functionality. It is not something I want to sink a bunch of time in unless I have a compelling reason to update.

Thanks for your thoughts on this. I mentioned it in this PR to gauge interest, and it is understandable that you would not want to push for the upgrade without a very good reason, which I admittedly do not have. I also see in the launcher that you are currently depending on JDKs with 32bit Windows support for which there does not seem to be a temurin offering at this time.

they made it ignore fields annotated with @Getter or @Setter in 7.0.0. Do you know why you are still having to annotate those fields as ignored?

Ah, great catch. I was following their migration guide which suggested going to 6.55.0 first before 7.0.0, and I think as a result 6.55.0 reported these erroneously and I fixed them before bumping to 7.0.0. Let me redo the version bump in one go and see what changes that ends up requiring.

I believe that is 27 less use cases of @SuppressWarnings("PMD.ImmutableField") than I had before, great catch!

There are also several // NOPMD in the code, some of them can be removed because they have been fixed upstream I think. Please check those.

I was able to remove a couple of // NOPMDs in the code without changes. I also checked old @SuppressWarnings("PMD.<Rule>")s but they were still needed without other changes.

It seems I do not have permission to push to this PR, so I have rebased this, updated to pmd 7.2.0, and pushed it directly to master. Thanks.


register FreeRedis as 3018

Register FreeRedis as 3018

Please update changes.md according to the PR template(you removed all somehow).


[Bug] setting symbolType of point in series does not also set that of legends and tooltip

Version
1.12.1
Link to Minimal Reproduction
https://www.visactor.io/vchart/theme/playground
Steps to Reproduce

open https://www.visactor.io/vchart/theme/playground
modify series.area.point.style.symbolType to be triangle

Current Behavior
symbolType of legends and tooltip not affected
Expected Behavior
symbolType of legends and tooltip being affected
Environment
- OS:
- Browser:
- Framework:

Any additional comments?
No response

@warin2020 试一下如下配置theme:
{
  series: {
    area: {
      point: { style: { symbolType: 'triangle' } },
      seriesMark: 'point'
    }
  }
}


When using channels, exceptions are not propagated properly

I encountered this issue when using channels in Saturn via add_channel in application CE.
Take the following code:
open System

open Microsoft.AspNetCore.Http
open Microsoft.Extensions.Logging
open Saturn
open Giraffe
open Saturn.Channels
open FSharp.Control.Tasks.Affine

let testController = controller {
    index (fun (ctx: HttpContext) ->
        let logger = ctx.GetLogger<String>()
        logger.Log(LogLevel.Information, "BEFORE FAIL")
        failwith "NOPE!"
         Exception is logged in console and the response is 500 "NOPE!":
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://localhost:5000/test - -
info: string[0]
      BEFORE FAIL
fail: Giraffe.Middleware.GiraffeErrorHandlerMiddleware[0]
      An unhandled exception has occurred while executing the request.
      System.Exception: NOPE!
         at Saturn.Controller.Yield@110-1.Invoke(HttpContext _arg1, Exception ex)
         at Saturn.Controller.controllerWithErrorHandler@355-4.Invoke(Exception _arg2)
         at Ply.TplPrimitives.tryWith[u](FSharpFunc`2 continuation, FSharpFunc`2 catch)
         at Ply.TplPrimitives.ContinuationStateMachine`1.System-Runtime-CompilerServices-IAsyncStateMachine-MoveNext()
         at Giraffe.Core.chooseHttpFunc@122.Invoke(Unit unitVar)
         at Ply.TplPrimitives.ContinuationStateMachine`1.System-Runtime-CompilerServices-IAsyncStateMachine-MoveNext()
         at Giraffe.Core.chooseHttpFunc@122.Invoke(Unit unitVar)
         at Ply.TplPrimitives.ContinuationStateMachine`1.System-Runtime-CompilerServices-IAsyncStateMachine-MoveNext()
         at Giraffe.SubRouting.routeWithPartialPath@24.Invoke(Unit unitVar)
         at Ply.TplPrimitives.ContinuationStateMachine`1.System-Runtime-CompilerServices-IAsyncStateMachine-MoveNext()
         at Giraffe.Core.chooseHttpFunc@122.Invoke(Unit unitVar)
         at Ply.TplPrimitives.ContinuationStateMachine`1.System-Runtime-CompilerServices-IAsyncStateMachine-MoveNext()
         at Giraffe.Middleware.Invoke@31.Invoke(Unit unitVar)
         at Ply.TplPrimitives.ContinuationStateMachine`1.System-Runtime-CompilerServices-IAsyncStateMachine-MoveNext()
         at Giraffe.Middleware.Invoke@64-5.Invoke(Unit unitVar)
         at Ply.TplPrimitives.tryWith[u](FSharpFunc`2 continuation, FSharpFunc`2 catch)
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5000/test - - - 500 7 application/json;+charset=utf-8 470.8031ms

Any ideas?
Could it be that services are added in such an order that the regular error handling is ignored, or something like that?

You could try remove everything past |> on this line (channel socket middleware comes before giraffe), it seems suspect and it looks unnecessary https://github.com/SaturnFramework/Saturn/blob/aac8fb1c80ebab3ed96e7c92660512a0c9a2b688/src/Saturn/Channels.fs#L149

I'll give it a go, thanks

This fixes it, thanks @NinoFloris . I don't know if this has any other implications, though.
Is there a possibility to fast-track this?


Backport #1024 tenant resolver

Backport https://github.com/opensearch-project/security-dashboards-plugin/pull/1024 to branches "1.3", "1.2", "1.1", 1.0"/
Follow up issue: https://github.com/opensearch-project/security-dashboards-plugin/issues/1012
Check List

[x] New functionality includes testing
[x] New functionality has been documented
[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@DarshitChanpura @peternied this PR had to be tagged with "backport 1.3", "backport 1.2", "backport 1.1", "backport 1.0" before merging.

@rutuja-amazon It looks like nearly all of those backports had issues, if you'd like to see them merged please investigate them.

@peternied I'm looking into why auto-backporting is causing issues.


fix counter for build tool snippets

This is the proper place for my pull request for the generated site. Sorry for pull requesting to the wrong repo!
I hope that the changes to what I hope they do :)

Yes, excellent - thanks.  Pushing to the site now.


"linking error, not in local cache" when running npm run ci

This is a mystery.
...
 FAIL  sample/test-fns/userInfo.test.js
  ● Test suite failed to run

    linking error, not in local cache

Also covered in https://github.com/akauppi/GroundLevel-firebase-es/issues/31

Jest has solved this; once we get a new 27.0.0.next candidate, the problem should go away.


Restructured projects and added Path

Projects javascript and jvm now both depend on shared.
Path is an image which takes one or more PathElements.

Maybe split this PR into one that fixes the test errors, and then a second one for the new path stuff?

I've "fixed" the unicode arrows.
@jonoabroad -- can you comment on the "Wrong" comments Richard mentioned above?

@davegurnell they are. I only confirmed the code compiles. I do not know if the parameter order is correct.

I'm closing this pull request and switching work on adding paths to feature/paths. There were a few reasons for closing this request:

there are a lot of formatting changes we don't want, esp. unicode
the pull request adds three different things (build changes, paths, and compilation fix)
I don't want to use implicit classes in this project in the way this patch does


Windows Compatibility

Hello @abdelhakim96
Do you still have a problem with Windows (which you did not describe initially) ? I will be closing this issue now. Reopen if it persists (but if you do so, please kindly include details 😅 )


Add Skills onto the site Main Menu

This PR adds Skills button on the site Main Menu that links to the Skills section.

@Error6251 any updates?

Thank you for this pull request. But your short code and reading time got some issue. You never used this short code anywhere and you also added a css on static that is not how we handle our style sheet. And the reading time have no specific design, it just blend with the blog contents. Can you please update those things, Otherwise I can't add this to the main repo.

@Error6251 My apologies, the shortcode and reading time was never meant to be in this commit. I have made the commit by mistake. I have removed them from the PR. Thank you for raising the issue.


Enable CxxStdlib on Windows

This improves the C++ interop support on Windows by mitigating a clang issue in the build system when building natively and enables the CxxStdlib module.

@swift-ci please build toolchain Windows platform

@swift-ci please smoke test

Please test with following PRs:
https://github.com/apple/swift-installer-scripts/pull/220
@swift-ci please build toolchain Windows platform

Please test with following PRs:
https://github.com/apple/swift-installer-scripts/pull/220
@swift-ci please build toolchain Windows platform

Okay, seems that packaging is not yet ready for this.  I am going to merge this anyway - we just won't be distributing the files for now.  This should at least give us some coverage from the testing side.

Bleh, seems that rebase is not enabled, okay, then I'll split this up.  The underlying change is the one that should resolve the issue that @eeckstein hit, the top most commit needs a bit more tweaking anyway to ensure that we don't collide on things.

Please test with following PRs:
https://github.com/apple/swift-installer-scripts/pull/220
@swift-ci please build toolchain Windows platform

Please test with following PRs:
https://github.com/apple/swift-installer-scripts/pull/220
@swift-ci please build toolchain Windows platform

@swift-ci please smoke test


exception thrown when tests fail with trial

from twisted.trial import unittest
from twisted.internet.defer import inlineCallbacks
from twisted.internet import defer, reactor

def badCode(d):
    raise Exception('boom!')
    d.callback(None)

class TestFailure(unittest.TestCase):

    @inlineCallbacks
    def testBadCode(self):
        d = defer.Deferred()
        reactor.callLater(0.1, badCode, d)
        yield d.addTimeout(0.2, reactor)

result
File "c:\python37\lib\site-packages\twisted\plugins\teamcity_plugin.py", line 11, in getitem
return self.value[key]
TypeError: 'Exception' object is not subscriptable

https://youtrack.jetbrains.com/issue/PY-32694


python38Packages.databricks-connect: 8.1.11 -> 8.1.12

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python38Packages.databricks-connect is: "Client for connecting to remote Databricks clusters"
meta.homepage for python38Packages.databricks-connect is: "https://pypi.org/project/databricks-connect"
meta.changelog for python38Packages.databricks-connect is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS

Warning: a test defined in passthru.tests did not pass

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/beeline passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/beeline.cmd passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/docker-image-tool.sh passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/find-spark-home passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/find-spark-home.cmd passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/find_spark_home.py passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/load-spark-env.cmd passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/load-spark-env.sh passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/pyspark passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/pyspark-bazel passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/pyspark.cmd passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/pyspark2.cmd passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/quicksilver passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/run-example passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/run-example.cmd passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/spark-class passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/spark-class.cmd passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/spark-class2.cmd passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/spark-shell passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/spark-shell-bazel passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/spark-shell.cmd passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/spark-shell2.cmd passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/spark-sql passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/spark-sql.cmd passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/spark-sql2.cmd passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/spark-submit passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/spark-submit.cmd passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/spark-submit2.cmd passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/sparkR passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/sparkR.cmd passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/sparkR2.cmd passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/databricks-connect passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/.find_spark_home.py-wrapped passed the binary check.

/nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin/.databricks-connect-wrapped passed the binary check.

7 of 34 passed binary check by having a zero exit code.

32 of 34 passed binary check by having the new version present in output.

found 8.1.12 with grep in /nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12

found 8.1.12 in filename of file in /nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12

directory tree listing: https://gist.github.com/1e7f7e09854e24ae5a783adad9b6f8c8

du listing: https://gist.github.com/9838824d56b713b18b4f189666c6a0ea

Rebuild report (if merged into master) (click to expand)

8 total rebuild path(s)

2 package rebuild(s)

2 x86_64-linux rebuild(s)
2 i686-linux rebuild(s)
2 x86_64-darwin rebuild(s)
2 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
python38Packages.databricks-connect
python39Packages.databricks-connect

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python38Packages.databricks-connect https://github.com/r-ryantm/nixpkgs/archive/1b9987bfd9678179115c9cacd444e7bad8df3a36.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12
ls -la /nix/store/j94c31wa851c6vclbsfnj18k0750hgkr-python3.8-databricks-connect-8.1.12/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

2 packages built:

python38Packages.databricks-connect
python39Packages.databricks-connect

Maintainer pings
cc @kfollesdal for testing.

Result of nixpkgs-review pr 137347 at 1b9987bf run on aarch64-linux 1

2 packages built successfully:

python38Packages.databricks-connect
python39Packages.databricks-connect


The show command could be a lot better

This is the help for the bundle show command:
$ bundle show --help
Usage:
  bundle show GEM [OPTIONS]

Options:
      [--paths=List the paths of all gems that are required by your gems.rb.], [--no-paths]
      [--outdated=Show verbose output including whether gems are outdated.], [--no-outdated]
      [--no-color], [--no-no-color]                                                           # Disable colorization in output
  -r, [--retry=NUM]                                                                           # Specify the number of times you wish to attempt network commands
  -V, [--verbose], [--no-verbose]                                                             # Enable verbose output mode

Description:
  Show lists the names and versions of all gems that are required by your gems.rb. Calling show with [GEM] will list the exact location of that gem on your machine.

We don't actually have --no-paths, --no-outdated, or --no-verbose options, so we should remove those if Thor will let us. Showing --no-color and --no-no-color is... pretty silly. Let's change that to (if we must) --color and --no-color, or better yet only --no-color.
Right now bundle show lists gem names, while bundle show --paths or bundle show NAME shows the paths to gems. Then, confusingly, bundle show --outdated shows detailed information including summaries, websites, and whether a gem is up to date or not. We'll probably want to keep the bundle show --paths and bundle show NAME commands around for the entire 2.x cycle so that tools will work with both Bundler 1 and Bundler 2, but we can make the user-facing commands better for sure.
Let's move bundle show to bundle list (with an alias of bundle ls). That seems to make sense, and gets us firmly into shared territory with npm list/npm ls.
Let's move bundle show --outdated to bundle info NAME, and completely remove the --outdated option. We already have an outdated command. (Both brew and npm use info. This seems like the right name.) T bundle info command without a gem name can either throw an error or print out bundle statistics, like number of Gemfile gems, total number of gems, and maybe... size of the bundle in MB? Sounds nice.
We can also add a bundle info --path NAME option to print out the path to a single gem. Later, when Bundler 1.x usage has dropped down to nearly nothing, we can deprecate bundle show NAME in favor of bundle info --path NAME.

@indirect there was a PR to fix the -no-no-color thing months ago that never got your r+

@segiddins that PR has no listed green builds on Travis, is no longer mergable, and the proposer has abandoned it. As I mentioned on the ticket, though, I'm happy for anyone to r+ a mergable version of it with passing specs.

Hey guys! I'm a newcomer to Bundler in terms of contributing. Is that okay if I pick this up?

@borsothy i selfishly would love some assistance on some of the checklist items attached to this issue => https://github.com/bundler/bundler-features/issues/122, though they may be less suited to you if you want something lighter. :)

@chrismo sure, let's see what I can do for you. Heading over to that conversation.

@chrismo can I change my mind? :D I'm not sure I'm prepared for those issues. Not Rubywise, but I have no idea about the Bundler codebase so far :|

oh, absolutely. your time is yours, however you want to contribute is fine by me 😄.  I just wanted to throw those out as something of a higher personal priority in case there was any overlap of interests.

This is being partially addressed by https://github.com/bundler/bundler/pull/5093

@indirect now that we have bundle info, should this be closed?

@indirect ping?

@segiddins seems like, yeah.