fm-index crate ported to vers

I need a FM-index in my XML project.
Unfortunately what I need is a special FM-index that can:

handle multiple stop symbols $ so that the order of these can be used to give each text a unique id
the unique text id and the offset of the id is somehow stored in the sampler
there's also a doc array to obtain a text id given a position in the underlying BWT structure

as described in Fast in-memory XPath search using compressed indexes, chapter 3 and also 4.2.3 and 4.3. I still haven't fully processed how it all works.
I think those things require a specialized fm-index, so in preparation and in order to increase my understanding I figured I'd try porting the fm_index crate, originally based around the fid crate, to vers. I'm not sure it helped my understanding a lot yet, but I thought I'd report it here:
https://github.com/faassen/fm-index/tree/vers
I ported things over to vers' BitVec, RsVec and WaveletMatrix (allowing me to remove fm-index's own implementation of wavelet matrix).
There are two test failures I don't understand yet: one in suffix_array and one when testing fm_index with UTF-8 data. (these already happened before I switched over to the waveletmatrix from vers)
Now this is dicey as I'm still getting test failures, but if I run the benches in general construction time has gone up by a little, but count() and locate() have increased their performance by a lot!
group                             after                                    before
-----                             -----                                    ------
construction/FMIndex/1000         1.04     59.1±1.56µs        ? ?/sec      1.00     57.0±0.73µs        ? ?/sec
construction/FMIndex/10000        1.03    737.4±4.59µs        ? ?/sec      1.00    716.2±8.12µs        ? ?/sec
construction/FMIndex/100000       1.22      8.9±0.12ms        ? ?/sec      1.00      7.3±0.10ms        ? ?/sec
construction/FMIndex/1000000      1.25     98.0±1.76ms        ? ?/sec      1.00     78.6±1.77ms        ? ?/sec
construction/RLFMIndex/1000       1.01     67.5±4.95µs        ? ?/sec      1.00     67.0±0.50µs        ? ?/sec
construction/RLFMIndex/10000      1.04   795.4±10.27µs        ? ?/sec      1.00   766.1±12.77µs        ? ?/sec
construction/RLFMIndex/100000     1.13      8.9±0.11ms        ? ?/sec      1.00      7.8±0.04ms        ? ?/sec
construction/RLFMIndex/1000000    1.03     97.1±0.88ms        ? ?/sec      1.00     94.2±1.93ms        ? ?/sec
count/FMIndex/0.005               1.00     78.0±1.46µs  3.1 MElem/sec      2.84    222.0±8.29µs 1126.2 KElem/sec
count/FMIndex/0.05                1.00    132.6±0.74µs 1885.8 KElem/sec    3.45    457.9±5.62µs 545.9 KElem/sec
count/FMIndex/0.5                 1.00    133.2±1.57µs 1876.5 KElem/sec    4.03   536.9±12.56µs 465.6 KElem/sec
count/RLFMIndex/0.005             1.00    270.6±4.42µs 923.7 KElem/sec     2.06    557.2±8.91µs 448.7 KElem/sec
count/RLFMIndex/0.05              1.00    479.0±3.47µs 521.9 KElem/sec     2.79  1337.0±23.22µs 187.0 KElem/sec
count/RLFMIndex/0.5               1.00    591.1±3.40µs 422.9 KElem/sec     2.78  1641.3±33.68µs 152.3 KElem/sec
locate/FMIndex/1                  1.00      3.7±0.04ms 67.1 KElem/sec      4.79     17.9±0.28ms 14.0 KElem/sec
locate/FMIndex/2                  1.00      9.8±0.18ms 25.5 KElem/sec      5.40     52.9±0.99ms  4.7 KElem/sec
locate/FMIndex/3                  1.00     21.1±0.31ms 11.9 KElem/sec      5.95    125.6±1.24ms  2037 Elem/sec
locate/RLFMIndex/1                1.00     10.6±0.15ms 23.5 KElem/sec      3.05     32.5±0.23ms  7.7 KElem/sec
locate/RLFMIndex/2                1.00     30.4±0.40ms  8.2 KElem/sec      3.26     98.9±0.67ms  2.5 KElem/sec
locate/RLFMIndex/3                1.00     71.8±0.89ms  3.5 KElem/sec      3.39    243.4±2.15ms  1051 Elem/sec

Besides the test failures there is still cleanup to be done: fm-index uses u64 in many places where vers wants usize, and there are a lot of 'as u64' and 'as usize' sprinkled through right now. I want to try updating the various internals to use usize so I can reduce the casting.
There's also some expense during construction time: the WaveletMatrix  originally in fm-index works with vectors of Character, which can be converted from various smaller sizes. In order to use WaveletMatrix from vers I have to convert this to u64 first, and that may involve some extra cost.
I hope to release this as fm-index-vers or something like that in the future. fm-index hasn't had an update in about 5 years and I think it makes sense to fork it. I'll inform the original author once I choose to do that.
But first those test failures need to be fixed. And I'd like to support my XPath use cases without hindering performance somehow. Maybe you'd like to take a look?

I will definitely take a look later.
If you need a more convenient constructor for WaveletMatrix, just create a feature request, that should be easy enough. If you're benchmarking make sure to use -C target-cpu=native or enable the BMI features manually, because vers benefits a lot from them.
About refactoring from u64 to usize: Here you have to estimate costs vs benefits, because I want to migrate to u64 too (because usize makes no sense in a bitvector) but this is also part of #9 because it breaks the API. So this will take some time to happen. However, once it does happen, you can expect a factor 2 speedup of vers' wavelet matrix, so you would probably want to migrate to 2.0.

Ah if vers APIs are also likely to change I will keep u64 around for now!

I wrote two comments onto your commits. The second one contains the fix for your failing test cases.

Thank you so much! I fixed it now.
Here are the benchmarks (this time on my laptop, so the results will be somewhat different):
construction/FMIndex/1000         1.03     45.4±0.93µs        ? ?/sec      1.00     44.2±1.65µs        ? ?/sec
construction/FMIndex/10000        1.02    647.8±3.15µs        ? ?/sec      1.00    635.5±8.88µs        ? ?/sec
construction/FMIndex/100000       1.10      7.8±0.07ms        ? ?/sec      1.00      7.1±0.09ms        ? ?/sec
construction/FMIndex/1000000      1.33    101.8±4.82ms        ? ?/sec      1.00     76.7±2.58ms        ? ?/sec
construction/RLFMIndex/1000       1.09     52.7±2.04µs        ? ?/sec      1.00     48.2±0.78µs        ? ?/sec
construction/RLFMIndex/10000      1.01    686.2±2.54µs        ? ?/sec      1.00    680.5±3.61µs        ? ?/sec
construction/RLFMIndex/100000     1.03      8.0±0.12ms        ? ?/sec      1.00      7.7±0.03ms        ? ?/sec
construction/RLFMIndex/1000000    1.09    101.8±1.87ms        ? ?/sec      1.00     93.7±4.28ms        ? ?/sec
count/FMIndex/0.005               1.00     67.6±1.55µs  3.6 MElem/sec      2.46    166.1±2.72µs 1505.5 KElem/sec
count/FMIndex/0.05                1.00    118.9±4.70µs  2.1 MElem/sec      3.04    360.9±7.14µs 692.7 KElem/sec
count/FMIndex/0.5                 1.00    116.1±2.15µs  2.1 MElem/sec      3.46    402.2±1.76µs 621.5 KElem/sec
count/RLFMIndex/0.005             1.00    232.5±4.62µs 1075.2 KElem/sec    1.91    444.2±6.78µs 562.8 KElem/sec
count/RLFMIndex/0.05              1.00    426.4±9.77µs 586.3 KElem/sec     2.51  1069.7±10.15µs 233.7 KElem/sec
count/RLFMIndex/0.5               1.00   494.0±11.52µs 506.1 KElem/sec     2.60   1285.7±8.28µs 194.4 KElem/sec
locate/FMIndex/1                  1.00      3.2±0.05ms 79.1 KElem/sec      4.49     14.2±0.04ms 17.6 KElem/sec
locate/FMIndex/2                  1.00      8.3±0.02ms 30.1 KElem/sec      4.99     41.4±0.57ms  6.0 KElem/sec
locate/FMIndex/3                  1.00     18.2±0.05ms 13.8 KElem/sec      5.39     97.9±0.35ms  2.6 KElem/sec
locate/RLFMIndex/1                1.00      8.9±0.02ms 28.0 KElem/sec      2.80     25.0±0.08ms 10.0 KElem/sec
locate/RLFMIndex/2                1.00     24.9±0.12ms 10.0 KElem/sec      3.02     75.3±1.13ms  3.3 KElem/sec
locate/RLFMIndex/3                1.00     57.7±0.29ms  4.3 KElem/sec      3.25    187.8±2.91ms  1363 Elem/sec

Similar to before.
I will try it with -C target-cpu=native next.

 In general though it's still a lot slower than the FMIndex. If you know, are there space or other benefits to using an RLFMIndex?

Concerning the u64 issue: you mention you can't just convert an usize to a u64 or vice versa, you're of course right. But I cannot really escape doing so right now. You mentioned before that in 2.0 you want to change the vers APIs to use u64. Does that mean that you plan to change rank and select to run u64 rather than usize? Even in the original version there were a few usize to u64 conversions, but perhaps not the other way around, which is more dangerous.
I'm happy to migrate to 2.0 once it emerges (if this whole XML library based on this is going anywhere).

rank and select to run u64 rather than usize

yes. everything that isn't explicitely an array index will change, since it is more than feasible to have more than 2^32 bits in a vector on 32-bit architectures.

you mention you can't just convert an usize to a u64 or vice versa

The comment on your commit was meant explicitly for size_of::<usize>() because that will change the result of the calculation if usize is smaller than u64, not about casting. Maybe I misunderstood the context of this calculation, but it seemed to me that you wouldn't want to change the type in size_of just everything else. Could be wrong though.
In general, casts don't really create overhead during runtime, they just make the code less readable and more error-prone, so casting a bit for now is fine.

If you know, are there space or other benefits to using an RLFMIndex

I don't really know anything about FMIndices, so I have no idea.

I've now released fm-index-vers. Thank you for your help with this!
I've held off on any usize/u64 changes for the time being. I'm eagerly anticipating version 2.0 of Vers! :)

I went ahead with the release as the author of fm-index put in a positive emoji. But now there's a comment saying they're open for a PR. I've now opened a PR so it's possible fm-index-vers will have a short life in favor of a vers-backed fm-index.


Switch Azure DevOps Sync tool to use a service principal instead of a PAT

Azure DevOps recently added service principal auth: https://learn.microsoft.com/en-us/azure/devops/integrate/get-started/authentication/service-principal-managed-identity?view=azure-devops
We should switch our sync workflows to use managed identities instead of a PAT.
Note that this will require the sync tool to make changes: https://github.com/danhellem/github-actions-issue-to-work-item/issues/142
Once this is merged and a new release is available, we can update our workflows. Here's a pattern of what we can do (note the token passing wasn't working as expected but this is a strong start): https://github.com/radius-project/docs/blob/v0.31/.github/workflows/issues.yml#L35-L65
AB#11648

I'm running into some issues with authenticating into Azure, will work with @AaronCrawfis on this.
https://github.com/radius-project/dashboard/actions/runs/8710057735/job/23893436352

successfully implemented on the dashboard repo, moving on to the others now.
https://github.com/radius-project/dashboard/actions/runs/8804755985

PRs for remaining repos:

https://github.com/radius-project/bicep-types-aws/pull/41


yaxis axisLine lineStyle color not respected in dark mode

Version
5.1.1
Steps to reproduce
If you are using the graph with dark mode  theme the yaxis color you specify using axisLine.lineStyle.color property is not respeced
Reproduce with:

GO TO: https://echarts.apache.org/examples/en/editor.html?c=area-rainfall
MODIFY  yAxis: like this:

yAxis: [
        {
            name: '流量(m^3/s)',
            type: 'value',
            max: 500,
            axisLine: {
                lineStyle: {
                    color: "orange"
                }
            }
        },
        {
            name: '降雨量(mm)',
            nameLocation: 'start',
            max: 5,
            type: 'value',
            inverse: true,
            axisLine: {
                lineStyle: {
                    color: "red"
                }
            }
        }
    ],

switch to dark mode and your custom colors are gone.

What is expected?
If I specify a custom color it will be kept in darkmode too
What is actually happening?
darkmode ignores my custom colors

That's because drak mode using built-in setting which would override costom setting. I recommend that generate your own version of drak mode, here is tutorial


agent: move pause bin to golang

gcc attribute constructors cannot take arguments and cannot
access main function arguments on musl. Such functionality is glibc
specific and only works on x86 and x86_64. As a result, the pause
binary always quits on musl causing sandbox share pidns to malfunction.

I faced a similar issue with other cgo implementations that I did in the past, the solution was to read the arguments from /proc/self/cmdline, @bergwolf what do you think?

@devimc Good point! I've updated the PR to follow your suggestion. Thanks!

Dropping dnm as the new fix does not introduce additional memory cost.

/test

PR updated. PTAL @liwei @devimc

/test

Since this pause process will be an init process in its pid namespace, does it need to do reap the exited orphan child processes?

@lifupan That's handled by the agent reaper instead.

/test

I think the commit subject needs an update 😉

Ouch! Updated. Thanks @liwei !

/test

@lifupan That's handled by the agent reaper instead.

Hi @bergwolf,  actually the kata-agent child subreaper cannot
reap a child from different pid namespace. Read from the latest kernel source codes https://elixir.bootlin.com/linux/latest/source/kernel/exit.c#L603, the logic to find a process's reaper is below:

give them to another thread in it's thread group, if such a member exists.
give it to the first ancestor process which prctl'd itself as a
child_subreaper for its children in the same pid namespace.
give it to the init process (PID 1) in our pid namespace
That's why kernel reparent the child process to container's (PID 1) process instead of the kata-agent process.

actually after kernel applying this patch,  https://lkml.org/lkml/2017/1/30/630 , the kata-agent cannot reap the orphan child processes in sub-namespace.

@lifupan Then it's a bigger problem. Let me take a closer look.

@lifupan @bergwolf Maybe we can fix the process reap issue in a separate PR and let this PR keeps going?

@lifupan @bergwolf Maybe we can fix the process reap issue in a separate PR and let this PR keeps going?

Agree, we can open a separated issue to track it.

/test

Odd. One of the Travis builds failed with a timeout... building a golang program!
$ go version
go version go1.11.11 linux/ppc64le

    :

go build -buildmode=pie -tags "" -o kata-agent \
	-ldflags "-X main.version=1.8.0-alpha2-f1e4e9d5eda55469d34d1b59f743a059bab8f01b -X main.seccompSupport=no "

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

The build has been terminated

Any thoughts @nitkon?

@jodh-intel: That was odd. I have restarted the Travis-CI and now it's green. travis_wait can be used to exceed the timeout limit but need to root-cause why it happens else it just would fail after the exceeded timeout.

Restarted initrd CI which had a transitory failure.

flake test for initrd?
Summarizing 1 Failure:

[Fail] state container [It] with workload [true], timeWait 5
/tmp/jenkins/workspace/kata-containers-agent-ubuntu-18-04-PR-initrd/go/src/github.com/kata-containers/tests/functional/state_test.go:45

Ran 3 of 8 Specs in 31.466 seconds
FAIL! -- 2 Passed | 1 Failed | 0 Pending | 5 Skipped --- FAIL: TestFunctional (31.47s)
FAIL

@bergwolf - this should be backport candidate as well, correct?
The commit message seems to indicate the fix is not using args, whereas you are just grabbing the args now via proc sysfs, right?  Can you update commit message accordingly?

/test

@egernst Yes, I agree it should go to stable. tag added. Thanks.

rebuilding ubuntu-18-04-initrd

functional tests are failing again, nothing related to this pr, I think
• Failure [21.258 seconds]
state
/tmp/jenkins/workspace/kata-containers-agent-ubuntu-18-04-PR-initrd/go/src/github.com/kata-containers/tests/functional/state_test.go:26
  container
  /tmp/jenkins/workspace/kata-containers-agent-ubuntu-18-04-PR-initrd/go/src/github.com/kata-containers/tests/vendor/github.com/onsi/ginkgo/extensions/table/table.go:92
    with workload [true], timeWait 5 [It]
    /tmp/jenkins/workspace/kata-containers-agent-ubuntu-18-04-PR-initrd/go/src/github.com/kata-containers/tests/vendor/github.com/onsi/ginkgo/extensions/table/table_entry.go:46

    Expected
        <int>: 1
    to equal
        <int>: 0


Can we have a Monospaced font?

In the font menu, can we have a Monospaced font. So I can share programming code?
Thanks!

You can already do that with a code-block feature!

Planning to add more fonts + support for syntax highlighting soon though!

Added a monospace font in the switcher!

+1 for syntax highlighting! highlight.js seems like a good solution :)


General cleanup/refactoring

Make the codebase nicer before an eventual release (remove unused functions/variables, improve variable names etc.)

Will split this into several issues


Umbrella issue for defining schema in some common machine readable format

Problem to solve
As discussed many times, having a definition of schema in some parsable format is highly desirable. Let's discuss here possible solutions, pros, and cons for them and decide which one we want to choose.
Please keep in mind that YAML is only a format that we use to describe this because it's typical for the configuration/infrastructure world, but our YAMLs are convertible to JSONs. Furthermore, probably APIs of the platform will expect those definitions in JSON anyways. Basically, YAMLs are more human readable than bare schema. YAMLs should be able to be easily validated against the schema defined with the chosen solution.
Nice to have

popular / widely supported
support validation (required a conditional one)
support aggregation (define a piece of schema and reuse it)
code generation for popular languages
use it seamlessly in https://github.com/OpenSLO/oslo
anything else?

Proposal

https://github.com/OpenSLO/OpenSLO/issues/87
https://github.com/OpenSLO/OpenSLO/issues/20 and https://github.com/OpenSLO/OpenSLO/pull/32
https://github.com/OpenSLO/OpenSLO/pull/113
https://github.com/OpenSLO/OpenSLO/pull/129
https://spec.openapis.org/oas/v3.1.0
something else?

OpenAPI gets my vote given that it will auto-document in Backstage.io: https://backstage.io/docs/features/software-catalog/descriptor-format#kind-api

OpenAPI  is meant more for speccing APIs, no?  This seems more like a request for a way to spec schemas.  There are a couple of interesting PRs to that effect open:
https://github.com/OpenSLO/OpenSLO/pull/129
https://github.com/OpenSLO/OpenSLO/pull/113

I agree that OpenAPI is more focused on APIs, so I don't think that's the right fit.
I think that this doesn't have to be an either/or proposition either though.  I like the idea of having the spec in Go structs in here from @kenfinnigan's PR #113 .  We could use those to  build on the work that @CRThaze did in #129 and generate the json-schema from the structs (probably via oslo) as well as cue.

fwiw, I've already used @kenfinnigan's Go structs with great success to generate OpenSLO yaml docs from a DB of SLOs that I've been maintaining.

Thanks @ian-bartholomew
I agree it's not a one and done situation. All we need is a central mechanism to define the schema, and CI processes can generate other formats as part of release or through other tools. Whichever we'd prefer

So now that we've merged the JSON-Schema definitions, I've been looking at ways we might leverage it. I think @kenfinnigan is correct in their comment. Some decision should be finalized about figuring out what the ultimate source of truth for a common machine-readable format should be, before we stumble too far down the road in one direction or another.
Currently, we have two handcrafted machine-readable definitions:

https://github.com/OpenSLO/OpenSLO/pull/129
https://github.com/OpenSLO/OpenSLO/pull/113

I'm of the opinion that we should hopefully be maintaining only one of these two, and deriving the other from it. This will avoid drift between them and reduce the complexity of future changes.
Though as the author of the JSON-Schema approach (and as someone who also needs the spec defined as JSON-Schema draft-07 for my Backstage work) I'm not an unbiased commenter; I'd like to make a pitch for having JSON-Schema be that primary source of truth.
Oslo Validation
99% of all validations we want Oslo to perform (and possibly more than are currently supported by Oslo) are currently implemented within the unit-test code currently checked into this repo; using far less code (and still less if you ignore the test-table of inputs in that file). This is a strong advantage of using a broadly adopted and language agnostic definition for the schema. However, as I pointed out in the PR, there are a couple of rules defined in the spec which are difficult to validate using a static definition like JSON-Schema. Those remaining rules are where additional logic will need to be written in Oslo to perform. Yet the overall code-footprint of Oslo could be greatly reduced by implementing the logic used in the unit-test, and then using targeted tests for certain fields after deserializing a document into a generic map[string]interface{} for the remaining validations.
That is not to say that there aren't still other benefits to having the spec represented as structs, and then using those structs within Oslo to perform the additional validations. That would, after all, probably result in cleaner and even shorter logic for those remaining validations. Instead, I'm trying to highlight that a native Go representation of the schema is not necessary for Oslo to validate OpenSLO documents.
SDK/Library Code
A key advantage to having a native Go struct representation of the schema, as I see it, would be to assist those wanting to write code which consumes and implements OpenSLO. However, that use-case is not limited to projects written in Go. And it's very easy to imagine that other languages would also benefit from having a library they can use to serialize/deserialize OpenSLO documents using objects native to the language. That then gives an advantage to making the source of truth a language agnostic standard like JSON-Schema, which can then be used to generate schema models in various languages using popular polyglot tools like quicktype.
OpenAPI
A quick note about OpenAPI and JSON-Schema. OpenAPI 2.0+ is technically an “extended subset” of JSON-Schema, and as of OpenAPI 3.0 the divergence is narrowing. There do exist some tools which attempt to translate between them by removing unsupported keywords from JSON-Schema documents, but those looking to have an API spec simply import our published JSON-Schema would likely need to perform some translation. This is a situation where the tools from generation from Go (or other language) native objects are probably more well suited. But we can still provide OpenAPI object definitions from JSON-Schema through either JSON-Schema -> OpenAPI Schema or JSON-Schema -> <Go or some other language model> -> OpenAPI Schema.
quicktype
I haven't used quicktype before, but the project seems fairly robust (>8K ⭐ and extensible language support).
Running the following produced some interesting results very quickly:
npm install -g quicktype
quicktype -s schema --src ./schemas/v1 -o models.go

The output left me scratching my head a bit and might need some massaging, or maybe our schema will need to be to help it along, but it seems promising and worth exploring.
Conclusion
Using Go structs as the source of truth certainly seems like it would be possible, and there do exist some tools that can render JSON-Schema from them. But as of right now the impression I'm getting is that going from JSON-Schema to Go and other languages will be more flexible, and is a more well trodden path. Either approach will, however, require a bit more investigation and so long as we end up with both JSON-Schema that Backstage can use, and native language objects, both being derived from a single source of truth; I'll be happy. No matter the direction.

Hi everybody, I've been trying to read through various issues and I have to admit it's not really easy to follow what's the current status of everything, or perhaps the intentions/goals/vision.
I see that a lot of discussion is centered around oslo and there's not much "else" built on top of OpenSLO out in the open, at least not from what I can find. I have a use case, where I'd love to let our developers manage their SLOs purely through Kubernetes, through OpenSLO and manage the specifics of our observability stack on the cluster level. I'd love to use OpenSLO for that and initially I thought I'm just going to figure out how to take the json schema in main repository and turn it into CRDs. Then I read through all these issues and I'm not quite sure that that would be the best approach. In some issues there were mentions of it not being up to date...
So, what would the recommended approach be, to get CRDs, in order to start building some tooling on top of them?

hey @djetelina, I think for now the best way to move forward is with oslo. As of v0.11.0 it is a usable Go package, here's an example usage: https://github.com/OpenSLO/oslo/blob/main/examples/example.go, so If there's going to be an operator managing these CRDs you could just use oslo to validate the extracted/converted definition. I understand the problem you're facing would be easier solved with a JSON Schema and I myself want to push in this direction, I'm still not sure if pure JSON Schema is the way to go or whether we'd be better off with sth like cue. Feel free to get involved in these discussions too :)

https://github.com/OpenSLO/OpenSLO/issues/87 - pinning for reference. If you'd have the schema to work with @djetelina would you consider making the CRDs public and part of the OpenSLO?

Definitely, it wouldn't make much sense to start building anything on these foundations if I (or my team) weren't willing to make them stronger by our contributions :)
I don't (and can't) promise anything, but I've managed to infect at least one of my colleagues with the idea of Kubernetes SLO operator, so we'll see where the idea is going to take us.
For some backstory, I've written very simplistic https://github.com/heureka/omni-slo-generator in the past which we're running in production, but one of it's major flaws is it just wrapping https://github.com/google/slo-generator which creates very inefficient load on our Mimir cluster (loads of 28d queries instead of using ruler), as well as being fairly complicated to get up and running, not solving alerts etc. As we spend more and more time helping our teams get it up and running leading into unnecessarily overscaled cluster, it's slowly starting to be worth our time to build a robust solution.

I am definitely for quicktype @CRThaze mentioned.
With some well defined JSON-Schemas including good titling for it to properly name things as we would like, we (or the consumers of the OpenSLO spec) would be able to (what seems like reliably) generate code for whatever project would need it.
From my (biased) point of view it would be great if we had:

The JSON-Schema as the source of truth
Validated Go structs generated by quicktype based on the JSON-Schema
Generated Kubernetes CRDs either directly from the JSON-Schema or from the Go structs

hey @fourstepper thanks for chiming in!
I'm torn on this one, mainly because I've had some decent JSON Schema exposure in the past, and It was pure pain. The main problems with JSON Schema revolve around the fact the validation results vary based on the implementation.
I've had yamlls and ajv not only resolve the URI's between the subschemas differently but also produce different errors altogether. I've dealt with the first issue by bundling them into a single file, but the latter had no answers.
Crafting a proper schema is not an easy task, what might work well on one implementation of the parser might not go so well on another one. There are also caveats to the schema itself, I've mentioned to @CRThaze here https://github.com/OpenSLO/OpenSLO/pull/151 the example of oneOf not doing what you think it might do with complex type (objects). The errors you're getting are crafted by the implementation, and they often tend to be ambiguous to say the least.
You have to also take the schema version into consideration, many implementations support draft 07 and don't plan on moving forward, this obviously limits what you can do with the schema.
Coming back to Go, both libraries mentioned in the aforementioned PR are not maintained anymore, leaving us with almost nothing. While we don't want to focus on Go and k8s specifically, we should take that into consideration since oslo is written in it.
Cue converts natively to both openapi and CRDs, having openapi spec generated opens up a way to convert it to JSON Schema eventually. If you have openapi you CAN generate into any language you'd like.
What it brings to the table I really like compared to JSON Schema is that it is reliable, it will produce the same exact error wherever you run it, given you use cue to do the validation. It reduces the toil of maintaining the JSON Schema to be compatible with different implementations. It can be also easily embedded into oslo and is generally much more readable and easier to write than JSON Schema.

@nieomylnieja You sound convincing :)
If OpenAPI is enough for languages to get up and running and we would be able to generate that using CUE, I think that would be enough on the "support other languages than Golang" front

We've decided to go with Go structs as the one source of truth.


[SPARK-34353][SQL] CollectLimitExec avoid shuffle if input rdd has single partition

What changes were proposed in this pull request?
if child rdd has only one partition, skip the shuffle
Why are the changes needed?
skip shuffle if possible
Does this PR introduce any user-facing change?
No
How was this patch tested?
existing testsuites

We may have more operators that adding shuffle in the doExecute method instead of the planner

@cloud-fan shuffle only is directly added in the doExecute method of TakeOrderedAndProjectExec, CollectLimitExec and ShuffleExchangeExec

related to https://github.com/apache/spark/pull/31409

Test build #134860 has started for PR 31468 at commit 6d199ff.

cc @viirya @maropu

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39447/

Test build #134860 has finished for PR 31468 at commit 6d199ff.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39447/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39447/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134860/

retest this please

Test build #134874 has started for PR 31468 at commit 6d199ff.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39460/

Test build #134874 has finished for PR 31468 at commit 6d199ff.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134874/

org.apache.spark.sql.CachedTableSuite.SPARK-34269: cache lookup with ORDER BY / LIMIT clause failed more than one time. But seems it passed without this change?

The fix itself looks fine.

retest this please

Test build #135069 has started for PR 31468 at commit 6d199ff.

scala> spark.sql("CREATE TABLE t (key bigint, value string) USING parquet")
res0: org.apache.spark.sql.DataFrame = []

scala> spark.sql("SELECT COUNT(*) FROM t").head
res1: org.apache.spark.sql.Row = [0]

scala> val t = spark.table("t")
t: org.apache.spark.sql.DataFrame = [key: bigint, value: string]

scala> t.count
res2: Long = 0

scala> t.rdd.getNumPartitions
res3: Int = 0

scala>

scala> spark.sql(s"CACHE TABLE v1 AS SELECT * FROM t LIMIT 10")
java.util.NoSuchElementException: next on empty iterator
  at scala.collection.Iterator$$anon$2.next(Iterator.scala:41)
  at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
  at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:63)
  at scala.collection.IterableLike.head(IterableLike.scala:109)
  at scala.collection.IterableLike.head$(IterableLike.scala:108)
  at scala.collection.mutable.ArrayOps$ofRef.scala$collection$IndexedSeqOptimized$$super$head(ArrayOps.scala:198)
  at scala.collection.IndexedSeqOptimized.head(IndexedSeqOptimized.scala:129)
  at scala.collection.IndexedSeqOptimized.head$(IndexedSeqOptimized.scala:129)
  at scala.collection.mutable.ArrayOps$ofRef.head(ArrayOps.scala:198)
  at org.apache.spark.sql.Dataset.$anonfun$count$1(Dataset.scala:3024)
  at org.apache.spark.sql.Dataset.$anonfun$count$1$adapted(Dataset.scala:3023)
  at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3705)
  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
  at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3703)
  at org.apache.spark.sql.Dataset.count(Dataset.scala:3023)
  at org.apache.spark.sql.execution.datasources.v2.BaseCacheTableExec.run(CacheTableExec.scala:65)
  at org.apache.spark.sql.execution.datasources.v2.BaseCacheTableExec.run$(CacheTableExec.scala:41)
  at org.apache.spark.sql.execution.datasources.v2.CacheTableAsSelectExec.run(CacheTableExec.scala:88)
  at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:42)
  at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:42)
  at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:48)
  at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
  at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3705)
  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
  at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3703)
  at org.apache.spark.sql.Dataset.<init>(Dataset.scala:228)
  at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:615)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:610)
  ... 47 elided

scala> spark.sql("SELECT COUNT(*) FROM v1").head
java.util.NoSuchElementException: next on empty iterator
  at scala.collection.Iterator$$anon$2.next(Iterator.scala:41)
  at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
  at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:63)
  at scala.collection.IterableLike.head(IterableLike.scala:109)
  at scala.collection.IterableLike.head$(IterableLike.scala:108)
  at scala.collection.mutable.ArrayOps$ofRef.scala$collection$IndexedSeqOptimized$$super$head(ArrayOps.scala:198)
  at scala.collection.IndexedSeqOptimized.head(IndexedSeqOptimized.scala:129)
  at scala.collection.IndexedSeqOptimized.head$(IndexedSeqOptimized.scala:129)
  at scala.collection.mutable.ArrayOps$ofRef.head(ArrayOps.scala:198)
  at org.apache.spark.sql.Dataset.head(Dataset.scala:2747)
  ... 47 elided

scala> val v1 = spark.table("v1")
v1: org.apache.spark.sql.DataFrame = [key: bigint, value: string]

scala> v1.count
java.util.NoSuchElementException: next on empty iterator
  at scala.collection.Iterator$$anon$2.next(Iterator.scala:41)
  at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
  at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:63)
  at scala.collection.IterableLike.head(IterableLike.scala:109)
  at scala.collection.IterableLike.head$(IterableLike.scala:108)
  at scala.collection.mutable.ArrayOps$ofRef.scala$collection$IndexedSeqOptimized$$super$head(ArrayOps.scala:198)
  at scala.collection.IndexedSeqOptimized.head(IndexedSeqOptimized.scala:129)
  at scala.collection.IndexedSeqOptimized.head$(IndexedSeqOptimized.scala:129)
  at scala.collection.mutable.ArrayOps$ofRef.head(ArrayOps.scala:198)
  at org.apache.spark.sql.Dataset.$anonfun$count$1(Dataset.scala:3024)
  at org.apache.spark.sql.Dataset.$anonfun$count$1$adapted(Dataset.scala:3023)
  at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3705)
  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
  at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3703)
  at org.apache.spark.sql.Dataset.count(Dataset.scala:3023)
  ... 47 elided

scala> v1.rdd.getNumPartitions
res7: Int = 0

scala> v1.repartition(3).count
res8: Long = 0

it seem that count method does not work if the number of partitions of a View is zero.
childRDD may have no partition, then the number output partition is zero;
while in existing impl, the ShuffledRowRDD will make sure SinglePartition, so that this issue will not be triggered.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39651/

Test build #135070 has started for PR 31468 at commit 0bfb465.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39652/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39652/

Test build #135069 has finished for PR 31468 at commit 6d199ff.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #135070 has finished for PR 31468 at commit 0bfb465.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135070/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39652/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135069/

retest this please

Test build #135075 has started for PR 31468 at commit 0bfb465.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39657/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39657/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39657/

Test build #135075 has finished for PR 31468 at commit 0bfb465.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135075/

Test build #135090 has started for PR 31468 at commit c9a5a81.

Test build #135092 has started for PR 31468 at commit 75c4d92.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39672/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39672/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39672/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39674/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39674/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39674/

Test build #135090 has finished for PR 31468 at commit c9a5a81.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135090/

retest this please

Test build #135097 has started for PR 31468 at commit 75c4d92.

Test build #135092 has finished for PR 31468 at commit 75c4d92.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135092/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39679/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39679/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39679/

Looks fine. Could you update the PR title and description, too? This PR is not only for a single partition case now.

Test build #135097 has finished for PR 31468 at commit 75c4d92.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135097/

retest this please

run one more tests on the latest master before merging this.

Test build #135391 has started for PR 31468 at commit 75c4d92.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39971/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135391/

retest this please

Test build #135510 has started for PR 31468 at commit 75c4d92.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40091/

Test build #135510 has finished for PR 31468 at commit 75c4d92.

This patch fails from timeout after a configured wait of 500m.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135510/

Jenkins retest this please

Test build #135523 has started for PR 31468 at commit 75c4d92.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40104/

Test build #135523 has finished for PR 31468 at commit 75c4d92.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135523/

Thanks! Merged to master.

Thank you so much !


Missing symfony deps

Need to add "psr/log": "~1.0", to require, since symfony/console only has require-dev and doesn't pull it in.

This is in the require section now, closing =) https://github.com/silverstripe/cow/blob/c1ab90c7fa97d95d410934866eafa3ee6a50cb7d/composer.json#L26


Decrease RAM usage of Import Sample Dialog

Currently its not possible to open a directory containing more than about 25 samples because of the large amounts of ram used by the Import Sample dialog that came from LGPT.
This occurs for several reasons:

I_Dir is just a subclass of T_SimpleList<Path> so that everyfile in a directory is actually stored in the list with its full path not just the filename, so the containing directories path is stored in every files Path object
The whole contents of a directory are read into the I_Dir list when entering the directory, even though only 15 (LIST_SIZE) can be displayed at any one time by the UI, but this is necessary in order to be able to sort the file list for a dir
The ImportSampleDialog itself maintains another list sampleList_ that it then copies (based on filter criteria, eg, if dir, if *.wav, the items from the list returned in the I_Dir object by picoTrackerFileSystem.

In my initial testing, it seems there is only a few (3-5?) KB of ram available for the ImportSampleDialog to use so the above code is very limiting in what size of dirs can be handled at the moment.
To try to improve this my plan is to create a new PagedImportSampleDialog class that uses paging, so that it only ever requests the 15 items it needs to display at any time. Then I need to create a new interface, I_PagedDir that supports being given a dir path and a s start index and it returns the 15 items from that start index.
But the tricky thing here is: sorting. In order to be able to sort a directory, the new I_PagedDir would need to read and store every filename in a dir and that could just be itself run out of ram in large dirs: eg. for a dir with 300 files, average filename length of 20 chars gives: ~5.9KB and blows the ram budget. SDFat is capable of dealing with files just by a "directory index" int BUT then there is no way to alphabetically sort the contents :-(
So the 2 options I can see for the above are:
a. allow upto thousands of files per dir BUT they will not be sorted when displayed (except for subdirs which could at least be sorted to be at the start of the list)
b. limit the number of files based on filename lengths this could be anywhere from 20 to 100 but still allow for displaying them in alpha sorted order.

Couldn't we just do a much slower but potentially fast enough approach? Thinking if we could build a list of directory indices and sort it by reading from SD each time we need to.
Another advantage of this approach is that we could use static memory for this list, say we want to support 100 files per directory, then we use a static 400 bytes (indices are uint32) for storing and sorting files in each view. Even wonder if we could cast indices to a smaller int in practice.
Otherwise I'd just go with no sorting probably.
Option b is not something we want to do. While there is a lot of dynamic memory being used in picoTracker, any new features should be developed with a fixed memory footprint in mind. Otherwise it's really hard to add/remove features without breaking things.


Change more info link

We should find a link that explains the CCPA or make a quick writeup within the extension (probably the latter) that explains the CCPA and gives more info on it

An easy fix could be just linking to the readme on Github, I tried to provide an accessible background discussion there

I think that’d be a good idea. The Readme looks fantastic BTW.
Rather than linking to Github, is there a way to pull up the file locally and display that? Doing that might be easier, and since the file is already packaged locally, might as well use it!

I ended up linking it towards whatever page is listed in the manifest


suggest dataset title depending on content

As the collection id will now be a random string, we need a human readable title which aligns with our dataset naming convention:
<short_dataset_description><information_content>
year -> to be derived from valid_time
region -> to be derived from dataset extent --> convert to 3 letter country or region code
short dataset description -> can initially be filename of uploaded dataset. The user is allowed to change this part and ONLY  this part
type -> POINT or POLY
information_content -> 3 digit code specifying presence of land cover, crop type and irrigation labels
Also tagging @cbutsko here...

I will try to come up with an initial Python script to do this...

plan is to add a new attribute to the RDM called "ref_id", which should host this information and the user can still edit the dataset title freely...

@jdegerickx I am not sure of a library which can return me the region based on the location. can you suggest?
Rest of the items i can generate from dataset.

@santoshkaranam, @cbutsko, I created a script to determine ref_id for a new dataset automatically --> https://github.com/WorldCereal/worldcereal-referencedata/blob/main/harmonization/assign_ref_id.py
As input you currently need a path to a file (can be replaced with geodataframe) and a custom part of the ref_id (e.g. LPIS).
Please have a look and check if it makes sense.

@jdegerickx from the first glance looks fine to me! do you mind if I take your code and refactor it to a FastAPI call, just like the other calls we have here?
@santoshkaranam Santosh, I guess for you it will also be the easiest way to use it if you can call it like that?
and I will need either a path so that I can read the processed dataset (just as in reprojection function), or maybe just a collection_id, and we can just query everything we need for the title/ref_id creation from the RDM directly.
Let me know what you guys think.

thanks @cbutsko ! Yes, feel free to convert to fastAPI so Santosh can actually use it ;)

@jdegerickx @cbutsko, if there is a shape file with countries, it would be easy for me to just make it a parquet file and put in cloud storage and use duckdb to query with a point from user dataset (or centroid point of 1st polygon) and get the region letters. It would be Kubernetes container friendly also. Referring to local files in a container is a bit tricky and increases the size of the deployed container.
The rest of the things I already have in my code. Year, Point/Polygon, fileName part (First 5–10 Letters may be) and Crop code and Irrigation Codes.  Also, I need to check in the database if it is a duplicate and append numbers only to the custom part.
Can you share the shape file With regions? I can put in cloud storage and provide the link. Then we can write python code to query and return region letters and convert to fastAPI.

@santoshkaranam, the shapefile is located on terrascope. You can find the path in the code I shared, you should be able to access it.
Just be careful with querying for the country code using a single point only, as the dataset might cover multiple countries or even the whole globe... In the code I provided, I first compute a union on all features before determining the country code...

added FastAPI call to retrieve dataset's country/region code f613347e63bed78054471f0fb38dbcc0667a374d

@cbutsko will integrate this to add new Title.

Currently, generating the title and collection IDs based on the format is done automatically. For duplicate names, the identifier is appended with numbers.
Title can also be edited to later as of now.
Sample duplicates.

@santoshkaranam, I have tested the procedure.
One remaining issue relates to automated assigning of sample_id to all samples in a dataset.
Right now, you only use the unique identifier as a basis for the sample_id. This results in sample_id's like:

Which is not that informative...
The preferred way here is that you actually use the full collection ID as the basis for the sample id, so we would have something like:
2022_glo_vito_point_100-1
2022_glo_vito_point_100-2
...
(also note the hyphen, which makes it easy to read)
Could this be adjusted?
Also this way, the identifier does not need to be unique, which is a huge advantage for the user!
(if you agree, can you remove the "Unique" in the guideline?

Now the sampleIDs are assigned using collection-


added PWA

fixes issue #21

Okay sure will do ASAP. Sorry for trouble I think I'm using ESlint which caused the issue.

Okay sure will do ASAP. Sorry for trouble I think I'm using ESlint which caused the issue.

@DevanshCodes : Don't worry, it's not a trouble. thanks for contributing. All you need to do is a simple rebase and we are good I think. 👍🏼

Hey @SudharakaP Can you check it now?

Hey @SudharakaP Can you check it now?

There's still unnecessary formatting and it seems that you haven't re-based from master; https://github.com/OpenArchitex/AccoTech/pull/22/files. One other easy method you could do this is run prettier --write "{,src/**/}*.{md,json,ts,tsx,css,scss,yml,html}" within the project folder which will reformat everything for you. Then you will be left with the actual changes you did without the formatting.

Hey @SudharakaP Can you check it now?

There's still unnecessary formatting and it seems that you haven't re-based from master; https://github.com/OpenArchitex/AccoTech/pull/22/files. One other easy method you could do this is run prettier --write "{,src/**/}*.{md,json,ts,tsx,css,scss,yml,html}" within the project folder which will reformat everything for you. Then you will be left with the actual changes you did without the formatting.

Done the same and pushed. Let me know if its okay or not.

@DevanshCodes : It's better but not quite. For some reason still you have other formatting changes included in your PR; https://github.com/OpenArchitex/AccoTech/pull/22/files. But that's okay. I am going to merge this and I'll fix it afterwards. Thanks much for the contribution. 👍🏼

@DevanshCodes : Thanks much for the contribution. Feel free to pick any other unassigned issues in the repo if you feel like it. 😄


Skip path items that are not HTTP verbs

As they don't correspond to resource routes, currently this just
includes parameters.
See https://github.com/swagger-api/swagger-spec/blob/master/versions/2.0.md#path-item-object
Addresses #59

Thanks @jszwedko, this is great! I had overlooked that parameters could be placed directly under the path. Knowing this now I can DRY up my own swagger :) I've raised another PR to add an example to the main test swagger here https://github.com/westfieldlabs/apivore/pull/61

@hornc awesome, thank you! Apologies, I meant to add a test, but must have forgotten.


documentation: tutorial, penguins example

I just started learning about AlgebraOfGraphics, in the hope of switching over from ggplot2 in R. I love it! I could use more help from the documentation. These tutorials have helped a lot for things I could not find here.
This PR is to fix 2 "errors" I ran across as a beginner, when learning to use AoG.
Here is a small example to reproduce the first error:
using AlgebraOfGraphics, CairoMakie
set_aog_theme!()
df1 = (x=[1,2,3,1,2,3], y=[4,3,1,5,3,2])
plt = data(df1) * mapping(:x,:y)

# replicating the tutorial example works:
axis = (width = 225, height = 225)
draw(plt; axis)

# but this fails, if I rename the variable 'axis'
# which was counterintuitive (at least for me not being familiar with Makie / AoG)
ax = (width = 225, height = 225)
draw(plt; ax)
# ERROR: MethodError: no method matching draw(::Layer; ax=(width = 225, height = 225)) ...

The suggested change makes it more explicit that axis is a keyword option, and helps to learn how to tweak figures.
The second "error" is that the png figure created with the save(...) example contains a tiny part of the plot only (top left corner). The suggested change doesn't have this issue, but uses svg instead. I don't know how to fix the issue with png, sorry (still learning!).
fg = draw(plt; axis=(; width = 225, height = 225))
save("figure.png", fg, px_per_unit = 3) # to reproduce the problematic (too small) png file
save("figure.svg", fg) # the svg file looks good.

Thank you! :) Great to have the docs more user friendly!
The problem with png output is an actual bug, which should be fixed by:
https://github.com/MakieOrg/Makie.jl/pull/2470

Thank you for the PR!
I see that the special syntax draw(penguin_frequency; axis) could be confusing if one does not know the rule, but I think we should go for a consistent alternative.
I would suggest one of the following options

replace the variable declaration axis = (width = 225, height = 225) with ax = (width = 225, height = 225) and update all the calls to be draw(...; axis=ax), or
leave the variable declaration axis = (width = 225, height = 225) as is, but use the explicit form draw(...; axis=axis)
leave the code as is, but add a small note at the beginning about the special syntax draw(...; axis).

Re pngs: ideally that part can be reverted now that the upstream bug is fixed?

That makes total sense. I went for option 2, which seemed best for a user with little Julia experience (compared to option 3). Also reverted the figure format to png.

Perfect, thanks!


Adds Probability Distribution

This pull request Implements ProbabilityDistribution interface with the following dependencies:
* AbstractProposition class
* AbstractTermProposition class
* TermProposition Interface (functional)
* AssignmentProposition class
The code explanation and comments are taken from AIMA-3e (Indian Subcontinent version)
Resolves #173

@norvig
Please review this pull request. I will be adding Full-Joint-distribution soon.


Support linebroken import lists

rustfmt wanted to make this change (and many larger, just like this). I'd like to have an option to _not do_ this, to preserve the list formatting, with trailing comma, so that the code is easy to read and edit.
@@ -1,22 +1,19 @@
 use std::fmt;
-use super::{
-    ArrayBase,
-    Data,
-    Dimension,
-};
+use super::{ArrayBase, Data, Dimension};

@@ -87,14 +80,8 @@ pub use stride_error::StrideError;
 pub use si::{Si, S};

 use iterators::Baseiter;
-pub use iterators::{
-    InnerIter,
-    InnerIterMut,
-    OuterIter,
-    OuterIterMut,
-    AxisChunksIter,
-    AxisChunksIterMut,
-};
+pub use iterators::{InnerIter, InnerIterMut, OuterIter, OuterIterMut,
+                    AxisChunksIter, AxisChunksIterMut};

If you're curious, these are my settings:
ideal_width = 80
max_width = 85
fn_call_width = 80
fn_return_indent = "WithWhereClause"
fn_args_density = "Compressed"
normalise_comments = false
where_density = "Tall"

This alone is a huge part of why we don't use rustfmt on our team.

My comment from a dup issue:

It's a common style to have import names in individual lines, in "Block" style. It's my personal preference, as well, since it allows faster lookup with eyes and easier addition/removal of items with small diff footprint.
Example:
use freetype::freetype::{
    FT_Face,
    FT_Get_Char_Index,
    FT_GlyphSlot,
    FT_Init_FreeType,
    FT_LOAD_DEFAULT,
    FT_Library,
    FT_Load_Glyph,
    FT_New_Face,
    FT_Render_Glyph,
    FT_Render_Mode,
    FT_Set_Pixel_Sizes,
};

Looking at other options with "Block" variant, I think the name of this option should be imported_names_style or imported_names_layout. (I don't see how layout and style options are different, so please advise.)
I think I can take a shot at implementing it myself. Is there a process to approve options before getting to implementation? (I didn't see anything on this in Contributing.md.)

Looking at the code now, and looks like ListTactic, used also for the where_layout option, is the best type to use for this option, with Vertical value giving us what's asked in this issue, Mixed  being the current behavior, and more options to choose from.
pub enum ListTactic {
    // One item per row.
    Vertical,
    // All items on one row.
    Horizontal,
    // Try Horizontal layout, if that fails then vertical.
    HorizontalVertical,
    // HorizontalVertical with a soft limit of n characters.
    LimitedHorizontalVertical(usize),
    // Pack as many items as possible per row over (possibly) many rows.
    Mixed,
}

This is the main reason why I do not use rustfmt...

rustfmt has imports_layout and imports_indent options to solve this issue (via #1785).
Currently these options are only available on the latest master branch. Please try!


Server fails to load SpaceEngineers.ObjectBuilders.dll

` setlocale('en_US.UTF-8') failed, using locale: 'C'. International characters may not work.
ing stderr to '/root/Steam/logs/stderr.txt'
on file ./public/steambootstrapper_english.txt which appeared to exist failed!!!
erifying installation...
ownloading Update...
!! Fatal Error: Steamcmd needs to be online to update. Please confirm your network connection and try again.
0/threadtools.cpp (3659) : Assertion Failed: Illegal termination of worker thread 'Thread(0x0x56ef2f10/0x0xf7c0bb'
07 17:54:07.320: App Version: 01_197_073
07 17:54:07.369: Is official: True [NO][NIS][NAMP]
07 17:54:07.372: Environment.ProcessorCount: 24
07 17:54:07.388: Environment.OSVersion: Microsoft Windows 7 Professional (Microsoft Windows NT 6.1.7601 Service Pack 1)
07 17:54:07.388: Environment.CommandLine: "Z:\appdata\space-engineers\SpaceEngineersDedicated\DedicatedServer64\SpaceEngineersDedicated.exe" -noconsole -path Z:\appdata\space-engineers\SpaceEngineersDedicated -ignorelastsession
07 17:54:07.388: Environment.Is64BitProcess: True
07 17:54:07.388: Environment.Is64BitOperatingSystem: True
07 17:54:07.390: Environment.Version: .NET Framework 4.8.3761.0
07 17:54:07.391: Environment.CurrentDirectory: Z:\appdata\space-engineers\SpaceEngineersDedicated\DedicatedServer64
07 17:54:07.391: CPU Info: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
does not meet minimum requirements, forcing Experimental mode
ould not load necessary assembly dependencies. C# redistributable may be missing. Affected file: file:///Z:\appdata\space-engineers\SpaceEngineersDedicated\DedicatedServer64\SpaceEngineers.ObjectBuilders.dll
`

Please don't just dump the log output here.
I would look into this line: "!! Fatal Error: Steamcmd needs to be online to update. Please confirm your network connection and try again.".
Did you try other containers like devidians?
https://github.com/Devidian/docker-spaceengineers

Please don't just dump the log output here.
I would look into this line: "!! Fatal Error: Steamcmd needs to be online to update. Please confirm your network connection and try again.".
Did you try other containers like devidians?
https://github.com/Devidian/docker-spaceengineers

Hi, the file exists in the install, and it is an issue with devidan container too
The base os is centos8
Do you currently have this container working / running on any system?

Hi, the file exists in the install, and it is an issue with devidan container too
The base os is centos8
Do you currently have this container working / running on any system?

Yeah, try ubuntu 20.04 or the latest lts debian as a base. Also, are you building this yourself or using the dockerhub image?

Yeah, try ubuntu 20.04 or the latest lts debian as a base. Also, are you building this yourself or using the dockerhub image?


New version: Google.Chrome.Dev.EXE version 128.0.6585.0

Created by 🥟 Dumplings in workflow run #6516.

Log
Updated: 128.0.6559.0 → 128.0.6585.0
Submitting WinGet manifests
Creating manifests
Uploading manifests and making commits
Creating a pull request

Microsoft Reviewers: Open in CodeFlow

It appears that this versoin became promoted from Canary to Dev.


NoSQL backend

I suspect butane will get a lot more interest if it also supported at least one non-SQL based backend.  This is something that is a fairly unique opportunity to butane that other Rust ORM's are not suited for.
The easiest would likely be redis or similar caching system, which very likely wouldnt be able to support the butane query system beyond pkey fetches.  But it would demonstrate that NoSQL is possible, and people interested in more featureful NoSQL like mongo, etc would have a base implementation to work from.

Does any ORM do this? I thought NoSQL meant document store and manual handling of relations (as in the client has to handle everything). That would seem like a massive undertaking to redo all of the relational modeling and handling (especially constraints) in a library written by 1-3 contributors.
Maybe I'm wrong as I don't use any document stores or NoSQL, so feel free to point me to NoSQL databases that would be good candidates that don't require writing DB logic in the client lib (butane)

Hi @LoveIsGrief , thanks for raising valid concerns.
My team is already pushing butane models into redis, so this task would involve moving our custom logic into butane, which isnt a small task because butane internals are necessarily more complex, and would need more generic tests created, but it is doable.  And we only need to do a basic NoSQL support - I am betting that the butane team will grow when people can see butane can do NoSQL, as they will want to improve/extend it.
Newer ORMs do often provide NoSQL support.  For example, Mongo ORMs.  Notably on that list, Prisma is Rust underneath and is quite nice, but we selected the less mature butane because it was pure Rust and thus has a similar user-base and set of objectives as we have.  https://www.prisma.io/docs/concepts/overview/prisma-in-your-stack/is-prisma-an-orm is a nice explanation of how it, and other ORMs like TypeORM, are not strictly tied to a relational structure.
https://github.com/franklin-ai/butane/commit/6e34de770e9801a816795ad554eb354dfbb9b9c7 is the rather over-simplified approach to avoid re-doing all of the butane internals to allow easy NoSQL support.  It uses https://github.com/jayvdb/once_cell_serde instead of the standard once_cell.  This approach was taken primarily so that we have few modifications to butane, as we want to stay close to main.  I am about to rebase that today, and it will be much smaller now that all of the fake changes are also in main.  That commit allows serialising the relations and the related data inline - that means there are extra structures in the JSON that describe the fkeys and many, which are not needed because the related data is embedded.  There are many ways to improve it, but it was suffice for our immediate needs.

It would be desirable to do this after https://github.com/Electron100/butane/pull/22 , otherwise it will make that feature harder to complete.


Fail gracefully if disable_warnings not available

Addresses #418. Rationale explained there.

Thanks :+1:

Thanks :+1:


[Event Request] codeunit 1173 "Document Attachment Mgmt" OnAfterSetDocumentAttachmentFiltersForRecRefInternal

🕮 Describe the request
Could you please add the following event, that we can set filters for custom tables, whenever this method is called?
internal procedure SetDocumentAttachmentFiltersForRecRefInternal(var DocumentAttachment: Record "Document Attachment"; RecRef: RecordRef; GetRelatedAttachments: Boolean)
var
    FieldRef: FieldRef;
    RecNo: Code[20];
    DocType: Option Quote,"Order",Invoice,"Credit Memo","Blanket Order","Return Order";
    LineNo: Integer;
    FieldNo: Integer;
    VATRepConfigType: Enum "VAT Report Configuration";
begin
    if GetRelatedAttachments then
        SetRelatedAttachmentsFilter(RecRef.Number(), DocumentAttachment)
    else
        DocumentAttachment.SetRange("Table ID", RecRef.Number());

    if TableHasNumberFieldPrimayKey(RecRef.Number(), FieldNo) then begin
        FieldRef := RecRef.Field(FieldNo);
        RecNo := FieldRef.Value();
        DocumentAttachment.SetRange("No.", RecNo);
    end;

    [...]
    //+EVENT
    OnAfterSetDocumentAttachmentFiltersForRecRefInternal(DocumentAttachment, RecRef, GetRelatedAttachments);
    //-EVENT
end;

[IntegrationEvent(false, false)]
local procedure OnAfterSetDocumentAttachmentFiltersForRecRefInternal(var DocumentAttachment: Record "Document Attachment"; RecRef: RecordRef; GetRelatedAttachments: Boolean)
begin
end;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 11936.


Provider install progress

Is your feature request related to a problem? Please describe.
daytona provider install might hang for a while and doesn't provide any feedback to the user.
Describe the solution you'd like
We should implement some kind of installation progress so the user knows something is going on.

/bounty $50

/attempt #734

/attempt #734
went through the issue , first thought was setting up a loading state to keep track of the installation and showing the user some kind of message or a loader.
would like to attempt and solve this issue

/attempt #734


Rules for the java-redis-client integration

I'm looking into this and found the Lettuce client very easy to provide rules for.  However, the Jedis client uses inheritance instead of composition so I am finding it more difficult to figure out the proper Byteman rules for it.
It would be simple if we could simply intercept the call to the constructor for Jedis and replace it with TracingJedis by using AT INVOKE Jedis.<init>, but this doesn't appear to help in the general case since we have to explicitly state the triggering class/method (no wildcards).

What about doing a PR to java-redis-client to change TracingJedis to be a wrapper? It could still support all of the current constructors for backward compatibility, but internally just wrap an instance of Jedis?
Agree about intercepting calls to the constructor - could have used that solution for the java-concurrent lib. Not sure if there is anything the ByteMan could do to help this situation? cc @adinn (when back from PTO)

I started down that path of refactoring the OT instrumentation for Jedis but then I realized that even if that wraps the Jedis client, the same problem exists -- intercepting the class construction and wrapping it without a generic way to intercept class instantiation in any possible triggering method.  For the Lettuce client it is simple because all instances have to get created through a builder, but Jedis doesn't have builders.

There was a feature request to instrument instance creation itself (and presumably not just the constructor function) a while back.  I've been studying the underlying ASM library to determine how feasible it is to do.

However, even if we could intercept and swap out/wrap the instance creation result, I think it would still have to be done in the context of a specific class+method, which precludes a generic rule for it without some kind of wildcard support on those fields.

Hi Ted/Ben,

I'll admit I don't really understand why ByteMan has this limitation. Is it intrinsic, or something the ByteMan project chooses as a design constraint?

However, even if we could intercept and swap out/wrap the instance creation result, I think it would still have to be done in the context of a specific class+method, which precludes a generic rule for it without some kind of wildcard support on those fields.

The proposal for an AFTER NEW|NEWARRAY|NEWMULTIARRAY location for rule injection does indeed provide an opportunity for rules to intercept object create operations in a way that differs from injection into a constructor. In particular, it would be possible for the rule to be passed the object created by the new operation and update it with a type-compatible alternative (i.e. a different object of the same type or subtype). However ...
Whatever might get implemented would still have to be limited to modifying bytecode of specific methods from specific classes at the point where a new operation occurs -- well, to be utterly precise, at the point where the new and associated constructor invocation have both completed (for reasons of type safety these two operations must remain indivisible). If you want a rule that injects code at every new operation for some given type without limiting the injection scope to a specific target method and class  then Byteman is not the tool for you. The only 'looseness' that Byteman allowsfor  in the definition of the rule target is via the use of interface rules, overriding rules and rules that omit a method signature and, hence, may apply to overloaded methods. That is indeed a long-standing policy decision.
The reason for this policy is not that it is impossible to implement, rather that online checking of every method of every loaded class for the presence of a potential injection point utterly kills performance. It's just not possible to make this innocuous. The same problem killed JBoss AOP's online transformation mode and has been seriously detrimental to use of many other injection-based tools/frameworks. If you need that type of 'wild-card' injection then there are two much better ways to do it:

Transform the source code or bytecode for the classes you wish to transform offline

Load and scan the classes you wish to transform in an auxiliary JVM or an auxiliary thread within the current JVM and generate per class rules to inject the side effects you want.

Neither of these is always easy to achieve but they both bypass the enormous performance hit that arises from having to scan every single loaded method.


Go: Clean up handling of import paths for test data. [phab:D1209]

Created by fromberger at 2017-01-03 18:13:02:
When assigning an import path to a package imported from a directory, check the
build context's source paths before falling back to the default.
Allow overriding the import path in go_indexpack rules.

shahms accepted this revision at 2017-01-03 18:16:29

fromberger closed via commit b7e926cfb6f3b1974582f658643af5d628056f81 at 2017-01-03 18:20:19


Installation instructions missing in the readme

The readme doesn't mention an npm package or give a CDN link; it just says to include <script src="web-animations.min.js"></script>, so I guess it means downloading the file from releases, which is a generally poor approach of managing dependencies. I had to look it up to find: https://www.npmjs.com/package/web-animations-js

The instructions are hidden here: https://github.com/web-animations/web-animations-js/blob/dev/docs/support.md#getting-the-polyfill


Hardware specification

describe the test hardware specification.
Now Intel CPUs have large-small (performance-efficient) cores.

Not planned yet.


Null pointer dereference in mrb_class

The following input demonstrates a crash, when supplied to mirb:
b = []
c = 0

class A < Fiber
end

x = A.new {a}.resume
y = A.new {a}.resume
m = A.new {}.resume

d = b.to_s
Array.new {}

begin
    break
    n = A.new {}
    z = A.new {}
    ensure
    z.resume
end

ASAN report:
==71334==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x0001064e481a bp 0x7fff59864130 sp 0x7fff59863fc0 T0)
==71334==The signal is caused by a READ memory access.
==71334==Hint: address points to the zero page.
    #0 0x1064e4819 in mrb_class class.h:50
    #1 0x1064f79b3 in mrb_vm_exec vm.c:1269
    #2 0x1064eda5f in mrb_vm_run vm.c:860
    #3 0x1064e59ae in mrb_run vm.c:2738
    #4 0x10651eca3 in ecall vm.c:326
    #5 0x106503036 in mrb_vm_exec vm.c:1789
    #6 0x1064eda5f in mrb_vm_run vm.c:860
    #7 0x10638b6f9 in main mirb.c:549
    #8 0x7fff8c15e234 in start (libdyld.dylib:x86_64+0x5234)

==71334==Register values:
rax = 0x00007fff59863fe0  rbx = 0x00007fff59864000  rcx = 0x7b206164626d6174  rdx = 0x0f641c2c8c4dac2e
rdi = 0x00001fffeb30c7fd  rsi = 0x0000100000000000  rbp = 0x00007fff59864130  rsp = 0x00007fff59863fc0
 r8 = 0x0000100000000000   r9 = 0x73dc72c87c940000  r10 = 0xf2f20000f2f20000  r11 = 0x00001e2f5984dde0
r12 = 0xf2f20000f1f1f1f1  r13 = 0x00001fffeb30d63c  r14 = 0xf2f20000f2f2f2f2  r15 = 0xf2f2f2f2f2040000
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV class.h:50 in mrb_class
==71334==ABORTING
Abort trap: 6

This issue was reported by https://hackerone.com/ahihi

I think 34dd258c fixed this too.


Grading: Assessment briefing not accessible while grading

The grading workspace should have the Assessment Briefing so that the grader can read the briefing while grading. The Briefing should be accessible using the briefcase icon, like in the student view of the assessment.

To have the mission briefing for the grading workspace, it is essential to access the Long summary of the assessment in the GradingWorkspace.tsx on the frontend side. This can be achieved by extracting the data using the submission_id. Consequently, modifications are required in the admin_grading_view.ex on the backend side. The long summary of the assessment should be incorporated into this file to ensure its content is available within each submission for every student.


[DRAFT PR - NOT FOR REVIEW] Revert PR 1827, use ebpf_store API to set/get sock_addr context

Description
Describe the purpose of and changes within this Pull Request.
Testing
Do any existing tests cover this change? Are new tests needed?
Documentation
Is there any documentation impact for this change?

This option has been tabled for now.


Update index.html

commit section will have someone take a look at the code.

Looks good, I'm going to merge this.


Jdtls Server exits with code 13

Description
I cannot get the jdtls language server to work. I have a build.gradle file which activates the root directory, and my build works correctly and everything, but the language server just doesn’t attach and it shows the error:
Client n quit with exit code 13 and signal
This might be a local problem, and if it does anyone have any suggestions on how to fix it?

edit:
someone else also has this issue here: https://solveforum.com/forums/threads/problems-when-trying-to-install-jdtls-lsp-for-neovim.1333866/
Neovim version
NVIM v0.8.0-dev+528-g3b1423bfa
Build type: Release
LuaJIT 2.1.0-beta3
Nvim-lspconfig version
No response
Operating system and version
macos 12.5.1
Affected language servers
jdtls
Steps to reproduce

install nvim-lspconfig using any package manager
install jdtls using nvim-lsp-installer (what I did) or manually
have require('lspconfig').jdtls.setup() in vim config.
create a new directory and cd there
create a java file in the directory
activate the root directory requirements with a build file (ex. gradle init)

Actual behavior
No clients attach to the buffer, and Client n quit with exit code 13 and signal
Expected behavior
The language server should attach.
Minimal config
local on_windows = vim.loop.os_uname().version:match 'Windows'

local function join_paths(...)
  local path_sep = on_windows and '\\' or '/'
  local result = table.concat({ ... }, path_sep)
  return result
end

vim.cmd [[set runtimepath=$VIMRUNTIME]]

local temp_dir = vim.loop.os_getenv 'TEMP' or '/tmp'

vim.cmd('set packpath=' .. join_paths(temp_dir, 'nvim', 'site'))

local package_root = join_paths(temp_dir, 'nvim', 'site', 'pack')
local install_path = join_paths(package_root, 'packer', 'start', 'packer.nvim')
local compile_path = join_paths(install_path, 'plugin', 'packer_compiled.lua')

local function load_plugins()
  require('packer').startup {
    {
      'wbthomason/packer.nvim',
      'neovim/nvim-lspconfig',
    },
    config = {
      package_root = package_root,
      compile_path = compile_path,
    },
  }
end

local load_config = function()
  vim.lsp.set_log_level 'trace'
  require('vim.lsp.log').set_format_func(vim.inspect)
  local nvim_lsp = require 'lspconfig'
  local on_attach = function(_, bufnr)
    local function buf_set_option(...)
      vim.api.nvim_buf_set_option(bufnr, ...)
    end

    buf_set_option('omnifunc', 'v:lua.vim.lsp.omnifunc')

    -- Mappings.
    local opts = { buffer = bufnr, noremap = true, silent = true }
    vim.keymap.set('n', 'gD', vim.lsp.buf.declaration, opts)
    vim.keymap.set('n', 'gd', vim.lsp.buf.definition, opts)
    vim.keymap.set('n', 'K', vim.lsp.buf.hover, opts)
    vim.keymap.set('n', 'gi', vim.lsp.buf.implementation, opts)
    vim.keymap.set('n', '<C-k>', vim.lsp.buf.signature_help, opts)
    vim.keymap.set('n', '<space>wa', vim.lsp.buf.add_workspace_folder, opts)
    vim.keymap.set('n', '<space>wr', vim.lsp.buf.remove_workspace_folder, opts)
    vim.keymap.set('n', '<space>wl', function()
      print(vim.inspect(vim.lsp.buf.list_workspace_folders()))
    end, opts)
    vim.keymap.set('n', '<space>D', vim.lsp.buf.type_definition, opts)
    vim.keymap.set('n', '<space>rn', vim.lsp.buf.rename, opts)
    vim.keymap.set('n', 'gr', vim.lsp.buf.references, opts)
    vim.keymap.set('n', '<space>e', vim.diagnostic.open_float, opts)
    vim.keymap.set('n', '[d', vim.diagnostic.goto_prev, opts)
    vim.keymap.set('n', ']d', vim.diagnostic.goto_next, opts)
    vim.keymap.set('n', '<space>q', vim.diagnostic.setloclist, opts)
  end

  -- Add the server that troubles you here
  local name = 'jdtls'
  nvim_lsp[name].setup {
    on_attach = on_attach,
  }

  print [[You can find your log at $HOME/.cache/nvim/lsp.log. Please paste in a github issue under a details tag as described in the issue template.]]
end

if vim.fn.isdirectory(install_path) == 0 then
  vim.fn.system { 'git', 'clone', 'https://github.com/wbthomason/packer.nvim', install_path }
  load_plugins()
  require('packer').sync()
  local packer_group = vim.api.nvim_create_augroup('Packer', { clear = true })
  vim.api.nvim_create_autocmd(
    'User',
    { pattern = 'PackerComplete', callback = load_config, group = packer_group, once = true }
  )
else
  load_plugins()
  require('packer').sync()
  load_config()
end

LSP log
https://gist.github.com/williamhCode/4c974065e13a309ab8b2766266a8c976

@williamhCode   some useful information. should problem with your java env
https://www.reddit.com/r/neovim/comments/u2vlmb/client_1_quit_with_exit_code_1_and_signal_0_jdtls/

Thanks a lot. i literally just uninstalled, restarted, then installed the JDK and it worked!

glad hear that.

This issue has been mentioned on Neovim Discourse. There might be relevant details there:
https://neovim.discourse.group/t/lspconfig-jdtls-server-not-attaching/3086/2

Hi there, I went to lvim discord and was advised to  delete ./.cache/jdtls. It worked for me.

I actually had a weird error where it would say something along the lines of "table nil error: provided begin function without end" or something like that. I'd have to close the message and it would not work like 10% of the time. I just switched to nvim-jdtls and that works extremely well. U can even clear the jdtls workspace from nvim. Highly recommend even if u just need basic functionality.

U can even clear the jdtls workspace from nvim

you also can do this without any plugin.  you can get the workspace from jtdls then remove the folder

U can even clear the jdtls workspace from nvim

you also can do this without any plugin.  you can get the workspace from jtdls then remove the folder

Yeah, but its just one command so saves some time. But anyways, I just finished my Java OOP class (exam is tmr), so I'll probably never touch java again lmao.
But anyways here's a video of the error showing up. It pops up real fast before the server starts working.
https://user-images.githubusercontent.com/83525937/207293128-662f132b-27c8-400d-8f32-697f415c2ec1.mov

honestly I have some distaste for java so I will not build any java environment. So there's nothing I can do sorry. Welcome to PR


Patch 1

Related Issue(s)

Description

Checklist

[] I have read the policy contribution guidelines.
[] I have added test manifests and resources covering both positive and negative tests that prove this policy works as intended.
[] I have added the artifacthub-pkg.yml file and have verified it is complete and correct.

Already fixed in #793


Fix excessive memory allocation described in #4

As described in #4 the hash computation allocates far to much temporary memory resulting in multiple GC runs per call. Especially f2d8b33 is critical to make the implementation usable in a multi-user environment.

This is probably a tough PR to get merged seeing as this also includes changing from Maven to Gradle as well as many unrelated changes (rephrasing stuff in the readme, ...).

This is probably a tough PR ... also includes changing from Maven to Gradle as well as many unrelated changes (rephrasing stuff in the readme, ...).

We (@dbusche and I) started evaluating, benchmarking and finally fixing the code from the latest version committed by @minebreaker, since it looked like a straight forward advancement (including hash encoding and cleanup). Don't know, why he did not post a pull-request.
Yes, the readme was adjusted, since @minebreaker moved to Java 8, removed unneccessary dependencies for the command-line version (which I think is really not required for a crypto lib), and finally added this really usefull hash-encoding stuff. Therefore I consolidated the documentation reflecting the latest state.
Yes, these changes are "unrelated" to the memory issue but If we started from the original @andreas1327250 code, the really useful @minebreaker stuff would have probably become unmergable.


Node Cassandra driver just hangs on execute()

var cql = require('node-cassandra-cql');
var client = new cql.Client({hosts: ['*.*.*.*'], keyspace: '*',
                             username:'*', password: '*'});

console.log('connected to ' , client);
console.log('Querying....');

client.execute('select * from example where field1=?', [1],
  function(err, result) {
    console.log('inside', result);

    if (err)
        console.log('execute failed',err);
    else
        console.log('got chat ' + result.rows[0].field1);

    client.shutdown();
  }
);

I am using this code, the execute() callbacks aren't getting called . To test I used an incorrect IP address, it immediately responds and this line console.log('execute failed',err) logs what is below.
execute failed { [PoolConnectionError]
  name: 'PoolConnectionError',
  info: 'Represents a error while trying to connect the pool, all the connections failed.',
  individualErrors:
   [ { Error: getaddrinfo ENOTFOUND
         at errnoException (dns.js:28:10)
         at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:79:26)
       code: 'ENOTFOUND',
       errno: 'ENOTFOUND',
       syscall: 'getaddrinfo',
       hostname: '*.*.*.*'
       host: '*.*.*.*'
       port: 9042 } ] }

If with right IP address nothing happens may because new cql.Client internally calls connect (asynchronously) before even connection is made execute is attempted ? All perfectly works in CQLSH, my servers are in AWS US west coast.
Any inputs welcome.

You are using the legacy Cassandra driver node-cassandra-cql, as the project readme states, it is no longer maintained:

node-cassandra-cql has graduated from community driver to being the foundation of the official Datastax Node.js Driver for Apache Cassandra.
There will be no more development in this repository. I encourage everyone to start migrating to the new driver as soon as you can, it's got some great new features that you should try out, along with an improved cql to javascript type mapping for prepared statements.

Use DataStax Node.js driver instead: https://github.com/datastax/nodejs-driver


Only check live EventPages for archiving; append date to duplicate slugs

Drafts that got archived were causing errors when, later, pages with the same slug were attempted to be archived again. This update only checks for live pages to archive (and only EventPages now), and if it finds a duplicate slug, it will append the event's start date to the slug before moving it.
How to test this PR

Refresh your DB with last night's dump
Pull branch
Run ./cfgov/manage.py runscript archive_events
Verify that it returns No past events to be archived were found.
Publish the draft current events with dates in the past
Run ./cfgov/manage.py runscript archive_events again
Verify that it moves them to the archive, with their start date appended to the slug

Notes and todos

[ ] Convert to management command
[ ] Add unit tests

Checklist

[ ] PR has an informative and human-readable title

PR titles are used to generate the change log in releases; good ones make that easier to scan.
Consider prefixing, e.g., "Mega Menu: fix layout bug", or "Docs: Update Docker installation instructions".

[ ] Changes are limited to a single goal (no scope creep)
[ ] Code follows the standards laid out in the CFPB development guidelines
[ ] Future todos are captured in comments and/or tickets
[ ] Project documentation has been updated, potentially one or more of:

This repo’s docs (edit the files in the /docs folder) – for basic, close-to-the-code docs on working with this repo
CFGOV/platform wiki on GHE – for internal CFPB developer guidance
CFPB/hubcap wiki on GHE – for internal CFPB design and content guidance

I think management commands are best as wrappers for tested scripts. So rather than repeat the script code here, could we call the script's run() function?

Yeah, I'm gonna end up with just the management command. I just left the existing script in place for the time being to make it easier to see the changes I was making to the logic of it. I'll pull it out before merging.

Otherwise, the script works as advertised.


support to numpy.interp.

We'd like to be able to backtrace the gradient of interp for implementing a toy model. Is anybody working on this?
If not we will file a PR. (cc: @energy)

No one is working on it as far as I know, but a PR would be great!

Is it a problem if I pull in a dependency on scipy.sparse? interp is a convolution with a linear kernel -- the easiest (cleanest) implementation is to compute the factors and store them as a sparse matrix, then transpose and multiply.

Any updates on this?
If not, I'm searching for an efficient autograd-compatible GPU implementation of interpolation. Any idea where to look? I implemented my own in PyTorch, but training is prohibitively slow.

Any updates on this?
If not, I'm searching for an efficient autograd-compatible GPU implementation of interpolation. Any idea where to look? I implemented my own in PyTorch, but training is prohibitively slow.

Have you improved something about interpolation autograd-compatible?


Add --first-branch option

Or something to that effect...
Running codeslave --branch '(-stable$|^master$)' returns all stable branches, but often we want to commit only to the earliest branch in this list. We should take the returned list, display them all in the output, but highlight the one that will be affected by the script when this option is passed. The list should be sorted and the first chosen.

Instead of having a different argument, I think this is just how --branch should work in general. Assuming that all changes from the earlier branches will be merged into the later branches, including master, we could instruct users to do: --branch '(-stable$)' to find the earliest stable branch that should be committed to. If you wanted the "first stable branch" sorted alphanumerically, and you included "master", you'd always get "master" because "m" comes before "v" in alphabet.

When talking about Workarea plugins, specifically, we might be releasing from a master branch for a given plugin. That's why --branch '(-stable$|^master$)' is necessary as a branch option.
I'm thinking:

We always plop master at the end of the derived branch list
--first-branch outputs the full list of branches discovered, but highlights the first branch indicating that that one will be operated upon.


Wien2k: core-hole parsing

cc @ondracka

@jfrodlova sorry to hijack your PR. We recently published this repo on Zenodo. If you would like to be included as an author, please send me your full name and orcid (if you have one).
@ndaelman-hu - I guess you will see this, but just in case

@jfrodlova sorry to hijack your PR. We recently published this repo on Zenodo. If you would like to be included as an author, please send me your full name and orcid (if you have one).

Hi @JFRudzinski, thanks for reaching out.
Here is a link to my ORCID page https://orcid.org/0000-0002-3113-9641 (Jitka Frodlová)

@ndaelman-hu Any chance for a review? I've already done some review locally before @jfrodlova submitted it here, but your feedback would be appreciated since you implemented the CoreHole in other parsers.

Thanks a lot @ndaelman-hu  for the review, much appreciated.

Many thanks for the code review @ndaelman-hu

Thanks for your remarks, @ndaelman-hu!
I checked the logger - the whole sentence is printed.

Thanks for your remarks, @ndaelman-hu! I checked the logger - the whole sentence is printed.

That's awesome! I learned something new today :) Thx.


How do I open a memberOf？

image version osixia/openldap:1.5.0,I want to open memberOf with variables.I want to learn. Thank you

You have to enable LDAP_RFC2307BIS_SCHEMA=true


dd fails on tests/dd/not-rewound.sh

With the test case, it show b - with gnu, it shows bde:
export C=./target/debug/coreutils
echo abcde > in
echo "bde" > ref
($C dd skip=1 count=1 bs=1; $C dd skip=1 bs=1) < in > out 2> /dev/null
diff -u ref out

comment the first line to come back on GNU
To run it:
bash util/run-gnu-test.sh tests/dd/not-rewound.sh

@jfinkels rings a bell ?

Yes! see https://github.com/uutils/coreutils/issues/3008#issuecomment-1028437526

oh, thanks, it is a dup then :)

Reopening as this seems to be broken still:
On GNU v8.30:
$ (dd skip=1 count=1 bs=1; dd skip=1 bs=1) < in > out 2> /dev/null && cat out
bde

On uutils dd (main branch, commit 0a385ee7a5297707045ad2f9c83f0b11df290e41)
$ (./target/release/dd skip=1 count=1 bs=1; ./target/release/dd skip=1 bs=1) < in > out 2> /dev/null && cat out
de

Another clue here is that this works as expected if stdout is not redirected to a file:
$ (./target/release/dd skip=1 count=1 bs=1; ./target/release/dd skip=1 bs=1) < in 2> /dev/null
bde


Hyperlink to v4 release notes results into a 404 error

In the migration guide for the v4 beta, there is a hyperlink for the release notes.

Clicking on this hyperlink, results into getting directed to the 404 Github error page.

This is the URL from the hyperlink: https://github.com/cocopon/tweakpane/releases/tag/4.0.0

Fixed.


[XLA] Fix std::array initialization take 2

#15511 did not fix the issue on MSVC.
The actual root cause is due to the presence of ( and ) around const C array. For std::array<int, 2> a ({1, 2, 3}), MSVC seems to see ({1, 2, 3}) as a pointer, which triggers C2100 compile error. Removing ( and ) fixes the issue.
#15213

Can one of the admins verify this patch?

Jenkins, test this please.


Reserved word 'char' breaks minification

I think that the use of 'char' as an identifier (in the getPathCharType function) is causing yuimin and other minifiers to break because it is reserved for a possible future keyword by the ECMAScript specification.
Swapping it out with 'character' fixes the issue for me.

I'll change the variable name, but I don't really see why this would break things. Maybe you should update your minifier and use UglifyJS instead ;)


Add support for Presentation State reports

Description of the new feature/enhancement
There are two reports that can be queried with the DECRQPSR (request presentation state) escape sequence. The one returns the active tab stops, and the other returns what they call "cursor information", which includes things like cursor position, SGR attributes, and character set designations.
There is also then a corresponding DECRSPS sequence (restore presentation state) which can restore the saved information from the above mentioned reports.
There are a number of uses for these sequences, but the main reason I want to support them is because they're needed to meet the requirements of the TSI (terminal state interrogation) feature, which is itself a requirement for level 3 compliance.
Proposed technical implementation details (optional)
This will require some additional work in the TerminalOuput class to track the actual IDs of the designated character sets so we can return that information in the report. I think most of the other information we need should be easy to access.

@al20878 I've got another couple of tests which I'd previously created for these reports. They were written as bash scripts, though, so just let me know if that's a problem and I can rewrite them in Python.
Also, note that they don't actually log anything, the results are displayed on the screen, so I'd be grateful if you could take a photograph of each run and post that here.
This is for the tab stop report:
https://gist.github.com/j4james/a25e6290772353c06f85c584007c2aa0
And this one is for the cursor information report:
https://gist.github.com/j4james/570a9b7dd28dbdd8b626be2770567210

I forgot to say, I'd ideally like to see this run on a VT525 - specifically the cursor information report - because I think it might include additional information for the active SGR colors that other terminals wouldn't include.

Shell scripts are fine, I'll be running them on Linux (Raspbian), so it's all good, I hope.  Downloaded both, will post the results.

@al20878 I've uploaded a revision of the dectabsr script with a one second delay after each column mode change, and a longer timeout on the report reader. Hopefully that should work better. When you have a chance, I'd appreciate you trying it again.
https://gist.github.com/j4james/a25e6290772353c06f85c584007c2aa0

Indeed, the new version runs absolutely clean.  The result:

@al20878 Perfect. That tells me everything I needed to know. Thank you so much for doing all this testing. This is going to keep me busy for a while, so I'll leave you in peace for a bit. 😉

No problem, @j4james!  I was happy to help :-)
Please feel free to reach out if you need any further testing.  And good luck!