Improve creation of aggregation areas

Initial tasks:

[x] #211
[ ] #212

We should also discuss a feature to pull grids from opportunity datasets that have already been uploaded, which, along with some front-end changes, could address https://github.com/conveyal/analysis-ui/issues/509

See also https://github.com/conveyal/analysis-ui/issues/473

Codecov Report

Merging #213 into dev will decrease coverage by 0.02%.
The diff coverage is 0%.

@@             Coverage Diff             @@
##               dev     #213      +/-   ##
===========================================
- Coverage     23.2%   23.17%   -0.03%
+ Complexity     104      103       -1
===========================================
  Files           62       62
  Lines         2422     2386      -36
  Branches       220      215       -5
===========================================
- Hits           562      553       -9
+ Misses        1825     1801      -24
+ Partials        35       32       -3

Impacted Files
Coverage Δ
Complexity Δ

...al/taui/controllers/AggregationAreaController.java
12.5% <0%> (-0.18%)
2 <0> (ø)

...n/java/com/conveyal/taui/AnalysisServerConfig.java
77.55% <0%> (-0.45%)
3% <0%> (ø)

...java/com/conveyal/taui/models/AnalysisRequest.java
0% <0%> (ø)
0% <0%> (ø)
:arrow_down:

...rc/main/java/com/conveyal/taui/AnalysisServer.java
46% <0%> (+0.12%)
5% <0%> (ø)
:arrow_down:

...l/taui/controllers/RegionalAnalysisController.java
10.07% <0%> (+0.86%)
2% <0%> (ø)
:arrow_down:

...om/conveyal/taui/controllers/BrokerController.java
20.75% <0%> (+1.11%)
6% <0%> (ø)
:arrow_down:

.../java/com/conveyal/taui/analysis/LocalCluster.java
80% <0%> (+3.07%)
2% <0%> (-1%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 23befd1...3412d38. Read the comment docs.

Related front-end change: add checkbox "Use union of shapefile features" to the create aggregation area form, and append ('union', union) or similar to the uploadAggregationArea action form data.

@trevorgerhardt if you have a minute in the coming week, could you add a checkbox input to "Combine to single aggregation area," which would set the union parameter in this PR (default to false)


full message deserialization 🪄

WIP. This pulls from @kafonek's kernel sidecar, likely to replace our current messaging.

💯 all yours if the code helps you out. I wish I had it in a cleaner state for you to lift and shift. Knowing what I know now, I'd at the least have impl'd try_into instead of into for a lot of those to bubble up error handling instead of .unwrap() or .expect() it.

I'm running into a bit of an issue with types within our daemon needing a common message type rather than separate request and response. I'm going to try out putting Message together as a trait.


What about adding it on Chocolatey?

What do you think about adding it on https://chocolatey.org so we can install it using the below command?
choco install makers

I would like to use it also on Windows PCs without rust and cargo.

good idea. will have to check out how....


Docs search is broken

Describe the bug

This error shows up in the console:
{name: 'ApiError', message: 'Invalid Application-ID or API key', status: 403, transporterStackTrace: Array(1)}
message: "Invalid Application-ID or API key"
name: "ApiError"
status: 403

Information:

OS: Windows 11
Browser: Chrome

Thanks for raising this. I had to upgrade the documentation packages which took some time!


added Bidirectional Conditional Insertion Sort

This is a Bidirectional Conditional Sort.
Note on stability with custom compare function.
let comp = new Comparator(function (a, b) {
    let x = a;
    let y = b;
    if (typeof a === "string") {
        x = a.length;
    }
    if (typeof b === "string") {
        y = b.length;
    }
    if (x === y) {
        return 0;
    }
    return x < y ? -1 : 1;
})

i.e.
Without
a, cc, bbb --> a, bbb, cc
With
a, cc, bbb --> a, cc, bbb
Even with the custom compare, you are not always guaranteed a stable sort
Example: Given: [aa, q, a, bbbb, ccc]
The algorithm will start with a SWAP from the right most index and the middle of the array.

START SWAP "ccc", "a"
END SWAP (New Array: aa,q,ccc,bbbb,a)
"aa" is greater than "a"
START SWAP "aa", "a"
END SWAP a,q,ccc,bbbb,aa
"ccc" is greater than "aa"
START INSRIGHT  (Given: a,q,bbbb,bbbb,aa)
END INSRIGHT (New Array: a,q,bbbb,aa,ccc)
"bbbb"is greater than "aa"
START INSRIGHT (Given: a,q,bbbb,aa,ccc)
END INSRIGHT  (New Array: a,q,aa,ccc,bbbb)

Finished:
[ 'a', 'q', 'aa', 'ccc', 'bbbb' ]

Codecov Report

Merging #67 into master will decrease coverage by 0.34%.
The diff coverage is 91.13%.

@@            Coverage Diff             @@
##           master      #67      +/-   ##
==========================================
- Coverage     100%   99.65%   -0.35%
==========================================
  Files          89       90       +1
  Lines        1927     2006      +79
  Branches      341      356      +15
==========================================
+ Hits         1927     1999      +72
- Misses          0        5       +5
- Partials        0        2       +2

Impacted Files
Coverage Δ

...g/bidirectional-conditional-insertion-sort/BCIS.js
91.13% <91.13%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d154015...afc0888. Read the comment docs.

Going to make a new one.


feat: use Promise.resolve to start promise chain.

Safe start a promise chain, simply use a literal value.

pTry(() => 0).then(
	val => console.log(++val)
);
// Or
pTry(0).then(
	val => console.log(++val)
);

Note:

WIP xxx.d.ts file.
If this pr might merge, it would upgrate the main version.

I don't really see the point of this (?). If you just want to start it with a literal value, you can just use Promise.resolve():
Promise.resolve(0).then(
	val => console.log(++val)
);

So it is, this just a wrapper that use one API.

I'm unfortunately going to pass on this. I would prefer to stay compatible with https://github.com/tc39/proposal-promise-try.


Move Envoy admin interface from TCP:15000 to a Unix Domain Socket

Bug description
In each sidecar, the Envoy proxy in the istio-proxy container is currently configured to expose its admin interface on a TCP socket bound to the loopback interface (by default, that port is 15000). That socket is accessible by any container within the pod, since they are in the same network namespace. Therefore, a malicious or compromised container may perform any admin operation on Envoy. Especially, it can force Envoy to quit, and therefore to continuously reconnect to Pilot, which can induce some load on Pilot, which in turn can affect the rest of the cluster.
Expected behavior
The admin interface should not be accessible outside of the istio-proxy container. The admin interface should be exposed as a Unix Domain Socket, local only to that container.
Tasks

[ ] Modify Envoy's bootstrap file template to:

[ ] Always expose the admin interface as a Unix Domain Socket.
[ ] Optionally also expose the admin interface as a TCP socket, depending on a template parameter.

[ ] Modify the pilot-agent to accept an additional boolean flag or environment variable to set the template parameter.
[ ] Define and document a new boolean pod annotation to enable the admin TCP interface.
[ ] Modify the injection template for both Helm and Operator to set that flag/variable depending on the pod's annotation. Default to true for Istio 1.5.
[ ] Modify all dependencies on TCP port 15000 in the Istio code, esp. in the istio-agent request command.
[ ] Fix all remaining documentation pages that recommends to perform curl commands to port 15000, and replace them with calls to istio-agent request commands.
[ ] Document the admin interface TCP socket as deprecated in 1.5, to be disabled by default in 1.6.

cc @howardjohn

This will probably break users, likely @Stono being one of them

I see. I suppose this is used by tricks to make Jobs work, or something like that?
Then I'd propose to make the move to UDS anyway, and support optionally exposing via TCP in addition to UDS if explicitly enabled by customer. How does that sound? @Stono?

Yeah, the quitting Envoy hacks are mostly solved by pilot agent exposing the quit endpoint. Not sure if there are other cases (that we should support)

Right, apart from the quitting Envoy, I don't see what features from the admin interface would be useful to an application container.

There are tools like scuttle and envoy-preflight that might, I forget if
they do that or make you share process namespace and look at the process
directly
On Wed, Dec 18, 2019, 6:19 PM Romain Lenglet<EMAIL_ADDRESS>wrote:

Right, apart from the quitting Envoy, I don't see what features from the
admin interface would be useful to an application container.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/19684?email_source=notifications&email_token=AAEYGXLNHCPYY72XWGNHSFTQZLKZZA5CNFSM4J4VLENKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHIFDEA#issuecomment-567300496,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEYGXLPFZRU4AHKUOXZQSLQZLKZZANCNFSM4J4VLENA
.

Adding links to your references:

scuttle
envoy-preflight

Both have the exact same description: "A wrapper for applications to help with running envoy as a sidecar."

It seems like both scuttle and envoy-preflight support using a UDS instead of the HTTP interface.
We could then mount the UDS in a volume shared among the containers within the same pod. I don't know how that could work with automatic injection though, because the application container would have to refer to a volume that is injected.

Hi this is correct, we use the http interface on pilot-agent to terminate the sidecar after the job container completes, currently we have 57 jobs deployed in this manner.
A better (less change inducing model for me) would be to have the http interface enabled by an annotation (so off by default), that way the jobs would continue to work as i could just annotate them with sidecar.istio.io/httpAdminEndpoint: "true".
When it comes to envoy-preflight, I think you're maybe being ambitious asking everyone using that on their day to day workloads to expose a shared volume between the sidecar and the application to facilitate a UDS interface.  I suppose that would be a trade off, between security and complexity.  In that scenario too; I'd suggest making it a decision of the operator with something like an annotation.

We can insert an iptables rule to reject connection from the user whose UID/GID is not 1337, right?

We can insert an iptables rule to reject connection from the user whose UID/GID is not 1337, right?

That wouldn't work. The hack to make Jobs work is to access the admin interface from within the application's container when it terminates. And application containers should not run with UID 1337, since they would bypass the proxy.

A better (less change inducing model for me) would be to have the http interface enabled by an annotation (so off by default), that way the jobs would continue to work as i could just annotate them with sidecar.istio.io/httpAdminEndpoint: "true".

Agreed, that would be a good compromise. Anyway you need to modify those pods to make them work with the Istio sidecar.

I mean in proxy-init container we can insert iptables rule to reject the connection to port 15000 from the process whose UID/GID is not 1337, so application container will be rejected to access 15000 port(since application containers should not run with UID 1337)

A better (less change inducing model for me) would be to have the http interface enabled by an annotation (so off by default), that way the jobs would continue to work as i could just annotate them with sidecar.istio.io/httpAdminEndpoint: "true".

Agreed, that would be a good compromise. Anyway you need to modify those pods to make them work with the Istio sidecar.

We do which makes this even more difficult to roll out actually.  Application deployments aren’t injected, pods are, which means they’re disconnected.  Therefore updating istio to a newer version will break live deployed services until they’re redeployed.
For me, this is a breaking change, the way to correctly roll it out would be

Add the annotation but default to enabled
Give deprecation notice
In future release change annotation default to false

I mean in proxy-init container we can insert iptables rule to reject the connection to port 15000 from the process whose UID/GID is not 1337, so application container will be rejected to access 15000 port(since application containers should not run with UID 1337)

This wouldn't solve the problem. The hack to make Jobs work currently requires application containers (NOT running with UID 1337) to access port 15000 to kill Envoy when the job is complete. Blocking access to UIDs other than 1337 would break that hack.

There may be a way we can detect that a pod is derived from a Job? In which case we could enable TCP only for that case?
istioctl kube-inject could easily distinguish Jobs from other resources in https://github.com/istio/istio/blob/7a4fb65717fe578a86472beb860b113c364c2cff/pkg/kube/inject/inject.go#L744.
The automatic injector could also detect that a Pod is running a Job: the Pod's metadata.ownerReferences.kind has value Job` in this case.

It’s not just jobs... honestly, make it configurable and let the operator decide

@Stono
I don't see any legitimate case other than Jobs?!
And to clarify my previous comment: I propose to make the default setting to enabling TCP to true only for Jobs, to make the transition easier for you. We'd still introduce the annotation to make this configurable.

I think I agree with you, I am just cautious that istio is in the wild, and you don’t know for sure how people are using the admin interface, without asking everyone.
So it seems excessive and potentially breaking to try and add Job inference into the injector when you can simply make it an annotation/toggle and leave it to the operator to decide what they want to do, then deprecate it in a controlled fashion.
Just my two cents 🤘

Agree with Karl, the less special cases (eg Jobs are different) the better
On Mon, Dec 23, 2019 at 11:27 AM Karl Stoney<EMAIL_ADDRESS>wrote:

I think I agree with you, I am just cautious that istio is in the wild,
and you don’t know for sure how people are using the admin interface,
without asking everyone.
So it seems excessive and potentially breaking to try and add Job
inference into the injector when you can simply make it an
annotation/toggle and leave it to the operator to decide what they want to
do, then deprecate it in a controlled fashion.
Just my two cents 🤘
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/19684?email_source=notifications&email_token=AAEYGXL4FXWNNQWC245ITWTQ2EGJTA5CNFSM4J4VLENKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHRY2EA#issuecomment-568560912,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEYGXJUZBANF7GCDCLKATLQ2EGJTANCNFSM4J4VLENA
.

OK, sounds good. I updated the tasks in this PR's description. @Stono, @howardjohn please confirm that this works for you.

I found an example of where we use the admin interface to stop the sidecar, not in a Job.  Airflow: we run Airflow Scheduler on K8s which creates Pods, not Jobs (which get injected with sidecars) - which run to completion then stop the sidecar.
❯ k -n data-platform-airflow get pods
NAME                                                                                READY   STATUS    RESTARTS   AGE
airflow-prometheus-exporter-559d667576-vn89w                                                           1/1     Running   0          21d
airflow-scheduler-58867cd7d8-2cb7k                                                                     1/1     Running   0          3d22h
airflow-web-cfdd75746-5mmt2                                                                            2/2     Running   0          3d22h
customerperformanceexportdataflowstockversionperformanceexportdaily-27bb19df62c545e0928f516be132f620   2/2     Running   0          15m
dataflow-stock-version-performance-export-daily-286630d7                                               2/2     Running   0          15m
transienttorawrawadverttextstockitemdatalaketransienttoraw-b1b58e6af8b045c3af25b067bcb956fd            2/2     Running   0          29m
transienttorawstockitemstrategyupdatedatalaketransienttoraw-a5cc8b7197fc492a9236c07ff18eeabc           2/2     Running   0          6h59m
transienttorawvehiclemetricsdatalaketransienttoraw-869326a9dee44d00bd74268cc9a0302c                    2/2    Running   0          29m

Instead of optionally enabling exposing the admin interface via TCP, how about optionally enabling exposing the UDS in an emptyDir volume? The UDS in that volume would be visible from any container in the pod.
That solves the problem of exposing this as a TCP service while still making the admin interface accessible from the rest of the pod (optionally).
@Stono would that work for you?

No it doesn’t as I really don’t have the engineering capacity right now to change everywhere we are doing this to use UDS instead.
I really don’t know why you can’t leave this tcp for those who want it?

No it doesn’t, as I really don’t have the engineering capacity right now to change everywhere we are doing this to use UDS instead.
I really don’t know why you can’t leave this tcp for those who want it and understand any associated risks?

Fine. I was exploring options.

Yeah you’d already mentioned uds earlier though so aren’t we going round in circles a bit?
Ultimately I’m just one user, so do what’s best for the product, not for me.

@abhide do you plan on doing this?  If not please let us know so we can find another owner

OK, I'm timing out on @abhide and finding a new owner.
Circling back on the result of a security related chat, the compromise we reached was: 1) continue listening on TCP loopback by default, 2) add an option to listen on UDS instead, 3) include advice to enable the UDS option in an upcoming 'how to harden your Istio cluster' doc (and sprinkle the same recommendation in the relevant existing docs).
@yxue can you pick this one up?  This has the urgency of an important feature request which means stability affecting bugs come first.  Thanks

I would suggest a slightly different plan:

Always enable listening on UDS. There's no harm in doing that.
Modify pilot-agent to always use that UDS.
Add an option to also listen on TCP loopback.

We cannot listen on both TCP loopback and UDS. The protobuf definition of admin address is here
https://github.com/envoyproxy/envoy/blob/4300ad4544b500982d39fd0ddf83f38b40b80828/api/envoy/config/core/v3/address.proto#L121-L132

Argh. I hadn't checked whether config.bootstrap.v2.Admin could take more than one config.bootstrap.v2.Admin proto. Changing this would be quite invasive. 😞

OK, then the solution is to just use UDS in the bootstrap, and have pilot-agent optionally open a TCP port and forward connections to the Envoy UDS.

That sounds good to me. I am going to follow the plan:

continue listening on TCP loopback by default
add an option to listen on UDS, default to TCP loopback
switch default to UDS and have pilot-agent optionally open a TCP port
document how to disable listening on TCP to harden the cluster

There are a plethora of debugging tools that are used to accessign the istio proxy admin endpoint on 15000. And most of our end users are also familiar with using the 15000 endpoint to get the dump via curl. If you change this to some UDS, then it creates a lot of pain for porting all the existing tools.
Secondly, I do not buy this quit envoy and induce load on pilot thing. There are several ways to induce load on pilot starting by simply hitting the Pilot service directly from Envoy. There is no end to DDOSing the system if we keep locking down all the debug endpoints, tools, etc. Also, the same malicious container or pod user could also mount the uds (as part of pod spec) and then overload envoy with it. I could overload enovy by simply calling Envoy on any port using <IP_ADDRESS>:envoyPort (virtual inbound listener for example) so much that envoy crashes due to file descriptor scarcity or what not. That doesnt' mean we lock down or get rid of virtual inbound and everything else.

@rshriram  Instead of replacing the TCP loopback address with UDS, what do you think of adding UDS as an option and default to TCP loopback address? In this case, we can provide a security option for users who care about it without breaking existing users.

Yes I dont mind if you want to create a UDS and enable it explicitly through annotation or what not. But TCP should always be the default. IMO it is unnecessary complexity, all in the name of a DDOS that is not even a legitimate threat.
Here are a few more DDOS vectors to consider:
I could choose to launch a 1000 pods in one namespace and churn them in a specific way to bring pilot down to its knees. That doesn't mean we disable the ability to create 1000 pods in a namespace or just stop automatically processing XDS requests.
I can choose to create a Sidecar crd for each one of my 100+ pods in the namespace with some very eccentric imports/exports or large number of inbound/outbound listeners. Does this mean we stop using the Sidecar resource or stop allowing people to create more than 3 listeners in each sidecar or disallow workload specific sidecars completely?
I can do the same with authN policies and authZ policies, or any workload based API, with 100s of ports, or 100s of rules for each workload. Do we disable all of these security resources by default in the name of "security" and DDOS mitigation on Pilot?
My point is that DDOS is a very gray area. Load != DDOS. A pod is a load, and all lifecycle around the pod is the same to pilot. Whether the pod is legitimately in a crashloopbackoff state or someone is frequently doing quit quit quit, it is all the same. We have ways to dampen this at pilot. Singling out just the envoy level stuff and devising complicated workarounds is not sustainable solution esp when the solution causes inconvenience to all our users.
All said, a http proxy via pilot agent passing stuff to the UDS is an interesting idea that can help us property tie the lifecycle of the agent and envoy, as pilot agent can exit after envoy exits properly. This should pose no breakage to end user as long as we dont do unnecessary things like blocking out /quit quit quit.  That said, given that @Stono and others like him have already managed to control sidecar lifecycle using quit-quit-quit, I am assuming that the pilot agent terminates automatically if Envoy died properly with a 0 exit code [cc @howardjohn / @hzxuzhonghu - is this correct?]. If this is correct, then I see no point in investing time and effort towards the http proxy on pilot agent.

Pilot agent has its own quit endpoint, but in a later release we also made the agent exit when envoy exits so either one works

Since UDS is also in networking namespace - shared with the app - I'm not sure what the difference is in terms of security. If app can run as 1337 - and bypass envoy completely - it also doesn't matter much if it can access /admin - there is nothing /admin does that app bypassing envoy can't do.
I think having a https + ACL interface to access admin and status from outside of the pod will be a better use of the time - replacing kubectl exec use in istioctl.

Since UDS is also in networking namespace - shared with the app - I'm not sure what the difference is in terms of security.

I believe that's not true for unix domain sockets that are bound to the filesystem, which is what we're considering here.

@costinm think of it like a path traversal attack.  You can trick a web server's file serving mechanism to serve a different file.   In this case, you could trick a proxy into forwarding a request to a port accessible within the netns (the admin port).
But you can't trick the proxy into forwarding a request to a UDS socket because it doesn't have the code to do that.

I think you are forgetting we have an explicit API in Istio for supporting UDS-based servers. Forwarding to TCP or UDS is equally easy using Istio APIs :-) We discussed in the past
the risk ( and benefits ) of exposing mysql UDS socket for example.
We are generally trying to adopt "zero trust" - and use L6 authn and authz policies instead of
relying on <IP_ADDRESS> or UDS. If the intent is to control who has access to the admin interface - we should use the same method we use for all our APIs, i.e. add a TLS filter and authz policy, and treat it  consistently with all other ports and services.
In other words: I would go the opposite way, and have /admin exposed with mTLS and an
auth policy allowing spiffe://.../istio-system/istiod and few other SA to access it.

I think UDS forwarding requires explicit user enablement (vs. say something based on ORIGINAL_DST which could potentially be hijacked).   So unless a user specifically configured UDS forwarding to the admin port, could Envoy be tricked into forwarding there?   At this point it's more of an Envoy code question @PiotrSikora @jplevyak @lambdai @yxue
Root question: does restricting the admin port to UDS make envoy any safer?

@duderino with zero knowledge on how Istio/k8s works, I can talk to the general case. Imagine you have an Envoy sidecar for a process in a VM. Even if binding to localhost, other local access to the VM can potentially access admin, which might constitute a privilege escalation, since admin is capable of mutating operations (e.g. quit, TAP) and data exfiltration (e.g. config dump). If you use appropriate network namespace controls this isn't an issue, but that is environment dependent. UDS can be locked down with regular file permissions.

@htuch can UDS be locked down such that the Envoy process can't be tricked into forwarding downstream requests to it?  I would imagine that the only way to prevent Envoy writing to the UDS would be to prevent everyone from writing to the UDS.
Or is it the case that Envoy just can't be tricked into forwarding downstream requests to it?  Or perhaps Envoy could only be tricked if certain routing rules were created (which my guess would be extremely rare)?

@duderino I don't see how it would forward to a UDS path unless that appeared explicitly in the CDS/EDS configuration somewhere. For TCP, I imagine dynamic forward proxy could do some interesting things. if enabled on a node (not an expert there though).

I might be missing something, but why was this issue closed? I don't see a solution for it.

Its closed as there is no work on this or plans to implement it, not because it was implemented (I think github is incorrectly calling it "completed")

@howardjohn a low hanging fruit way to achieve blocking access to port 15000 outright from application containers would be to leverage the owner module of iptables for filter table of OUTPUT chain of that pod's net namespace. We can either make it configurable to block specific UID / GID or just block non-root. The user of this feature would have to configure security context of their container to run as non-root for this to be effective, but at least there is a path to achieving sidecar/app container boundary that the OP was looking for here.
https://serverfault.com/questions/570668/how-to-block-access-to-a-local-user-to-a-local-port
I personally have an interest in this feature and would be willing to contribute if there's interest in accepting this approach.


Microsoft Account Logout on Exit in VSCode (Linux Snap Version)

Type: Bug
Description:
I have encountered a bug in the latest version of VSCode installed via Snap on Linux. When I exit the application, there is a significant chance that it automatically logs out the Microsoft account that is currently signed in.
This issue does not appear to occur on the Windows version, and I am unable to confirm whether it is reproducible with installations using .deb or other methods.
Steps to Reproduce:

Install VSCode via Snap on a Linux platform.
Sign in to a Microsoft account.
Exit the application multiple times.
Check if the Microsoft account remains logged in.

Expected Behavior:
The Microsoft account should remain logged in after exiting the application.
Actual Behavior:
The Microsoft account logs out upon exiting the application with a significant probability.
Additional Information:
Operating System: Linux
Installation Method: Snap
VSCode Version: Latest
Frequency of Issue: High (happens often, nearly always)
Thank you for your attention to this issue. I look forward to any updates or fixes.
NOTE: My English is not very good so this report is translated by LLM.
VS Code version: Code 1.95.1 (65edc4939843c90c34d61f4ce11704f09d3e5cb6, 2024-10-31T05:14:54.222Z)
OS version: Linux x64 5.15.0-124-generic snap
Modes:

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz (4 x 3371)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: enabled_ondirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_graphite: disabled_offvideo_decode: enabledvideo_encode: disabled_softwarevulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: disabled_offwebnn: disabled_off

Load (avg)
1, 1, 1

Memory (System)
7.71GB (2.34GB free)

Process Argv
--no-sandbox --force-user-env --crash-reporter-id 417c1325-b73e-4545-bd04-847c8dd386f2

Screen Reader
no

VM
0%

DESKTOP_SESSION
ubuntu

XDG_CURRENT_DESKTOP
Unity

XDG_SESSION_DESKTOP
ubuntu

XDG_SESSION_TYPE
x11

Extensions (17)

Extension
Author (truncated)
Version

tongyi-lingma
Ali
1.4.5

vscode-office
cwe
3.4.3

competitive-programming-helper
Div
2024.10.1729780523

code-runner
for
0.12.2

marp-vscode
mar
2.8.0

vscode-language-pack-zh-hans
MS-
1.95.2024103009

black-formatter
ms-
2024.4.0

python
ms-
2024.18.0

vscode-pylance
ms-
2024.10.1

cpptools
ms-
1.23.0

hexeditor
ms-
1.10.0

material-icon-theme
PKi
5.12.0

glassit
s-n
0.2.6

errorlens
use
3.20.0

vscode-luogu
ylt
4.10.4

markdown-all-in-one
yzh
3.6.2

file-size
zh9
1.0.0

(1 theme extensions excluded)

A/B Experiments
vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscod805cf:30301675
binariesv615:30325510
vsaa593:30376534
py29gd2263:31024239
c4g48928:30535728
azure-dev_surveyone:30548225
962ge761:30959799
pythongtdpath:30769146
pythonnoceb:30805159
asynctok:30898717
pythonmypyd1:30879173
2e7ec940:31000449
pythontbext0:30879054
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
dsvsc021:30996838
da93g388:31013173
dvdeprecation:31068756
dwnewjupytercf:31046870
impr_priority:31102340
nativerepl2:31139839
refactort:31108082
pythonrstrctxt:31112756
wkspc-onlycs-t:31132770
wkspc-ranged-t:31151552
cf971741:31144450
iacca1:31171482
notype1:31157159
5fd0e150:31155592
dwcopilot:31170013

Close Issue: Cause Found, Not Related to VSCode
I have identified the cause of the issue, and it is not related to VSCode. Therefore, I will be closing this issue. Thank you for your support!

May I know what was the issue

Close Issue: Cause Found, Not Related to VSCode I have identified the cause of the issue, and it is not related to VSCode. Therefore, I will be closing this issue. Thank you for your support!

May I know what the issue was?

Close Issue: Cause Found, Not Related to VSCode I have identified the cause of the issue, and it is not related to VSCode. Therefore, I will be closing this issue. Thank you for your support!

May I know what the issue was?

With my pleasure.

Network Fault
It seems that the MS Account will log in after VSCode finishes loading the extensions. So, if there are many extensions running on a slow machine, the user may need to wait a long time for VSCode to log in.


RM-22582 Dependency tracking

A requirement from Release Management is that dependencies are trackable. The easiest way to do this was use smithy/glide. I don't think any tests have to actually be executed in smithy, just a glide.lock generated, so I left tests in travis where non-workiva people could see them.
I also updated the go versions in travis to newer versions; as well as changed the environment variable setup to be one build with both variables instead of two builds with one environment variables each.
@stevenosborne-wf @dustinhiatt-wf
@travisreed-wf Can you verify this satisfies RM requirements?

Raven

Number of Findings: 0

lgtm

As long as this looks like an accurate reflection of your dependencies, I am happy
https://w-rmconsole.appspot.com/pulls/Workiva/go-datastructures/175/dependencies/

lgtm

+1


Fix for issue #164

Fixes issue #164 by

Converting back slashes or forward slashes in relative paths appropriately (depending on the OS)
Replacing fix back slash characters in some path related methods through System.IO.Path.DirectorySeparatorChar
Storing linked merge file settings paths with forward slashes instead of back slashes.
As this works across macOS, Windows and Linux I would prefer this to avoid later problems
with SCM when users work on the same project but different platforms.

Have to rework this so I'll close it.
Found out that the back slashes in the linked merge file settings aren't the problem and currently trying to figure out what's going on.

Found the problems with linked merge file settings and fixed them by adding additional checks in SettingsMerger.cs and GlobalSettingsFileOptions.cs.
Also fixed a problem that could at least occur on macOS if a relative path was saved in GlobalSettingsFileOptions.cs but it didn't start with a dot character.

I can now also confirm that a relative path without .\ at the start will also lead to problems on Windows, so my additional check and adding these characters if necessary fixes it for Windows too.

Screwed my repo and so starting from scratch, also veryfing if this is still a problem with the current code..


Nativescript 5 angular animation affecting margin and padding of the inner component

Environment
Provide version numbers for the following components (information can be retrieved by running tns info in your project folder or by inspecting the package.json of the project):

CLI:  playground
Cross-platform modules: "tns-core-modules": "5.0.1"
iOS Runtime: 5.0.0
NativeScript-Angular: 6.2.0 also 7.0.2
Angular: 6.1.10 aslo 7.0.0

Describe the bug
It seems when having a transform animation changes on left/top. The padding/margin is cut off from the inner component.

To Reproduce

Open the app
Click Toggle button
See the left padding/margin gets cut off

Expected behavior
The left padding/margin should be there
Sample project
https://play.nativescript.org/?template=play-ng&id=5Wu3ZR&v=5

Hi, @gogoout,
Thank you for the provided sample project.  I was able to recreate the issue for iOS. We will investigate further the case and will check what is causing the problem.

Hi @tsonevn ! Any updates on this?

+1

I have done some more testings on this issue.
https://play.nativescript.org/?template=play-ng&id=or2KMG&v=5
So this bug is not caused by angular animation system. Indeed, it is caused by the introducing of the safe area layout process in ios.
https://github.com/NativeScript/NativeScript/blob/master/tns-core-modules/ui/core/view/view.ios.ts
@profile
    public layout(left: number, top: number, right: number, bottom: number, setFrame = true): void {
        const { boundsChanged, sizeChanged } = this._setCurrentLayoutBounds(left, top, right, bottom);
        if (setFrame) {
            this.layoutNativeView(left, top, right, bottom);
        }

        if (boundsChanged || (this._privateFlags & PFLAG_LAYOUT_REQUIRED) === PFLAG_LAYOUT_REQUIRED) {
            let position = { left, top, right, bottom };
            if (this.nativeViewProtected && majorVersion > 10) {
                // on iOS 11+ it is possible to have a changed layout frame due to safe area insets
                // get the frame and adjust the position, so that onLayout works correctly
                const frame = this.nativeViewProtected.frame;
                position = ios.getPositionFromFrame(frame);
            }

            this.onLayout(position.left, position.top, position.right, position.bottom);
            this._privateFlags &= ~PFLAG_LAYOUT_REQUIRED;
        }

        this.updateBackground(sizeChanged);
        this._privateFlags &= ~PFLAG_FORCE_LAYOUT;
    }

In here, if the View inited outside the screen, the position return by  position = ios.getPositionFromFrame(frame); will reset the position to 0. Which cut the padding and margin of the View. Which means the cut off padding margin is happening in the period of entering animation
The workaround for this issue is to set the offset within the range of the margin or padding. (For this playground project, set the transition x within -20 will solve the issue).
Also I think this bug should be moved to the core nativescript repo.


Productize Encoding APIs

Pri 1

[ ] Hide System.Text.Utf8 and System.Text.Utf16 namespaces, except for Utf8String
[ ] Add TextEncoder overloads hinted in comments in source.
[ ] Move UnicodeCodePoint to some subnamespace, or hide for now, if not needed by bullet above
[ ] Support other encodings, e.g. 8859-*

Pri 2

[ ] Rename EncodingData to something better
[ ] Rename TextEncoder.Id type, and Scheme property

@joshfree, @terrajobst, @shiftylogic, @ahsonkhan, @botaberg, I added this issue to track work related to formatting APIs. Feel free to add/tweak workitems.

System.Text.Utf8 and System.Text.Utf16 namespaces need to be designed properly

Is this just a catch all item or can you quickly enumerate some additional issues here?

Let's just change the workitem to "hide the APIs except for Utf8String". I will make the update.

@joshfree, @ahsonkhan, I updated the workitem list. I tried to make it clearer, added priorities, and sorted in the order I think the work should be done.

All of the items except for 8859-* support are done. I created #1727 to track the remaining item and so closing this issue.


9 pictures from 2005

Here my 9 images from 2005 from the reddit post:
https://www.reddit.com/r/3Blue1Brown/comments/15v13ls/comment/jx8460g/

Here my 9 images from 2005 from the reddit post: https://www.reddit.com/r/3Blue1Brown/comments/15v13ls/comment/jx8460g/

Thank you so much for the contribution to the collection. Almost 2 decades ago, that's quite the treasure you dug up <3
I'll include the two most interesting.


Yale YRD450-ZW2/YRMZW2-US Communication Issues

Is your problem within Home Assistant (Core or Z-Wave JS Integration)?
NO, my problem is NOT within Home Assistant or the ZWave JS integration
Is your problem within Z-Wave JS UI (formerly ZwaveJS2MQTT)?
NO, my problem is NOT within Z-Wave JS UI
Checklist

[X] I have checked the troubleshooting section and my problem is not described there.

[X] I have read the changelog and my problem was not mentioned there.

Describe the bug
What causes the bug?
Attempting to control the lock state (locked/unlocked) via RF
What do you observe?
Most often, the lock physically actuates but fails to update the new status (locked > unlocked, or unlocked > locked) within a timely manner - typically takes 30-60 seconds.
If I physically actuate the lock bolt myself, I typically see the status reflected in a very timely manner, within a second or so. The issue seems to be limited to attempting to control the lock via RF.
What did you expect to happen?
I expect both the actuation of the bolt and the updated status to happen within a timely windows rather than the observed 30-60 seconds.
Steps to reproduce the behavior:

Attempt to toggle the lock state via HA
Usually, see the lock bolt actuate
Wait 30-60 seconds
Have HA reflect the new lock state

Device information
Manufacturer: Yale/Assa Abloy
Model name: YRD450-ZW2 w/ YRMZW2-US module (ZW+ v1, S0 security)
Node ID in your network: 22
How are you using node-zwave-js?

[X] zwave-js-ui (formerly zwavejs2mqtt) Docker image (latest)
[ ] zwave-js-ui (formerly zwavejs2mqtt) Docker image (dev)
[ ] zwave-js-ui (formerly zwavejs2mqtt) Docker manually built (please specify branches)
[ ] ioBroker.zwave2 adapter (please specify version)
[ ] HomeAssistant zwave_js integration  (please specify version)
[ ] pkg
[ ] node-red-contrib-zwave-js (please specify version, double click node to find out)
[ ] Manually built from GitHub (please specify branch)
[ ] Other (please describe)

Which branches or versions?
node-zwave-jsversion: 10.3.0
zwave-js-ui version: 8.1.0
Did you change anything?
yes (please describe)
If yes, what did you change?
I have attempted to use the lock both as undefined in current FW (lock is too new and has no device json) as as well as using lock by adding the code from this PR https://github.com/zwave-js/node-zwave-js/pull/5154 to the config folder.
I have attempted to (temporarily) move the stick about 5 feet closer to the lock. The total distance between stick and lock is less than 50 feet direct line of sight no walls. This did not make an impactful difference.
Did this work before?
Don't know, this is a new device
If yes, where did it work?
This is a new setup, I've just migrated to z-wave js. This lock previously worked reliably on Hubitat running the latest Hubitat code.
Attach Driver Logfile
zwavejs_2022-10-21.log
The event occurred around 2022-10-21T19:34 in the log file. I have emailed you my S0 code with reference to my GH user name.

Update - toggling the lock via the keypad externally also seems to induce these communication issues where toggling the bolt from the inside does not. I’ll pull a new log shortly with me performing a lock from the keypad.

Sorry for taking so long to respond, I'm still catching up with some issues after my leave.
Is this still happening? If so, can you provide a new log, preferably on loglevel "debug", not "silly"?

I've avoided the issue by running my ZW stick up in the attic as close to the front door lock as possible. The issue really seemed to be routing with S0 security. It would always force it's way back directly to the master node and then timeout trying to nonce handshake.
I am happy to move the stick back and run a few repairs though to capture a log if you feel there's value in doing that and possibly making the situation/timing of it better. I'd love to not have my stick running over a tcp bridge. Appreciate the info/help.

I can take a look, but this sounds like an issue in the physical world then.


Use annotation in python type unification

We're not actually using the annotation during assignment type unification, which causes problems for this case:
arr: List[Sprite] = []
for s in arr:
    s.destroy()  # <- can't find property "destroy"

Fixes "Chomper" for https://github.com/microsoft/pxt-arcade/issues/2083

Merging for tomorrow testing.


Revert "Remove Yate."

Reverts caskroom/homebrew-cask#28436.

When @reitermarkus opened this PR and I merged it, clicking the download link on the homepage sent you a website with a Captcha, but they seem to have now removed that.

I remember looking at it and seeing it should indeed be removed, but couldn’t remember the actual problem, now.


account for multiplicity in spot prediction/res-dep BW

since we have miller indices and spacegroup by the time we predict spots, we know the multiplicities of each spot. perhaps there is merit in adjusting the threshold accordingly. (granting that will require some thought to determine what "accordingly" means)

This should be straightforward to implement. I'm not sure if it'll make much difference, but it's an interesting idea.

would it make sense to replace the KDE with logistic regression? with a set of strong reflections and a set of not strong reflections as the training data and wavelength, rlp, multiplicity, and possibly the image number as predictive variables?


feature request: cross hair

Would it be possible to have a cross hair ✛ in the middle of the screen :)

That does make sense for racing. Will consider this for the next release if it doesn't take too much memory.

Will be included in next release, implemented in master


A small update

Todo: Update readme
@ryanburnette Usage is roughly the same, except it now supports nested complex objects, compatible with Asp.net binding.
var qurl = new Qurl();
qurl.query(); // {}
qurl.query('foo'); // undefined
Setting params
qurl.query('foo', 'bar');
qurl.query('array', [1,2,3]);
qurl.query('nested', [1, 2, {
chums : [{
name: 'bob',
age: 27
}]
});
qurl.query('jeff', false);
qurl.query('jeff'); // false (as Boolean)
qurl.query('nested') // returns the object
qurl.query(); //returns an object of all params
qurl.remove('jeff');
qurl.removeAll();
It'd be nice to get some unit tests for this, although I don't currently have the time to get those written up.

Awesome thanks for the work on enhancements! Testing would be great and
it's something I've been planning to do with my next patch.
On Thu, May 14, 2015 at 12:34 PM Robert White<EMAIL_ADDRESS>wrote:

Todo: Update readme
@ryanburnette https://github.com/ryanburnette Usage is roughly the
same, except it now supports nested complex objects, compatible with
Asp.net binding.
var qurl = new Qurl();
qurl.query(); // {}
qurl.query('foo'); // undefined
Setting params
qurl.query('foo', 'bar');
qurl.query('array', [1,2,3]);
qurl.query('nested', [1, 2, {
chums : [{
name: 'bob',
age: 27
}]
});
qurl.query('jeff', false);
qurl.query('jeff'); // false (as Boolean)
qurl.query('nested') // returns the object
qurl.query(); //returns an object of all params
qurl.remove('jeff');
qurl.removeAll();
It'd be nice to get some unit tests for this, although I don't currently
have the time to get those written up.
You can view, comment on, or merge this pull request online at:
https://github.com/ryanburnette/Qurl/pull/8
Commit Summary

Refactor, added support for nested objects
Collection fix
Small bug fix, cleaned up API, remove params

File Changes

M qurl.js
https://github.com/ryanburnette/Qurl/pull/8/files#diff-0 (232)

Patch Links:

https://github.com/ryanburnette/Qurl/pull/8.patch
https://github.com/ryanburnette/Qurl/pull/8.diff

—
Reply to this email directly or view it on GitHub
https://github.com/ryanburnette/Qurl/pull/8.

Glad you like it. Sweet, the API should be quite simple to test, but as it stands I have manually tested all implemented functionality and all use cases I can think of, it's more for future development.

I've got it on my to-do list to test everything you added today.

I'll update the readme tomorrow morning

@Rob-pw will this be done soon? I can only use Qurl if I'm able to pass an object of parameters :)

Alternatively I'd like to pass a serialized form array through and have all the parameters set that way :)

@jbrooksuk - Only the readme is left to update at this point, will likely do this during lunch today. You can pass an object and set parameters as follows:
var qurl = new Qurl();
qurl.query({
  myObj : [1,2, {
    withProps: true
  }]
}); //An optional third parameter (true) can be passed if you wish to override any params currently set

'pass a serialized form array' - Could you give an example usage?

I'm going to tinker on this a bit this morning.

Is everything good to merge the PR? Is anything required on my end?

@Rob-pw Man this has been forever since I looked at this project. Are you still using any of this code?

@ryanburnette Indeed. Not at the moment, though perhaps at some point, what are you proposing? I haven't researched alternatives, but I can imagine Qurl being used where independent modules each control a segment of query params.

@Rob-pw Only took me until 2020 to merge it. Might actually use this one day soon.


Watchtxid return errror

Return the following error when a new watchtxid is requested. BUT the watch is working as expected.
Proxy logs:
2020-04-14T20:05:10+0000 22402 [watchtxidrequest] id_inserted: 34
./requesthandler.sh: line 476: syntax error: bad substitution
2020-04-14T20:05:10+0000 22402 Entering response_to_client()...
2020-04-14T20:05:11+0000 22402 [main] exiting
2020-04-14T20:05:11+0000 22402 [requesthandler] exiting

Hi @Tomtibo !  Yes, we are aware of the bug, @gabidi fixed it in his PR here: https://github.com/SatoshiPortal/cyphernode/pull/180
It will be merged soon.  In the meantime, you can fix the typo like this: https://github.com/SatoshiPortal/cyphernode/pull/180/files#diff-b76cf609b11d32f75478eb2f6775ebcfR329
Let me know if it works for you.  Thanks!

OK thanks!

Yes it's work, I've just added the missing '}'.


BUG! exception in phase 'semantic analysis' in source unit 'BuildScript' Unsupported class file major version 65

can you provide more detail about the issue? possibly a screenshot or copy of what is in the debug console


Sync with Camunda Modeler v3.6.0

Closes #160

GitHub always complains to have conflicts when doing the sync (even though we have none), cf. #126
Once ending reviewing it, I'll do it locally, as we do it the last times.

Looks good to me, just removed some unnecessary stuff. Once AppVeyor sends his OK, I'll merge it.


[WFLY-10620] org.omg.api deps should be removed from jsf-multi-installer

JIRA : https://issues.jboss.org/browse/WFLY-10620

retest this please


Additional texture pixel formats [BC3, BC7, ETC2 with sRGB, ASTC 4x4]

This request is adding support for some broadly used compressed texture formats, which could be very useful with the basis_universal texture transcoding library.
Extended set of "known" formats with SRGB support:

BC3_SRGBA
BC7_SRGBA
ETC2_SRGB8
ETC2_SRGB8A8

"Brand-new" ASTC format:

ASTC_4x4_RGBA,
ASTC_4x4_SRGBA

4x4 subset is selected as the most broadly used and the only one of ASTC formats that is supported by the transcoding library.

Giving it a quick test now...

And merged. Thanks!
Only thing I changed was moving the SRGB pixel formats closer to their non-SRGB equivalents in the sg_pixel_format enum, and a small if-else coding style change.


Disintegrator updated incorrectly

In the most recent update, Disintegrator lost added spell damage and gained battlemage.
In PoB, the added physical damage (For attacks) was removed, and the spell damage remained. This should be reversed.


Dev/GSOC2018_MolVS_Integration

PR Summary
This PR is in reference to the Open Chemistry GSoC project RDKit-MolVS Integration Project.
Brief
The aim of the project was to integrate the widely used molecular standardization tool MolVS into the RDKit project. MolVS is built using the RDKit framework and so it follows that unification of the tools is desirable. Hence this project aimed to rewrite MolVS into the RDKit C++ source. Additionally, the project aimed to expand the current capabilities of MolVS by allowing the input of client-modifiable lists.
The original project proposal can be found here.
The project was be divided into the following components:

Standardization
Validation
Charges
Tautomers
Fragments

Outcomes
Source code was written for each component with exception to the tautomers piece (see below in TODO).
Extra functionality was made in the following:

Validation has four different modes: (1) RDKit Validation, (2) MolVSValidation, (3) AllowedAtomsValidation, (4) DisallowedAtomsValidation.
The default file used to define normalizations can be customized to a file supplied by the user. This is also the case for customizing the definitions in the acid base file and fragments file.

The code was wrapped using Boost.Python so that it can be used from python. The module is available at rdkit.Chem.MolStandardize.rdMolStandardize (see below tutorial for examples).
Tutorial
A tutorial was made in the form of an IPython Notebook to demonstrate the functionality of the written code in python. This can be found here.
Unit Tests
Unit tests were written alongside the source code and include a test for each component (Standardization, Validation, Charges, Fragments and Normalization). These unit tests used mostly the same examples as the corresponding MolVS unit tests.
Additionally, to test the equivalent functionality of the written C++ code with respect to MolVS, a short and a long test set of 1k and 100k molecules respectively were made for each component by downloading from PubChem Substance and choosing a subset of molecules.
TODO
As anticipated, the handling of tautomers is a difficult and contentious area and took longer than expected and therefore this work is still ongoing. Most of the tatuomer enumeration piece was written but tautomer canonicalization is still to do and further tests need to be written.
Additionally, although most of the framework is written, not all the same functionality as MolVS has been written or exposed by the python wrap. For example, the isotope_parent() and stereo_parent() standardize functions have not been written and charge_corrections has not been exposed as a standardization parameter.

Will this be integrated?
Please advertise on the users list. Looks useful.

Thanks for the feedback!
The current plan is to finish the review and then integrate the code; hopefully for the next release.
@susanhleung will present the work at the UGM next week and we will see where the discussion there goes.

This looks really useful - it would be great to have this in KNIME too! (see #2049 !)

Firstly this is going to be amazing useful. Well done Susan (and the same to Matt for the inspiration).
I thought I found a bug and was working up an example, but then saw that it seems to be present in Susan's notebook:

I'm assuming that Reionizer() is supposed to convert a carboxylic acid to the charged caboxylate.
The screenshot above suggests this is not happening.
More generally though, is the aim of the Reionizer() to provide a 'sensible' representation at neutral pH?
I would certainly hope so, though of course without a pKa predictor this can only be a rule based approximation.

Just for the record, there seem to be a lot of of compiler warnings introduced by these changes.
At the UGM hackathon Greg mentioned that these will be handle during the handling of the PR.

Firstly this is going to be amazing useful. Well done Susan (and the same to Matt for the inspiration).

Thank you very much @tdudgeon

I thought I found a bug and was working up an example, but then saw that it seems to be present in Susan's notebook:

I'm assuming that Reionizer() is supposed to convert a carboxylic acid to the charged caboxylate.
The screenshot above suggests this is not happening.
More generally though, is the aim of the Reionizer() to provide a 'sensible' representation at neutral pH?
I would certainly hope so, though of course without a pKa predictor this can only be a rule based approximation.

Thank you for pointing this out and this came up during the project as well when Greg pointed it out. Perhaps the name "Reionizer" is misleading... It works the same as the MolVS Reionizer by first applying charge corrections to ensure, for example, that free metals are correctly ionized. Then, if a molecule with multiple acid groups is partially ionized, ensure the strongest acids ionize first. For this molecule no metals are found and then it only adjusts charges if it detects an acid group AND a basic group. This is the same as the MolVS behavior (you can try it with MolVS)

@susanhleung OK so that makes sense. But makes it less useful.
Is there something can be used to handle a 'MoleculeAtNeutalPH()' standardization?
(of course with the already stated proviso that without a pKa predictor you can only do this using a dumb rules based approach, but that will probably cover 99% of the cases).

@tdudgeon: I think the framework is there to support that; we “just” need the transformations describing the ionized states. I would guess that this is the piece that ends in 1000 arguments

@susanhleung: I patched a bunch of warnings this morning and encountered some questions. I will send a PR against your repo either this evening or tomorrow morning so that you can take a look

@susanhleung This is looking really nice.  If you'd like me to transfer all of @mcs07 test cases to the Java JUnit test, let me know!

@sroughley : everything should now be set up so that the test file works. I'm not sure that we need all of the test cases in Java, but having a few more than what's currently there certainly would be nice.

@susanhleung It seems that when running from python tons of messages are output describing what was done during standardization. I see 2 problems with this:

it should be possible to control how much output you get (including none)
this output should be written to STDERR not STDOUT - that's where RDKit generally seems to write its informational messages.

I was initially going to agree with @tdudgeon, but then I realized that the appropriate changes have already been done. :-)
Tim: the output is being produced using the RDLogger functionality. You can disable it as follows:
# fully verbose:
In [2]: rdMolStandardize.StandardizeSmiles('[Na+].CC(=O)[O-]')
[15:41:34] Initializing MetalDisconnector
[15:41:34] Running MetalDisconnector
[15:41:34] Initializing Normalizer
[15:41:34] Running Normalizer
Out[2]: 'CC(=O)[O-].[Na+]'

In [3]: from rdkit import RDLogger

In [4]: RDLogger.DisableLog('rdApp.info')

In [5]: rdMolStandardize.StandardizeSmiles('[Na+].CC(=O)[O-]')
Out[5]: 'CC(=O)[O-].[Na+]'

@greglandrum And it's expected that the output goes to STDOUT not STDERR?

yeah, rdInfoLog goes to stdout.
It always has, but it's not used a lot inside the rest of the codebase.
We may want to think about adding a verbosity flag to the standardizer in the future for cases where you don't care to see all that output

@greglandrum OK, but just to be clear the current status seems a bit strange to me - when the molecule get's sanitized any messages get written to STDERR but when it gets standardized the messages go to STDOUT, so when reading a series of molecules and standardizing them you get output on both streams.

I agree that messages go to different places. From the perspective of someone using python in the notebook or from the command line, I don't think this is really visible. I believe this is only relevant for scripts/services that are capturing those two streams into separate places. Agreed?
Currently the messages from sanitization are (I think) exclusively errors or warnings - that seems logical to send to stderr.
The question then is if the informational messages generated by the standardizer should also go to an error channel (stderr) or stdout. I don't have a super strong opinion here, though I lean towards stdout (obviously).
@bp-kelley, @coleb any thoughts on whether rdInfoLog should also go to stderr instead of stdout?

@sroughley : everything should now be set up so that the test file works. I'm not sure that we need all of the test cases in Java, but having a few more than what's currently there certainly would be nice.

OK, thanks @greglandrum  - I will maybe take a look at least populating a couple of tests for each of the methods at some point.  I tend to agree that as it is a wrapper a minimal test set to check the wrapper works is fine with a detailed test set for the c++ layer

In my opinion, logging even info or diagnostic should never go to stdout for one simple reason: it breaks piping output to another program.
I.e.
molstandardize in.sdf | compute_cool_feature > results.sdf

Brian Kelley

On Sep 28, 2018, at 5:47 AM, Greg Landrum<EMAIL_ADDRESS>wrote:
I agree that messages go to different places. From the perspective of someone using python in the notebook or from the command line, I don't think this is really visible. I believe this is only relevant for scripts/services that are capturing those two streams into separate places. Agreed?
Currently the messages from sanitization are (I think) exclusively errors or warnings - that seems logical to send to stderr.
The question then is if the informational messages generated by the standardizer should also go to an error channel (stderr) or stdout. I don't have a super strong opinion here, though I lean towards stdout (obviously).
@bp-kelley, @coleb any thoughts on whether rdInfoLog should also go to stderr instead of stdout?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

I agree with @bp-kelley on this one, prefer messages going to stderr. Piping into unix utilities to break by default.

It's unanimous! I'll do a PR to switch rdInfoLog to stderr. :-)

I think this is really nice work. Two questions:

Is the integrated MolVS feature-complete compared to Matt’s original version (that is a bit unclear from the postings  I found so far)
Where will future improvements occur? At the original MolVS repository with regular transfers to RDKit, or already as part of the integrated version in RDKit? Any thoughts? :-)

Markus

|  Markus Sitzmann
|<EMAIL_ADDRESS>
On 29. Sep 2018, at 06:59, Greg Landrum<EMAIL_ADDRESS>wrote:
It's unanimous! I'll do a PR to switch rdInfoLog to stderr. :-)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

It's not feature complete yet. The tautomer handling isn't finished.
The next steps are going to depend somewhat on the community and motivation. I think the tautomer bit should be finished, but what comes after that is really open. Aside from more documentation of course...
@mcs07 will have to decide what happens with the original MolVS long term. It's certainly too early at the moment to think about retiring it though.


[feature] Suggestions

Hi!
Thank you for a fantastic service.
Maybe you can use these suggestions in your development.

Edit title directly in the table - e.g. by clicking or double-clicking like in Windows File Explorer.
Update frequency (3 hours, 12 hours etc.) based on tag allowing for more flexible edits (Edit: Now exists with "Groups" - Good job!)
Different update frequencies for day/night. Ie every 3 hours from 09.00 to 21.00 and then every 6 hours (that would be one time at 03.00 and then again at 09.00 where it recommences every 3 hours).
Better table sorting options in the main overview
Better standard filtering to remove cookie text for the standard HTTP GET request.
Allow multiple tags
More documentation about regex etc. and use cases
Backup should include settings
Highlight triggered word in different colour/bold
Better possibility to mass-add watches with same config, ie. "www.website.com?q=SearchA", "www.website.com?q=SearchB" and look for new unique lines.
More visual way to setup Jinja2 for e-mail notifications.
Select watches with mouse click and select area instead of clicking 25 times for 25 watches in the table.
Suggest similar pages to the one you are tracking
Reduce 403 access denied requests from intelligent websites

Thanks again!

I was too quick with this one. Different update times would be nice!
"Update frequency (3 hours, 12 hours etc.) based on tag allowing for more flexible edits (Edit: Now exists with "Groups" - Good job!)"


Could not load file or assembly Azure.Core

Receiving an error when trying to import the DLConversionV2 module in PowerShell, I assume it's an installed module version mismatch but I'm not sure which module and what version.
Import-Module : Could not load file or assembly 'Azure.Core, Version=<IP_ADDRESS>, Culture=neutral,
PublicKeyToken=92742159e12e44c8' or one of its dependencies. The system cannot find the file specified.
At line:1 char:1

Import-Module DLConversionV2

  + CategoryInfo          : NotSpecified: (:) [Import-Module], FileNotFoundException
  + FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.PowerShell.Commands.ImportModuleCommand

Please disregard, the fix for the Azure.Core.DLL also works for this issue.
Uninstall ExchangeOnlineManagement 3.5 and Installed version 3.4


Fix multiple warnings and proptype issues

Changes

Fixes multiple missing defaults (scale, focalX, focalY, isFill) and preserve them when using drop targets on an existing element
Fixes PropType mismatch coming from #524
Fixes warning about WithMasks missing children

Possibly fixes #807 and #381.

@swissspidy verified that it does


Add group upgrade timeout and canaries timeout logic

By default, the group upgrade as a whole will default to 4h but
it can be set with the new timeout field.
If canaries are used and that canaries batch takes too long, then
not move to next batch but move directly to upgrade timed out state.

/retest

/retest


The iframe element only supports the marimo.app

Describe the bug
I can open https://marimo.app/l/aojjhb?embed=true using iframe in marimo.new, but cannot open any other url except marimo.app
This online notebook demonstrates the problem : https://marimo.app/l/hgb5vm
Environment
marimo.new (WASM)
Code to reproduce
This online notebook demonstrates the problem : https://marimo.app/l/hgb5vm

This is actually a security issue. In order to run our playground (marimo.app) effectively with wasm, we enable some security flags on the top-level document. Specifically, cross-origin-embedder-policy and cross-origin-opener-policy.
If you want to embed an iframe, you must be in control of the sever serving the iframe's contents and similarly add the Cross-Origin-Resource-Policy header. For example,
Cross-Origin-Resource-Policy: cross-origin if the resource is served from another location than your website. If you set this header, any website can embed this resource.

Thank you for the explanation,

you must be in control of the sever serving the iframe's contents and similarly add the Cross-Origin-Resource-Policy header

I added the following content and restarted nginx, but it didn't seem to take effect.
add_header Cross-Origin-Resource-Policy cross-origin always;
add_header Cross-Origin-Embedder-Policy unsafe-none always;

Can you share a link with me so I can verify the correct headers are set? If you'd prefer, you can DM me on discord https://marimo.io/discord

Sure! Here is the link:  https://snap.codelab.club/snap.html
Here is the current nginx headers:
    add_header Cross-Origin-Resource-Policy cross-origin;
    add_header Cross-Origin-Resource-Policy credentialless;

Going to close this, please re-open if you are blocked.


[AutoPR network/resource-manager] Network july release branch is merging to master

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/3500

(message created by the CI based on PR content)

Codecov Report

Merging #3003 into master will increase coverage by 0.12%.
The diff coverage is 8.27%.

@@            Coverage Diff             @@
##           master    #3003      +/-   ##
==========================================
+ Coverage   56.35%   56.48%   +0.12%
==========================================
  Files        8124     8436     +312
  Lines      166280   173104    +6824
==========================================
+ Hits        93713    97777    +4064
- Misses      72567    75327    +2760

Impacted Files
Coverage Δ

...rk/azure/mgmt/network/network_management_client.py
28.77% <8.27%> (-3.22%)
:arrow_down:

azure-common/azure/common/client_factory.py
90.24% <0%> (-8.15%)
:arrow_down:

...t/compute/v2018_04_01/compute_management_client.py
35.44% <0%> (-7.9%)
:arrow_down:

.../resource/resources/v2018_05_01/models/identity.py
43.75% <0%> (-6.25%)
:arrow_down:

...ource/resources/v2018_05_01/models/identity_py3.py
43.75% <0%> (-6.25%)
:arrow_down:

.../azure/mgmt/media/models/live_event_preview_py3.py
35.29% <0%> (-4.71%)
:arrow_down:

...edia/azure/mgmt/media/models/live_event_preview.py
35.29% <0%> (-4.71%)
:arrow_down:

...compute/v2018_04_01/operations/disks_operations.py
39.55% <0%> (-4.28%)
:arrow_down:

...ute/v2018_04_01/operations/snapshots_operations.py
24.05% <0%> (-3.81%)
:arrow_down:

...rvicebus/models/migration_config_properties_py3.py
47.05% <0%> (-2.95%)
:arrow_down:

... and 336 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eed1e22...4d0a880. Read the comment docs.


[Bug]: [Helm] Setting Redis authentication doesn't set the depending redis chart values

Is there an existing issue for this?

[X] I have searched the existing issues

Description
Having no password as a default redis authentication is a critical security flaw in the Appsmith Business Helm Chart. So we need to change it to something safer.
When I deploy the Appsmith Business Helm Chart setting the value redis.auth.enabled: true and redis.auth.password, this is not propagated to the Appsmith instance in the ConfigMap, and the deployed Redis doesn't receive this configuration, which causes the deployment to fail.
Steps To Reproduce

Deploy the Appsmith Business Helm Chart setting the value redis.auth.enabled: true and redis.auth.password
Deployment will fail

Public Sample App
No response
Environment
Production
Issue video log
No response
Version
Self Hosted Business Helm Chart Version 3.0.4

Hi @pratapaprasanna, any updates? Do you have any timeframe for this fix to come? This is a critical security issue.

We are working on this and we shall be coming on fix for the same soon.


Improvements to SRL output format

With --outputFormat SRL the (ideal) result for "Mary told John off yesterday for eating pie" is something like:
told_off ARG0 Mary
told_off ARG1 John
told_off TMP yesterday
told_off CAU eating pie
eating ARG0 John
eating ARG1 pie

It would be nice if the format were enhanced to:

stem the predicates as in the logical form (told_off → tell_off, eating → eat)
include token offsets

So the improved format might look like:
tell_off-1,3 ARG0 Mary-0
tell_off-1,3 ARG1 John-2
tell_off-1,3 TMP yesterday-4
tell_off-1,3 CAU eating-6 pie-7
eat-6 ARG0 John-2
eat-6 ARG1 pie-7

I agree - will take a look.

Actually, I just realized that notation would be confusing because the offsets might be confused with PropBank frame numbers. Perhaps:
tell_off@1,3 ARG0 Mary@0
tell_off@1,3 ARG1 John@2
tell_off@1,3 TMP yesterday@4
tell_off@1,3 CAU eating@6 pie@7
eat@6 ARG0 John@2
eat@6 ARG1 pie@7

OK, I've added a new output format for this. Use the option "-o srl_indices" with the latest version of the code.
Sorry for the delay, I was having too much fun with my NAACL submission...


Update README.md

added github  sandramsc credentials

Updated location of github name as suggested

Updated location of github name to line no. 488

Hey Sandra, I believe the position should be after Ashim Devnath since (m --> p --> s), that is after line no. 488 as of latest merges of #2605. feel free to correct me if i am wrong

Updated location of github name to line no. 488


Does not work on Linux

The addon does not work (at least not on Debian9 Linux). It tries to open the freetube:// link in a browser window which results in a "adress not known, no registered protocol" message - which is correct, the protocol is not registered in firefox.
I can't find any addon code that registers the protocol. Is that code integrated in freetube? That would be a design flaw since freetube can't know which browser profile is located where and which browser profile has the addon.
I could try to debug if you pointed me to the code that actually registers the protocol with firefox.
Edit: I've found code in https://github.com/FreeTubeApp/FreeTube/blob/development/src/js/init.js but that's only working inside freetube, isn't it? I think you at least missed this: https://github.com/electron/electron/issues/4857#issuecomment-201124281 while configuring the deb file. I don't know whether this takes care of the problem inside firefox.
I still think the addon should register the protocol in the browser profile where the addon is installed (and only there). Which I think is what was suggested here: https://github.com/FreeTubeApp/FreeTube/issues/44#issuecomment-396980988

The protocol handler is registered via the desktop file on Linux. (at least on Gnome)
So adding MimeType=x-scheme-handler/freetube; to the freetube.desktop file will make the extension work.
So it should look something like this:
[Desktop Entry]
Name=FreeTube
GenericName=YouTube Player
Comment=An open source desktop YouTube player built with privacy in mind.
Exec=freetube %U
Terminal=false
Type=Application
Icon=freetube
Categories=Network;
MimeType=x-scheme-handler/freetube;

After adding the line to the desktop file one should run update-desktop-database from a terminal or restart the session/computer to activate the protocol.

Doesn't work for me either. Ubuntu 18.04 in Brave (chromium based) browser. I click youtube links and it opens as normal in the browser. The extension seemingly does nothing. I have Freetube desktop app installed via Flatpak.

The protocol handler is registered via the desktop file on Linux. (at least on Gnome)
So adding MimeType=x-scheme-handler/freetube; to the freetube.desktop file will make the extension work.

So where do I find this freetube.desktop file? Ubuntu 18.04 Gnome DE

@filipkemuel Adding MimeType=x-scheme-handler/freetube; to the desktop file does work indeed, but FreeTube doesn't seem to handle the freetube:// scheme correctly. The application sets the search bar input to "freetube-bin" and nothing more seems to happen.
I'm running Arch and installed FreeTube from the AUR.

It now seems to work fine without tweaks. As someone above mentioned about a fix added upstream, perhaps this is what solved it.

I use Manjaro with freetube-bin package in AUR.
In my case, I changed freetube.desktop file, but Firefox still shows "The address wasn't understood Firefox doesn't know how to open this address, because one of the following protocols (freetube) isn't associated with any program or is not allowed in this context. You might need to install other software to open this address."
I also can't see the option to configure Firefox to run the freetube binary under preference.

@filipkemuel Adding MimeType=x-scheme-handler/freetube; to the desktop file does work indeed, but FreeTube doesn't seem to handle the freetube:// scheme correctly. The application sets the search bar input to "freetube-bin" and nothing more seems to happen.
I'm running Arch and installed FreeTube from the AUR.
EDIT:
I managed to make it work : I had configured Firefox to run the freetube binary (which firefox suggested by default). Under Preferences -> General -> Applications I changed it to use xdg-open instead, now everything works fine !

This worked perfectly for me! To those wondering, to change to xdg-open, click on 'use other' and select /bin/xdg-open

@filipkemuel Adding MimeType=x-scheme-handler/freetube; to the desktop file does work indeed, but FreeTube doesn't seem to handle the freetube:// scheme correctly. The application sets the search bar input to "freetube-bin" and nothing more seems to happen.
I'm running Arch and installed FreeTube from the AUR.
EDIT:
I managed to make it work : I had configured Firefox to run the freetube binary (which firefox suggested by default). Under Preferences -> General -> Applications I changed it to use xdg-open instead, now everything works fine !

This worked perfectly for me! To those wondering, to change to xdg-open, click on 'use other' and select /bin/xdg-open

I changed the setting to xdg-open. Unfortunately it's still the same. The apps opens, but doesn't jump to the video link.

Unfortunately ive had the same observation as @stevenfang1997, for me the app loads the video if it is already open but if the link runs the app it will not load the video

I have some new observations. I added MimeType=x-scheme-handler/freetube in the desktop file, and changed the Firefox preference to xdg-open. After opening a Youtube link, I saw a error message "File not found: freetube://(link to the Youtube Video)".
I'm using freetube-bin from AUR repository.

According to the app developer, this problem seems to exist exclusively in the AUR and portable versions. In the non AUR and portable versions, you just need to disable "Grab Videos Locally" in the settings and it should be good to go.

i tried /bin/xdg-open freetube://[URL] in the terminal and doesnt open the video

I Added MimeType=x-scheme-handler/freetube; to /usr/share/applications/freetube-bin.desktop and configured Firefox to open freetube protocol with /bin/xdg-open
It still gives me an error Unable to create io-slave. klauncher said: Unknown protocol 'freetube'.
I'm using freetube-bin from AUR repository.

I Added MimeType=x-scheme-handler/freetube; to /usr/share/applications/freetube-bin.desktop and configured Firefox to open freetube protocol with /bin/xdg-open
It still gives me an error Unable to create io-slave. klauncher said: Unknown protocol 'freetube'.
I'm using freetube-bin from AUR repository.


API 1.7

It's been almost an entire month since AppDB updated their API and made this app no longer function properly. Are we going to get an update supporting the new API any time soon?
Maybe you should pass it off to someone who will actively develop this and not take months on end like the last time.....

It's been almost an entire month since AppDB updated their API and made this app no longer function properly. Are we going to get an update supporting the new API any time soon?
Maybe you should pass it off to someone who will actively develop this and not take months on end like the last time.....

feel free to fork the project yourself. Anyone willing to dedicate their time to “actively developing this” as you put it would have done so already.

Initial work has been done to support API 1.7, but a lot is missing. Draft PR is up here


Resolve rubocop offenses for auth_controller.rb

Since the last one was merged so fast I decided to do another file.
This time I'm not sure about how I fixed the long lines. Does anyone have a better idea?

I changed the long conditionals like @NuckChorris suggested and rubocop was ok with it.
I squashed everything so feel free to merge if nothing else needs to be changed.

:+1: You're really tearing through these! LGTM!

I don't have that much free time but sometimes I still don't know what to do with it :sweat_smile:


Add LabelsToTagsMap and EnsureStringArray functions to sdk plugin. closes #3

What name are we using for this function? Maybe it should be StringArrayToMap?

What name are we using for this function? Maybe it should be StringArrayToMap?

yup that's the question. StringArrayToMap is good

yup that's the question. StringArrayToMap is good


Build failure on BioC-devel due to Vignette/ExperimentHub

Hi, there is an issue with vignette rebuilding:
Error: processing vignette 'diffcyt_workflow.Rmd' failed with diagnostics:
package or namespace load failed for 'HDCytoData':
 .onLoad failed in loadNamespace() for 'HDCytoData', details:
  call: FUN(X[[i]], ...)
  error: 'Levine_32dim_SE' not found in ExperimentHub

Yours,
Steffen

-------- Forwarded Message --------
From: "Shepherd, Lori"<EMAIL_ADDRESS>To: bioc-devel<EMAIL_ADDRESS>Subject: [Bioc-devel] Annotation/Experiment  Hub issue
Date: Tue, 07 May 2019 12:32:58 +0000

We are in the process of fixing the issue with the AnnotationHub and
ExperimentHub for the devel 3.10  builds.  I hope to have a solution
shortly.   Currently maintainers whose packages are failing because of
the hubs do not have to take any action at this time, this is on our
end.


Fix DEBUG definition warning part 2

This pull request checks for both the definition and value of a macro to prevent -Wundef from generating a warning in FLAnimatedImage's logging support code. I missed this in https://github.com/Flipboard/FLAnimatedImage/pull/66, which enabled -Wundef and added prevention for the warnings it generates.

Merged at https://github.com/Flipboard/FLAnimatedImage/commit/43d625ef7d3f86329fbd75a139c1a966c631ed1e

:+1:


Create a Shards Game Tutorial

A Bishi-Bashi-like game to teach users about conditional sentences, arrays, showing images and variables.

[ ] Develop the game
[ ] Write a tutorial for the making of it

Hey team! Please add your planning poker estimate with Zenhub @ChelseaChX @ShahFragcolor


Decide if we support fluid spacing in javascript tokens

Typography and spacing tokens are static for javascript tokens. This is due to limitations in CSS with nested calc() only being supported using css-variables. Since js tokens can be scoped to only import one, this might lead to unintended bugs are referecend css-variables might not be present/imported.

Decide if we need to support js tokens or not.
If we decide to support js tokens we can update the js tokens to reference css-variables and document that you need to include the tokens.css in addition to using js tokens.

This is withdrawn as we plan to remove js tokens #1016


fix(customPropTypes): fix logic and warning messages

This is a breakout of #449.
This PR fixes a number of issues with customPropTypes.  Typos, bad messages, issues with logic, etc.  I also reworked them to use lodash/fp which cleaned things up a bit.
Later, we'll add the additional propTypes for shorthand, icon, image, children, etc.

Current coverage is 99.49% (diff: 100%)

Merging #579 into master will not change coverage

@@             master       #579   diff @@
==========================================
  Files           112        112
  Lines          1767       1767
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits           1758       1758
  Misses            9          9
  Partials          0          0

Powered by Codecov. Last update a89545d...97190f8

Released in<EMAIL_ADDRESS>


Nested component unmounts during drag

First -> Thank you for this library!

I have a nested set of components that are all draggable using a drag handle div.

When a component is being dragged, a new ghost image is additionally rendered to that component, and the ghost component is translated to mouse coordinates.

When any set of those components are dragged over others in the hierarchy, a feedback is shown on the component. The feedback is maintained through each component's own local state. Each component detects a mouse over event to show visual drag feedback while a global drag flag is set to true.

It turns out that dragging children to their parents can cause the parents to redraw the children completely -- which means unmount the children and create new components.

How do you prevent this kind of situation? I was thinking of making an overlay to put the ghost component in, but that doesn't really solve the problem because the useGesture hook is still cleaned up. Neither can I put the domTarget as something else in the DOM because the drag handle is in each component.

Hi @atifsyedali thanks for reporting this issue. Would there be a way for you to open a codesandbox with a minimal repro explaining your situation? Thanks!

Hi,
I think I have a very unique situation in that I have several nested divs and parent divs redraw during drag.  For now, I have implemented DnD myself as we are on a tight timeline. Going to close this issue as the problem was not related to this library.
Thanks.


[SPARK-27907][SQL] HiveUDAF with 0 rows throw NPE when try to serialize

What changes were proposed in this pull request?
When query returns zero rows, the HiveUDAFFunction.seralize throws NPE
create table abc(a int)
insert into abc values (1)
insert into abc values (2)
select histogram_numeric(a,2) from abc where a=3 //NPE
Job aborted due to stage failure: Task 0 in stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in stage 4.0 (TID 5, localhost, executor driver): java.lang.NullPointerException
at org.apache.spark.sql.hive.HiveUDAFFunction.serialize(hiveUDFs.scala:477)
at org.apache.spark.sql.hive.HiveUDAFFunction.serialize(hiveUDFs.scala:315)
at org.apache.spark.sql.catalyst.expressions.aggregate.TypedImperativeAggregate.serializeAggregateBufferInPlace(interfaces.scala:570)
at org.apache.spark.sql.execution.aggregate.AggregationIterator.$anonfun$generateResultProjection$6(AggregationIterator.scala:254)
at org.apache.spark.sql.execution.aggregate.ObjectAggregationIterator.outputForEmptyGroupingKeyWithoutInput(ObjectAggregationIterator.scala:97)
at org.apache.spark.sql.execution.aggregate.ObjectHashAggregateExec.$anonfun$doExecute$2(ObjectHashAggregateExec.scala:132)
at org.apache.spark.sql.execution.aggregate.ObjectHashAggregateExec.$anonfun$doExecute$2$adapted(ObjectHashAggregateExec.scala:107)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndexInternal$2(RDD.scala:839)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndexInternal$2$adapted(RDD.scala:839)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:327)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:291)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:327)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:291)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:94)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:122)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:425)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1350)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:428)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Hence add a check not avoid NPE
How was this patch tested?
Added new UT case

Can one of the admins verify this patch?

Can one of the admins verify this patch?

@srowen @vanzin @dongjoon-hyun @squito

Can one of the admins verify this patch?

I guess anything's better than an NPE. What's the result in this case?

null will be returned
+------------------------+
|histogram_numeric( a, 2)|
+------------------------+
|                    null|
+------------------------+

Found another NPE in org.apache.spark.sql.hive.HiveUDAFFunction#eval, updated fix and test code

Interesting, I wonder if the 'correct' answer should be the empty result set or null? @gatorsmile or @cloud-fan do you have insight into this one?

Interesting, I wonder if the 'correct' answer should be the empty result set or null? @gatorsmile or @cloud-fan do you have insight into this one?

I tried the same query on hive and it returned null, so i think we can keep it consistent with hive behaviour.?

LGTM

I have updated as per the review comments

ok to test

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11312/
Test PASSed.

Test build #106061 has started for PR 24762 at commit 9a4ed49.

Test build #106061 has finished for PR 24762 at commit 9a4ed49.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106061/
Test PASSed.

SPARK-24935 causes this regression at 2.3.4/2.4.3/3.0.0 and only 2.4.3 is released. I backported this to branch-2.3 since 2.3.4-SNAPSHOT has this regression, too.


Improve ApiVersioning support by automatically generate routes using Controllers definitions

Following #1393 and #1355, I'm only suggesting a possible approach for a better API Versioning support.
It would be really useful and less mistake-prone if NSwag could handle the different decleared versions of an API automatically.
Actually, by following #1393, all the different versions of API needs to be registered and configure manually in Startup (version number matching the one in the Controller Attribute, files, etc.).
A possible different approach could be to have something like this:
app.UseSwaggerUi3(typeof(Startup).GetTypeInfo().Assembly, config =>
{
	/* Looks inside Controllers for all ApiVersion() attributes and generates a route and a JSON file for each one.
	 * Also the DefaultUrlTemplate property could be taken from Route() controller attribute, if present.
	 */
	config.GeneratorSettings.UseMultipleVersions = true;
	config.SwaggerRoutes.LoadFromControllers();

	// Generic settings for all routes
	config.GeneratorSettings.Title = "My Amazing API";
	config.GeneratorSettings.Description = "Amazing API functionalities.";

	// Specific settings for single route
	foreach (var route in config.SwaggerRoutes)
	{
		route.DefaultUrlTemplate = "{v:apiVersion}/{controller}/{action}/{id?}";
		route.DefaultPropertyNameHandling = PropertyNameHandling.CamelCase;
	}
});

Thank you!

For reference, here are the existing tests: https://github.com/RSuter/NSwag/blob/master/src/NSwag.SwaggerGeneration.AspNetCore.Tests/ApiVersionProcessorWithAspNetCoreTests.cs
Controllers:
https://github.com/RSuter/NSwag/tree/master/src/NSwag.SwaggerGeneration.AspNetCore.Tests.Web/Controllers

Some news about that?


RDF4J Server java.util.concurrent.ExecutionException

I'm calling the RDF4J server from two different executions of a program that is updating some data on the server. The problem point is a clean-the-old before adding the new data. Both executions are at this code in my client at the same time.
More discussion and stack traces from RDF4J versions 2.0.1 and 2.1.3 are shown at https://groups.google.com/forum/#!topic/rdf4j-users/qtqso7TFnWM

As mentioned in the discussion: I have sofar been unable to reproduce the issue. OP will be looking into providing an MCVE.

This test is a Maven and Eclipse project. I ran it from the command line with
=> mvn clean install
=> mvn exec:java -Dexec.mainClass="org.rjk.test.rdf4jtest.Runner"
Runner starts two threads to run two executions in parallel. The main program uploads a large file to the server, then tries to clean up some possibly old data. The error on the server occurs during the clean up step. It seems dependent on how big the upload is. I tried reducing the size of the nt file; if it's much smaller there is no error; at this test size there's usually one thread successful and one fails; my production code, with a much bigger file, breaks in both threads. Once the exception is thrown on the server, i usually have to stop and restart it before I can test again or even get the workbench to access successfully.
Dick
rdf4jtest.zip

Thanks for the test case! I assume you closed it by mistake as the problem still exists :) I'll take a closer look as soon as possible.

I can reproduce the problem. Investigating further.

Ok, I still need to verify, but I think we may have unwittingly stumbled upon a critical error in how RDF4J server handles non-transactional requests. If I'm right, we'll have found (and hopefully fixed) a bug that may have been present since at least 2012...

Definitely closed by accident. First time on this site. I thought "Close and comment" meant "close this comment", not "close this issue and comment".
It looks like you're making progress on this. Thanks! We definitely need it since we expect to run that main process on perhaps hundreds of  ~140,000 statement models (not all at once, but definitely more than one at a time).
(About to press "Comment" and see if I get it right.)

How can I get hold of this for my work?- Wait for a release? Pull
(something) and build? Or ...?
On Wed, Dec 7, 2016 at 6:43 PM, Jeen Broekstra<EMAIL_ADDRESS>wrote:

Closed #680 https://github.com/eclipse/rdf4j/issues/680 via f213620
https://github.com/eclipse/rdf4j/commit/f213620b0bff15a800868edda81a7794dc498890
.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/eclipse/rdf4j/issues/680#event-886121892, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AXJlvQh3jnmpZXlPYEa42xYMz7pHrq7eks5rF0S2gaJpZM4LATs0
.

We will be releasing a patch release with this fix in it, probably middle of next week. However if you're able to take it for a test now, you can grab a nightly build or build directly from source - would be good to get independent confirmation that the issue is indeed fixed.

Btw if you want to test by just installing an updated RDF4J Server, you can just download a snapshot build war file. Grab the most recent one war file there, download, rename (to rdf4j-server.war), then (re)deploy and you should be good to go. Not recommended in a production environment :)

I have grabbed the latest war file,
rdf4j-http-server-2.2-20161209.051730-24.war. So far running my test I'm
finding no errors reported in the test application or RDF4J server, but the
triples are not always being written. At first I was seeing both contexts
appearing when I ran the test from within eclipse, but when I ran from the
command line sometimes only one context and sometimes neither were written.
I added "conn.clear(context)" prior to writing to RDF4J, then I started to
get the intermittent results from both eclipse and command line. What I did
not do was update my test app to use the newer jars. I'm going to try to
pull and build and rerun the test with jars matching the war file.
On Thu, Dec 8, 2016 at 4:38 PM, Jeen Broekstra<EMAIL_ADDRESS>wrote:

Btw if you want to test by just installing an updated RDF4J Server, you
can just download a snapshot build war file
https://oss.sonatype.org/content/repositories/snapshots/org/eclipse/rdf4j/rdf4j-http-server/2.2-SNAPSHOT/.
Grab the most recent one war file there, download, rename (to
rdf4j-server.war), then (re)deploy and you should be good to go. Not
recommended in a production environment :)
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/eclipse/rdf4j/issues/680#issuecomment-265863289, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AXJlvTgyXtlLcEbniV4XmvuodQeStFpWks5rGHjggaJpZM4LATs0
.

That sounds odd. If this problem persists, can you show me your updated code (if any) and/or what I should be querying for in the test db to verify that all expected data is present?

I've attached my updated test. I did not include the huge data file, since
I did not change that, but you'll still need it. I made changes to improve
logging and I added 'clearContext' to Uploader.
To check results: The same data is being loaded into two different
contexts. After a run I check Workbench. If everything loaded you'll see
two contexts named in 'Contexts' view, and 'Summary' will show :
Repository Size
Number of Statements    139998
Number of Labeled Contexts    2

If you go to a context, then  pick any subject URI, the 'Explore' view that
comes up will show a set of quads in pairs, only different in context. For
example, exploring <
http://purl.rjk.test.org/sip/graphs/rjk-demo-opt/i/00000012> will show 4
triples, but each triple repeated in 2 quads, 1 for each context.
If only one context made it (which is what I'm usually seeing):
Repository Size
Number of Statements    69999
Number of Labeled Contexts    1

and single sets of quads.
And, sometimes, nothing makes it:
Repository Size
Number of Statements    0
Number of Labeled Contexts    0

These different results occur without error in my test code and without
error on RDF4J Server.
On Fri, Dec 9, 2016 at 11:22 PM, Jeen Broekstra<EMAIL_ADDRESS>wrote:

That sounds odd. If this problem persists, can you show me your updated
code (if any) and/or what I should be querying for in the test db to verify
that all expected data is present?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/eclipse/rdf4j/issues/680#issuecomment-266178000, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AXJlvcqFeZgMPYvS2IJPJ0sUfa99NpHWks5rGijugaJpZM4LATs0
.

@rknowles9 unfortunately GitHub is not clever enough to pick up e-mail attachments. You'll have to comment via the web UI and add the attachment that way...

I've managed to reproduce some further related issues. I'm reopening this.

Root cause is a concurrency/synchronization issue in SailSourceModel.

Created a followup issue, #693, which is related not but not identical. Closing this, discussion can continue on the new issue. @rknowles9  thanks again for assisting with debugging this. I think I have it covered now. I'll ping you when I have a new snapshot build available, if you're available to take it for quick test spin again that'd be great.


Save/Load config as yaml

This PR converts pickled configurations into yaml configuration.
It introduces a breaking change to the current UI (meaning that old experiments won't work). As the repo is in its initial phase, I did not add any conversion scripts to load pickle files and convert them into yamls to keep the repo clean.
Fixes  #4

Add also an option to pass a yaml via CLI

Shall we maybe also add a NEW section to the Readme updating about more major changes?

Shall we maybe also add a NEW section to the Readme updating about more major changes?

Yes changelog makes sense.

@pascal-pfeiffer could you share the configuration files you had issues with?

@pascal-pfeiffer could you share the configuration files you had issues with?
Did we save stop_tokens as a list at some point?

I think, I had some mix in my install. Seems to work just fine with the latest commit. Let me do one more review and I think we are good to merge it.


🎉 2.0.3

ℹ About this release

Version: 2.0.3
Type: patch
Last Release: 2.0.2 (10/24/2024, 8:07:01 PM) [?]
Commits to merge: 13 [?]

📦 Dependencies

Update @prisma/client from ^5.21.1 to ^5.22.0
Update @sentry/node from ^8.33.1 to ^8.37.1

📦 Development Dependencies

Update @eslint/js from ^9.11.1 to ^9.13.0
Update @sebbo2002/semantic-release-docker from ^5.0.1 to ^5.0.2
Update @types/node from ^22.7.5 to ^22.9.0
Update eslint from ^9.11.1 to ^9.13.0
Update mocha from ^10.7.3 to ^10.8.2
Update prisma from ^5.21.1 to ^5.22.0
Update tsup from ^8.3.0 to ^8.3.5
Update tsx from ^4.19.1 to ^4.19.2
Update typedoc from ^0.26.8 to ^0.26.11

:tada: This PR is included in version 2.0.3 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release
v2.0.3

Your semantic-release bot :package::rocket: