refactor!(libsinsp): coherent metrics interface metrics_collector class + text-based Prometheus exposition format support

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area API-version

/area build

/area CI

/area driver-kmod

/area driver-bpf

/area driver-modern-bpf

/area libscap-engine-bpf

/area libscap-engine-gvisor

/area libscap-engine-kmod

/area libscap-engine-modern-bpf

/area libscap-engine-nodriver

/area libscap-engine-noop

/area libscap-engine-source-plugin

/area libscap-engine-savefile

/area libscap
/area libpman
/area libsinsp

/area tests

/area proposals

Does this PR require a change in the driver versions?

/version driver-API-version-major

/version driver-API-version-minor

/version driver-API-version-patch

/version driver-SCHEMA-version-major

/version driver-SCHEMA-version-minor

/version driver-SCHEMA-version-patch

What this PR does / why we need it:

Everyone hated the initial linsinsp get_* methods wrt stats or metrics, which were expanded as an intermediary step to support the sinsp state counters metrics. As promised, they have all been removed now.
It simplifies the consumption of metrics by combining libscap and libsinsp metrics into one vector.
The type of metrics is uniformly gated by pushing flags down, as done previously.
Easier memory management is achieved by using std::vector as the data structure for m_metrics for safer management. This also helps reduce the knowledge the client needs about the metrics or the number of possible metrics. The client can simply loop over metrics and gate actions based on the metrics' string names.
These changes do not affect the hot path, and performance is not of primary concern.
Previous metrics are preserved; only code refactoring has been done.
A significant renaming attempt has been made to create a more coherent metrics concept while retaining the concepts of scap stats and sinsp stats for what concerns the hot path. Unfortunately, creating a super consistent and transparent stats or metrics naming convention is challenging. In this regard, this PR is a small step forward at best. Due to the renaming, this PR has a large diff, but in reality, it's more about moving and reorganizing code.
Memory unit conversion has been moved to the class natively.
The complete metrics_v2 schema now features not just the unit but also a new metric type indicating whether the metric is monotonic or reflects the current snapshot state.

There are currently no plans to introduce a metrics writer or move the ticker and stats intervals from Falco into libs at the moment, but this could be discussed in the future.
Which issue(s) this PR fixes:
https://github.com/falcosecurity/libs/issues/1463

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

refactor!(libsinsp): coherent metrics interface, new light weight `metrics_collector` class + technical debt cleanup

Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.
:memo: Please follow instructions in the contributing guide to update your commits with the DCO
Full details of the Developer Certificate of Origin can be found at developercertificate.org.
The list of commits missing DCO signoff:

e9553f0 update(scap): update scap_stats_v2 schema

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: incertum
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

CC @federico-sysdig to help making sure it's now a better design :) Thank you!

/milestone 0.15.0

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: federico-sysdig, incertum
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Minor additions to the public convert_metric_to_prometheus_text
Also added some comments around why a Registry might be excessive for our metrics use cases. I believe that this simple vector containing all metrics, following our internal metrics_v2 schema, should be good enough and easier to manage. Looking forward to hearing all of your thoughts?
During yesterdays maintainer call @leogr brought it to our attention that ideally https://github.com/falcosecurity/falco-exporter should be deprecated -- I added a new macro  METRICS_V2_RULE_COUNTERS in preparation to support counts related to the actual rules and alerts. Perhaps initial support for Falco makes sense, and later, we can move it to libs, as I am not certain if libs ready for those stats quite yet
Finally, what are your thoughts on this straightforward approach to support Prometheus by exposing prom text lines at /metrics through Falco's existing web server?

One requirement should be that the solution is acceptable for most of us for the foreseeable future (e.g. for Falco 1.0.0) until the next refactor is due one day.

Hi 👋 everyone, @mickael-carl who is very knowledgeable about large scale real-life Prometheus applications is willing to help! @mickael-carl I pushed a humble refactor commit incorporating some of your initial feedback.

One thing I totally got wrong was to assume that it is ok to not expose # TYPE -- however if you don't ship it, end users could misuse / break Prometheus queries against a wrong type. Therefore, it should almost be regarded as a no go to not send it. It's now adjusted.
In addition, I carefully read the documentation regarding what "not to do" in regards to labels and naming conventions.
Not a fan of prom base_units, IMO just ship the unit and you are fine. Also seconds doesn't work for us for the libbpf stats metrics (we need nanoseconds in those cases).

More details to be added later. @mickael-carl I know you are busy the next few days, no rush! Thanks for all the tips you shared so far!

@sgaist ty! re the proposed refactor related to using std::filesystem e.g. https://github.com/falcosecurity/libs/pull/1652#discussion_r1482147014 I would propose defering this to a dedicated PR as we may have other cases of the same in other parts of the code base. WDYT?
re the formatting, noted. I'll do one final commit at the end and include all linting and formatting and typo corrections etc.

During yesterdays maintainer call @leogr brought it to our attention that ideally https://github.com/falcosecurity/falco-exporter should be deprecated -- I added a new macro  METRICS_V2_RULE_COUNTERS in preparation to support counts related to the actual rules and alerts. Perhaps initial support for Falco makes sense, and later, we can move it to libs, as I am not certain if libs ready for those stats quite yet

falco-exporter metrics are about triggered rules in Falco, so I guess that supporting the counters here is fine, but the actual implementation will live in Falco.
Also, I want to confirm that I totally onboarding in deprecating falco-expoter once we have a replacement)

Finally, what are your thoughts on this straightforward approach to support Prometheus by exposing prom text lines at /metrics through Falco's existing web server?

I agree with this :+1:

Hi, this PR seems to have stalled after my last push two weeks ago.
I wanted to point out the risks we are all accepting by delaying this PR. I hope it helps in deciding on the prioritization, acknowledging that there are competing priorities and other equally important code changes in the pipeline.

A metrics code refactor was highlighted as important by other maintainers. All these metrics are incredibly needed to ensure Falco continues to work well in production. They have even helped us find optimization potential. For instance, we didn't know before that the old default thread table size was too small for modern infrastructures, resulting in dropped events and broken process trees, even though we had the events in userspace.
Prometheus support was requested about two releases ago and is also requested by the TAG Environmental Sustainability - Green Reviews WG initiative. Isn't it great that someone is stepping up and writing the code? We have even received expert Prometheus-related change requests already, which is a privilege.
To make the Prometheus support happen, we need one additional libs PR after this PR. After that, Falco needs to be synced first with these libs changes, and then we need the Falco PR that exposes the new /metrics endpoint plus testing. Assuming no one else is willing to help and I do everything, and we want to make it for Falco 0.38.0, here is my availability: No availability at all in May, only two weeks availability in March, and two weeks availability in April.
Leo also posted that he would like to deprecate falco-exporter. This would mean another significant Falco PR is needed should we want to make all this happen for Falco 0.38.0.

Lastly, I would like to highlight that we have no complete and formal coding guidelines, and that preferences have varied greatly in past PR reviews.

I just want to confirm this is a priority for Falco 0.38 as agreed during our last core maintainer meeting

@leogr quick question: maybe it makes sense for me to add the Prometheus metrics conversions into this PR, so its all done, else we may not have cycles for another PR review? It should be a fairly small additional commit.
In addition I just opened the falco libs sync PR as WIP.

Note: @federico-sysdig wanted to cleanup some part in https://github.com/falcosecurity/libs/pull/1693 after this PR is done.

@LucaGuerra @FedeDP any clue? :thinking:

Already saw it here: https://github.com/falcosecurity/libs/pull/1715.
No clue for now.

@sgaist since we are doing a handoff on the Falco side and you will work on the webserver extension, could you double-check as this is what you will use to work with. Thanks!

@sgaist: changing LGTM is restricted to collaborators

In response to this:

I found some stuff that could be simplify to avoid code duplication.
Note that I am not sure about the where I put the helper methods so feel free to move them around following best practice.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@sgaist thank you for the valuable code deduplication cleanup!

@sgaist: changing LGTM is restricted to collaborators

In response to this:

From the looks of it, things are now in a good state.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

From the looks of it, things are now in a good state.

ACK, also wouldn't be the end of the world if we needed a small follow up cleanup PR as long as at least we now agree on the broader interface for Falco 0.38.0. Eventually any code is legacy anyways ...

LGTM label has been added.  Git tree hash: d159b325191cf03aa465ddc5991239d331ba6c98

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: federico-sysdig, incertum, leogr, sgaist
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [incertum,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/milestone 0.15.0

I've put/changed the milestones just to organize the PRs for the upcoming release, of course, feel free to change them if you have different ideas/plans :)
Same on all the other PRs!

New changes are detected. LGTM label has been removed.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: federico-sysdig, incertum, sgaist
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: federico-sysdig, incertum, sgaist
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LGTM label has been added.  Git tree hash: 8135fee71eb30024775bdf2d0ad1c1a1ce1860fd

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, FedeDP, federico-sysdig, incertum, sgaist
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Andreagit97,FedeDP,incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Throw error if files are missing.

What was happening if you ran this with a non existent consignment was it would try to get the files from the api and the api would return nothing.
This empty list was passed to the s3 downloader which would download nothing.
The program would continue until we had a bagit package with an empty data directory.
Arguably, the API should be rejecting non-existent consignments but for now, this will throw an error if no files are returned by the api.

The error you've suggested makes sense, I've changed it. I also realised that there are no tests for that error so I've added in a test to MainSpec to check for it.

The error you've suggested makes sense, I've changed it. I also realised that there are no tests for that error so I've added in a test to MainSpec to check for it.


how to speed up proposal sample network of nerfacto

"I would like to enhance the speed of the training process and have noticed that the forward (fwd) and backward (bwd) operations of the proposal network are quite time-consuming. Additionally, the proposal network seems to be unstable when trained on masked images containing relatively small objects (comprising about 15% of the pixels).
I am considering that it may be challenging for the proposal network to fit such a compact distribution along an long and extended marching ray. Perhaps utilizing a smaller axis-aligned bounding box (AABB) generated by COLMAP could be beneficial?"

@blacksino Hey, I have the same problem. The proposal network of nerfacto costs too much time, did you get some improvements?


Tests are broken

Because of this: https://github.com/thenineteen/Semiology-Visualisation-Tool/commit/8395a5916a535106096a63056498bcec135e43b2

It passes on pandas version 1.0.3
could you check your machine pandas version?
see: https://github.com/blue-yonder/tsfresh/issues/485

actually my bad! not a version issue
I forgot to push the database update!


Remove Host header attribute from proxied request

Solves issue https://github.com/cloudflare/unsee/issues/250

Before I merge this I need to check if we should delete it or overwrite it, but seems like you found the root cause. I'll do that a bit later and once I'm confident that the right solution I'll release 0.9.2.
Would be also nice to add some tests before we merge a fix.
Thanks!

From https://golang.org/src/net/http/request.go#L133:
	// For incoming requests, the Host header is promoted to the
	// Request.Host field and removed from the Header map.

So I do think that we should drop Host header, but for completes we might also set req.Host

req.Host = upstreamURL.Host
req.Header.Del("Host")

Just to be sure - @svenmueller can you verify if it's still working for you with those 2 lines?
Without enforcing req.Host I'm able to proxy a request with custom (bogus) Host header and it's being send upstream, which shouldn't be allowed, so I'd like to add that.

if i dump the whole request, it looks like this. The host header is still set.
POST /proxy/alertmanager/test/api/v1/silences HTTP/1.1
Host: unsee.mydomain.com
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: de,en-US;q=0.7,en;q=0.3
Connection: Keep-Alive
Content-Length: 421
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Cookie: optimizelySegments=...; optimizelyEndUserId=...; _ga=...; amplitude_idmydomain.com=...; GCP_IAAP_XSRF_NONCE=...
Referer: https://unsee.mydomain.com/?q=
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0
Via: 1.1 google
X-Cloud-Trace-Context: ...
X-Forwarded-For: xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx
X-Forwarded-Proto: https
X-Goog-Authenticated-User-Email: ...
X-Goog-Authenticated-User-Id: ...
X-Goog-Iap-Jwt-Assertion: ...
X-Requested-With: XMLHttpRequest

{...}

Ok, overwriting the host value seems to work. I can see that the correct hostname is logged in the logs of the target Alertmanager. Only problem is that the Basic Auth information is not passed to the proxied target, which results in a 401.

Ensuring we pass the correct Host header is needed, but I need to setup proper test coverage for this. I'll follow up this PR with more changes.

I've added more tests and confirmed that basic auth is missing for proxied requests - see #256


Request Feature: load json from local storage

Added the method to load json from local storage, but I still need to check if the parameter is a string, url, or file path

Not yet, I'm away for a while.
But I'll definitely do, cuz my project is heavily dependent on it.
But you can try it on your end lemme know.


Drop conan

The project can be simplified a lot by dropping Conan support.

6df18f6b9b55ba946805c4878fb4f269a80ce639 removed conan from the CI for darwin and windows.

8103b0107b1d1e52e3c85568989f2d73fc1f25bd replaced the last conan build on the CI with vcpkg


[SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into …

What changes were proposed in this pull request?
This PR adds some tests converted from window.sql to test UDFs. Please see the contribution guide of this umbrella ticket - SPARK-27921.
Diff comparing to 'window.sql'

...  diff --git a/sql/core/src/test/resources/sql-tests/results/window.sql.out b/sql/core/src/test/resources/sql-tests/results/udf/udf-window.sql.out
index 367dc4f513..e556d25f7d 100644
--- a/sql/core/src/test/resources/sql-tests/results/window.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/udf/udf-window.sql.out
@@ -21,10 +21,10 @@ struct<>

 -- !query 1
-SELECT val, cate, count(val) OVER(PARTITION BY cate ORDER BY val ROWS CURRENT ROW) FROM testData
+SELECT udf(val), cate, count(val) OVER(PARTITION BY cate ORDER BY val ROWS CURRENT ROW) FROM testData
 ORDER BY cate, val
 -- !query 1 schema
-struct<val:int,cate:string,count(val) OVER (PARTITION BY cate ORDER BY val ASC NULLS FIRST ROWS BETWEEN CURRENT ROW AND CURRENT ROW):bigint>
+struct<CAST(udf(cast(val as string)) AS INT):int,cate:string,count(val) OVER (PARTITION BY cate ORDER BY val ASC NULLS FIRST ROWS BETWEEN CURRENT ROW AND CURRENT ROW):bigint>
 -- !query 1 output
 NULL   NULL    0
 3      NULL    1
@@ -38,10 +38,10 @@ NULL        a       0

 -- !query 2
-SELECT val, cate, sum(val) OVER(PARTITION BY cate ORDER BY val
+SELECT udf(val), cate, sum(val) OVER(PARTITION BY cate ORDER BY val
 ROWS BETWEEN UNBOUNDED PRECEDING AND 1 FOLLOWING) FROM testData ORDER BY cate, val
 -- !query 2 schema
-struct<val:int,cate:string,sum(val) OVER (PARTITION BY cate ORDER BY val ASC NULLS FIRST ROWS BETWEEN UNBOUNDED PRECEDING AND 1 FOLLOWING):bigint>
+struct<CAST(udf(cast(val as string)) AS INT):int,cate:string,sum(val) OVER (PARTITION BY cate ORDER BY val ASC NULLS FIRST ROWS BETWEEN UNBOUNDED PRECEDING AND 1 FOLLOWING):bigint>
 -- !query 2 output
 NULL   NULL    3
 3      NULL    3
@@ -55,20 +55,27 @@ NULL        a       1

 -- !query 3
-SELECT val_long, cate, sum(val_long) OVER(PARTITION BY cate ORDER BY val_long
-ROWS BETWEEN CURRENT ROW AND<PHONE_NUMBER> FOLLOWING) FROM testData ORDER BY cate, val_long
+SELECT val_long, udf(cate), sum(val_long) OVER(PARTITION BY cate ORDER BY val_long
+ROWS BETWEEN CURRENT ROW AND CAST(2147483648 AS int) FOLLOWING) FROM testData ORDER BY cate, val_long
 -- !query 3 schema
-struct<>
+struct<val_long:bigint,CAST(udf(cast(cate as string)) AS STRING):string,sum(val_long) OVER (PARTITION BY cate ORDER BY val_long ASC NULLS FIRST ROWS BETWEEN CURRENT ROW AND CAST(2147483648 AS INT) FOLLOWING):bigint>
 -- !query 3 output
-org.apache.spark.sql.AnalysisException
-cannot resolve 'ROWS BETWEEN CURRENT ROW AND<PHONE_NUMBER>L FOLLOWING' due to data type mismatch: The data type of the upper bound 'bigint' does not match the expected data type 'int'.; line 1 pos 41
+NULL   NULL    1
+1      NULL    1
+1      a<PHONE_NUMBER>
+1      a<PHONE_NUMBER>
+2      a<PHONE_NUMBER>
+2147483650     a<PHONE_NUMBER>
+NULL   b<PHONE_NUMBER>
+3      b<PHONE_NUMBER>
+2147483650     b<PHONE_NUMBER>

 -- !query 4
-SELECT val, cate, count(val) OVER(PARTITION BY cate ORDER BY val RANGE 1 PRECEDING) FROM testData
+SELECT udf(val), cate, count(val) OVER(PARTITION BY cate ORDER BY val RANGE 1 PRECEDING) FROM testData
 ORDER BY cate, val
 -- !query 4 schema
-struct<val:int,cate:string,count(val) OVER (PARTITION BY cate ORDER BY val ASC NULLS FIRST RANGE BETWEEN 1 PRECEDING AND CURRENT ROW):bigint>
+struct<CAST(udf(cast(val as string)) AS INT):int,cate:string,count(val) OVER (PARTITION BY cate ORDER BY val ASC NULLS FIRST RANGE BETWEEN 1 PRECEDING AND CURRENT ROW):bigint>
 -- !query 4 output
 NULL   NULL    0
 3      NULL    1
@@ -82,10 +89,10 @@ NULL        a       0

 -- !query 5
-SELECT val, cate, sum(val) OVER(PARTITION BY cate ORDER BY val
+SELECT val, udf(cate), sum(val) OVER(PARTITION BY cate ORDER BY val
 RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING) FROM testData ORDER BY cate, val
 -- !query 5 schema
-struct<val:int,cate:string,sum(val) OVER (PARTITION BY cate ORDER BY val ASC NULLS FIRST RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING):bigint>
+struct<val:int,CAST(udf(cast(cate as string)) AS STRING):string,sum(val) OVER (PARTITION BY cate ORDER BY val ASC NULLS FIRST RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING):bigint>
 -- !query 5 output
 NULL   NULL    NULL
 3      NULL    3
@@ -99,10 +106,10 @@ NULL       a       NULL

 -- !query 6
-SELECT val_long, cate, sum(val_long) OVER(PARTITION BY cate ORDER BY val_long
+SELECT val_long, udf(cate), sum(val_long) OVER(PARTITION BY cate ORDER BY val_long
 RANGE BETWEEN CURRENT ROW AND<PHONE_NUMBER> FOLLOWING) FROM testData ORDER BY cate, val_long
 -- !query 6 schema
-struct<val_long:bigint,cate:string,sum(val_long) OVER (PARTITION BY cate ORDER BY val_long ASC NULLS FIRST RANGE BETWEEN CURRENT ROW AND<PHONE_NUMBER> FOLLOWING):bigint>
+struct<val_long:bigint,CAST(udf(cast(cate as string)) AS STRING):string,sum(val_long) OVER (PARTITION BY cate ORDER BY val_long ASC NULLS FIRST RANGE BETWEEN CURRENT ROW AND<PHONE_NUMBER> FOLLOWING):bigint>
 -- !query 6 output
 NULL   NULL    NULL
 1      NULL    1
@@ -116,10 +123,10 @@ NULL      b       NULL

 -- !query 7
-SELECT val_double, cate, sum(val_double) OVER(PARTITION BY cate ORDER BY val_double
+SELECT val_double, udf(cate), sum(val_double) OVER(PARTITION BY cate ORDER BY val_double
 RANGE BETWEEN CURRENT ROW AND 2.5 FOLLOWING) FROM testData ORDER BY cate, val_double
 -- !query 7 schema
-struct<val_double:double,cate:string,sum(val_double) OVER (PARTITION BY cate ORDER BY val_double ASC NULLS FIRST RANGE BETWEEN CURRENT ROW AND CAST(2.5 AS DOUBLE) FOLLOWING):double>
+struct<val_double:double,CAST(udf(cast(cate as string)) AS STRING):string,sum(val_double) OVER (PARTITION BY cate ORDER BY val_double ASC NULLS FIRST RANGE BETWEEN CURRENT ROW AND CAST(2.5 AS DOUBLE) FOLLOWING):double>
 -- !query 7 output
 NULL   NULL    NULL
 1.0    NULL    1.0
@@ -133,10 +140,10 @@ NULL      NULL    NULL

 -- !query 8
-SELECT val_date, cate, max(val_date) OVER(PARTITION BY cate ORDER BY val_date
+SELECT val_date, udf(cate), max(val_date) OVER(PARTITION BY cate ORDER BY val_date
 RANGE BETWEEN CURRENT ROW AND 2 FOLLOWING) FROM testData ORDER BY cate, val_date
 -- !query 8 schema
-struct<val_date:date,cate:string,max(val_date) OVER (PARTITION BY cate ORDER BY val_date ASC NULLS FIRST RANGE BETWEEN CURRENT ROW AND 2 FOLLOWING):date>
+struct<val_date:date,CAST(udf(cast(cate as string)) AS STRING):string,max(val_date) OVER (PARTITION BY cate ORDER BY val_date ASC NULLS FIRST RANGE BETWEEN CURRENT ROW AND 2 FOLLOWING):date>
 -- !query 8 output
 NULL   NULL    NULL
 2017-08-01     NULL    2017-08-01
@@ -150,11 +157,11 @@ NULL      NULL    NULL

 -- !query 9
-SELECT val_timestamp, cate, avg(val_timestamp) OVER(PARTITION BY cate ORDER BY val_timestamp
+SELECT val_timestamp, udf(cate), avg(val_timestamp) OVER(PARTITION BY cate ORDER BY val_timestamp
 RANGE BETWEEN CURRENT ROW AND interval 23 days 4 hours FOLLOWING) FROM testData
 ORDER BY cate, val_timestamp
 -- !query 9 schema
-struct<val_timestamp:timestamp,cate:string,avg(CAST(val_timestamp AS DOUBLE)) OVER (PARTITION BY cate ORDER BY val_timestamp ASC NULLS FIRST RANGE BETWEEN CURRENT ROW AND interval 3 weeks 2 days 4 hours FOLLOWING):double>
+struct<val_timestamp:timestamp,CAST(udf(cast(cate as string)) AS STRING):string,avg(CAST(val_timestamp AS DOUBLE)) OVER (PARTITION BY cate ORDER BY val_timestamp ASC NULLS FIRST RANGE BETWEEN CURRENT ROW AND interval 3 weeks 2 days 4 hours FOLLOWING):double>
 -- !query 9 output
 NULL   NULL    NULL
 2017-07-31 17:00:00    NULL    1.5015456E9
@@ -168,10 +175,10 @@ NULL      NULL    NULL

 -- !query 10
-SELECT val, cate, sum(val) OVER(PARTITION BY cate ORDER BY val DESC
+SELECT val, udf(cate), sum(val) OVER(PARTITION BY cate ORDER BY val DESC
 RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING) FROM testData ORDER BY cate, val
 -- !query 10 schema
-struct<val:int,cate:string,sum(val) OVER (PARTITION BY cate ORDER BY val DESC NULLS LAST RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING):bigint>
+struct<val:int,CAST(udf(cast(cate as string)) AS STRING):string,sum(val) OVER (PARTITION BY cate ORDER BY val DESC NULLS LAST RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING):bigint>
 -- !query 10 output
 NULL   NULL    NULL
 3      NULL    3
@@ -185,57 +192,57 @@ NULL      a       NULL

 -- !query 11
-SELECT val, cate, count(val) OVER(PARTITION BY cate
-ROWS BETWEEN UNBOUNDED FOLLOWING AND 1 FOLLOWING) FROM testData ORDER BY cate, val
+SELECT udf(val), cate, count(val) OVER(PARTITION BY cate
+ROWS BETWEEN UNBOUNDED FOLLOWING AND CAST(1 as int) FOLLOWING) FROM testData ORDER BY cate, val
 -- !query 11 schema
 struct<>
 -- !query 11 output
 org.apache.spark.sql.AnalysisException
-cannot resolve 'ROWS BETWEEN UNBOUNDED FOLLOWING AND 1 FOLLOWING' due to data type mismatch: Window frame upper bound '1' does not follow the lower bound 'unboundedfollowing$()'.; line 1 pos 33
+cannot resolve 'ROWS BETWEEN UNBOUNDED FOLLOWING AND CAST(1 AS INT) FOLLOWING' due to data type mismatch: Window frame upper bound 'cast(1 as int)' does not follow the lower bound 'unboundedfollowing$()'.; line 1 pos 38

 -- !query 12
-SELECT val, cate, count(val) OVER(PARTITION BY cate
-RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING) FROM testData ORDER BY cate, val
+SELECT udf(val), cate, count(val) OVER(PARTITION BY cate
+RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING) FROM testData ORDER BY cate, udf(val)
 -- !query 12 schema
 struct<>
 -- !query 12 output
 org.apache.spark.sql.AnalysisException
-cannot resolve '(PARTITION BY testdata.`cate` RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING)' due to data type mismatch: A range window frame cannot be used in an unordered window specification.; line 1 pos 33
+cannot resolve '(PARTITION BY testdata.`cate` RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING)' due to data type mismatch: A range window frame cannot be used in an unordered window specification.; line 1 pos 38

 -- !query 13
-SELECT val, cate, count(val) OVER(PARTITION BY cate ORDER BY val, cate
+SELECT udf(val), cate, count(val) OVER(PARTITION BY cate ORDER BY val, cate
 RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING) FROM testData ORDER BY cate, val
 -- !query 13 schema
 struct<>
 -- !query 13 output
 org.apache.spark.sql.AnalysisException
-cannot resolve '(PARTITION BY testdata.`cate` ORDER BY testdata.`val` ASC NULLS FIRST, testdata.`cate` ASC NULLS FIRST RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING)' due to data type mismatch: A range window frame with value boundaries cannot be used in a window specification with multiple order by expressions: val#x ASC NULLS FIRST,cate#x ASC NULLS FIRST; line 1 pos 33
+cannot resolve '(PARTITION BY testdata.`cate` ORDER BY testdata.`val` ASC NULLS FIRST, testdata.`cate` ASC NULLS FIRST RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING)' due to data type mismatch: A range window frame with value boundaries cannot be used in a window specification with multiple order by expressions: val#x ASC NULLS FIRST,cate#x ASC NULLS FIRST; line 1 pos 38

 -- !query 14
-SELECT val, cate, count(val) OVER(PARTITION BY cate ORDER BY current_timestamp
+SELECT udf(val), cate, count(val) OVER(PARTITION BY cate ORDER BY current_timestamp
 RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING) FROM testData ORDER BY cate, val
 -- !query 14 schema
 struct<>
 -- !query 14 output
 org.apache.spark.sql.AnalysisException
-cannot resolve '(PARTITION BY testdata.`cate` ORDER BY current_timestamp() ASC NULLS FIRST RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING)' due to data type mismatch: The data type 'timestamp' used in the order specification does not match the data type 'int' which is used in the range frame.; line 1 pos 33
+cannot resolve '(PARTITION BY testdata.`cate` ORDER BY current_timestamp() ASC NULLS FIRST RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING)' due to data type mismatch: The data type 'timestamp' used in the order specification does not match the data type 'int' which is used in the range frame.; line 1 pos 38

 -- !query 15
-SELECT val, cate, count(val) OVER(PARTITION BY cate ORDER BY val
+SELECT udf(val), cate, count(val) OVER(PARTITION BY cate ORDER BY val
 RANGE BETWEEN 1 FOLLOWING AND 1 PRECEDING) FROM testData ORDER BY cate, val
 -- !query 15 schema
 struct<>
 -- !query 15 output
 org.apache.spark.sql.AnalysisException
-cannot resolve 'RANGE BETWEEN 1 FOLLOWING AND 1 PRECEDING' due to data type mismatch: The lower bound of a window frame must be less than or equal to the upper bound; line 1 pos 33
+cannot resolve 'RANGE BETWEEN 1 FOLLOWING AND 1 PRECEDING' due to data type mismatch: The lower bound of a window frame must be less than or equal to the upper bound; line 1 pos 38

 -- !query 16
-SELECT val, cate, count(val) OVER(PARTITION BY cate ORDER BY val
+SELECT udf(val), cate, count(val) OVER(PARTITION BY cate ORDER BY val
 RANGE BETWEEN CURRENT ROW AND current_date PRECEDING) FROM testData ORDER BY cate, val
 -- !query 16 schema
 struct<>
@@ -245,13 +252,13 @@ org.apache.spark.sql.catalyst.parser.ParseException
 Frame bound value must be a literal.(line 2, pos 30)

 == SQL ==
-SELECT val, cate, count(val) OVER(PARTITION BY cate ORDER BY val
+SELECT udf(val), cate, count(val) OVER(PARTITION BY cate ORDER BY val
 RANGE BETWEEN CURRENT ROW AND current_date PRECEDING) FROM testData ORDER BY cate, val
 ------------------------------^^^

 -- !query 17
-SELECT val, cate,
+SELECT udf(val), cate,
 max(val) OVER w AS max,
 min(val) OVER w AS min,
 min(val) OVER w AS min,
@@ -286,7 +293,7 @@ FROM testData
 WINDOW w AS (PARTITION BY cate ORDER BY val)
 ORDER BY cate, val
 -- !query 17 schema
-struct<val:int,cate:string,max:int,min:int,min:int,count:bigint,sum:bigint,avg:double,stddev:double,first_value:int,first_value_ignore_null:int,first_value_contain_null:int,last_value:int,last_value_ignore_null:int,last_value_contain_null:int,rank:int,dense_rank:int,cume_dist:double,percent_rank:double,ntile:int,row_number:int,var_pop:double,var_samp:double,approx_count_distinct:bigint,covar_pop:double,corr:double,stddev_samp:double,stddev_pop:double,collect_list:array<int>,collect_set:array<int>,skewness:double,kurtosis:double>
+struct<CAST(udf(cast(val as string)) AS INT):int,cate:string,max:int,min:int,min:int,count:bigint,sum:bigint,avg:double,stddev:double,first_value:int,first_value_ignore_null:int,first_value_contain_null:int,last_value:int,last_value_ignore_null:int,last_value_contain_null:int,rank:int,dense_rank:int,cume_dist:double,percent_rank:double,ntile:int,row_number:int,var_pop:double,var_samp:double,approx_count_distinct:bigint,covar_pop:double,corr:double,stddev_samp:double,stddev_pop:double,collect_list:array<int>,collect_set:array<int>,skewness:double,kurtosis:double>
 -- !query 17 output
 NULL   NULL    NULL    NULL    NULL    0       NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL    1       1       0.5     0.0     1       1       NULL    NULL    0       NULL    NULL
    NULL    NULL    []      []      NULL    NULL
 3      NULL    3       3       3       1       3       3.0     NaN     NULL    3       NULL    3       3       3       2       2       1.0     1.0     2       2       0.0     NaN     1       0.0     NaN
     NaN     0.0     [3]     [3]     NaN     NaN
@@ -300,9 +307,9 @@ NULL        a       NULL    NULL    NULL    0       NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL    1       1       0.25    0.

 -- !query 18
-SELECT val, cate, avg(null) OVER(PARTITION BY cate ORDER BY val) FROM testData ORDER BY cate, val
+SELECT udf(val), cate, avg(null) OVER(PARTITION BY cate ORDER BY val) FROM testData ORDER BY cate, val
 -- !query 18 schema
-struct<val:int,cate:string,avg(CAST(NULL AS DOUBLE)) OVER (PARTITION BY cate ORDER BY val ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW):double>
+struct<CAST(udf(cast(val as string)) AS INT):int,cate:string,avg(CAST(NULL AS DOUBLE)) OVER (PARTITION BY cate ORDER BY val ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW):double>
 -- !query 18 output
 NULL   NULL    NULL
 3      NULL    NULL
@@ -316,18 +323,25 @@ NULL      a       NULL

 -- !query 19
-SELECT val, cate, row_number() OVER(PARTITION BY cate) FROM testData ORDER BY cate, val
+SELECT udf(val), cate, row_number() OVER(PARTITION BY cate ORDER BY val) FROM testData ORDER BY cate, udf(val)
 -- !query 19 schema
-struct<>
+struct<CAST(udf(cast(val as string)) AS INT):int,cate:string,row_number() OVER (PARTITION BY cate ORDER BY val ASC NULLS FIRST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW):int>
 -- !query 19 output
-org.apache.spark.sql.AnalysisException
-Window function row_number() requires window to be ordered, please add ORDER BY clause. For example SELECT row_number()(value_expr) OVER (PARTITION BY window_partition ORDER BY window_ordering) from table;
+NULL   NULL    1
+3      NULL    2
+NULL   a       1
+1      a       2
+1      a       3
+2      a       4
+1      b       1
+2      b       2
+3      b       3

 -- !query 20
-SELECT val, cate, sum(val) OVER(), avg(val) OVER() FROM testData ORDER BY cate, val
+SELECT udf(val), cate, sum(val) OVER(), avg(val) OVER() FROM testData ORDER BY cate, val
 -- !query 20 schema
-struct<val:int,cate:string,sum(CAST(val AS BIGINT)) OVER (ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING):bigint,avg(CAST(val AS BIGINT)) OVER (ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING):double>
+struct<CAST(udf(cast(val as string)) AS INT):int,cate:string,sum(CAST(val AS BIGINT)) OVER (ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING):bigint,avg(CAST(val AS BIGINT)) OVER (ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING):double>
 -- !query 20 output
 NULL   NULL    13      1.8571428571428572
 3      NULL    13      1.8571428571428572
@@ -341,7 +355,7 @@ NULL        a       13      1.8571428571428572

 -- !query 21
-SELECT val, cate,
+SELECT udf(val), cate,
 first_value(false) OVER w AS first_value,
 first_value(true, true) OVER w AS first_value_ignore_null,
 first_value(false, false) OVER w AS first_value_contain_null,
@@ -352,7 +366,7 @@ FROM testData
 WINDOW w AS ()
 ORDER BY cate, val
 -- !query 21 schema
-struct<val:int,cate:string,first_value:boolean,first_value_ignore_null:boolean,first_value_contain_null:boolean,last_value:boolean,last_value_ignore_null:boolean,last_value_contain_null:boolean>
+struct<CAST(udf(cast(val as string)) AS INT):int,cate:string,first_value:boolean,first_value_ignore_null:boolean,first_value_contain_null:boolean,last_value:boolean,last_value_ignore_null:boolean,last_value_contain_null:boolean>
 -- !query 21 output
 NULL   NULL    false   true    false   false   true    false
 3      NULL    false   true    false   false   true    false
@@ -366,12 +380,12 @@ NULL      a       false   true    false   false   true    false

 -- !query 22
-SELECT cate, sum(val) OVER (w)
+SELECT udf(cate), sum(val) OVER (w)
 FROM testData
 WHERE val is not null
 WINDOW w AS (PARTITION BY cate ORDER BY val)
 -- !query 22 schema
-struct<cate:string,sum(CAST(val AS BIGINT)) OVER (PARTITION BY cate ORDER BY val ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW):bigint>
+struct<CAST(udf(cast(cate as string)) AS STRING):string,sum(CAST(val AS BIGINT)) OVER (PARTITION BY cate ORDER BY val ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW):bigint>
 -- !query 22 output
 NULL   3
 a      2

How was this patch tested?
Tested as guided in SPARK-27921.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

Add to whitelist

add to whitelist

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12989/
Test PASSed.

Test build #107874 has started for PR 25195 at commit 195bcbd.

@younggyuchun, do you know why it works with UDFs?
 -- !query 19
-SELECT val, cate, row_number() OVER(PARTITION BY cate) FROM testData ORDER BY cate, val
+SELECT udf(val), cate, row_number() OVER(PARTITION BY cate ORDER BY val) FROM testData ORDER BY cate, udf(val)
 -- !query 19 schema
-struct<>
+struct<CAST(udf(cast(val as string)) AS INT):int,cate:string,row_number() OVER (PARTITION BY cate ORDER BY val ASC NULLS FIRST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW):int>
 -- !query 19 output
-org.apache.spark.sql.AnalysisException
-Window function row_number() requires window to be ordered, please add ORDER BY clause. For example SELECT row_number()(value_expr) OVER (PARTITION BY window_partition ORDER BY window_ordering) from table;
+NULL   NULL    1
+3      NULL    2
+NULL   a       1
+1      a       2
+1      a       3
+2      a       4
+1      b       1
+2      b       2
+3      b       3

@younggyuchun, do you know why it works with UDFs?
 -- !query 19
-SELECT val, cate, row_number() OVER(PARTITION BY cate) FROM testData ORDER BY cate, val
+SELECT udf(val), cate, row_number() OVER(PARTITION BY cate ORDER BY val) FROM testData ORDER BY cate, udf(val)
 -- !query 19 schema
-struct<>
+struct<CAST(udf(cast(val as string)) AS INT):int,cate:string,row_number() OVER (PARTITION BY cate ORDER BY val ASC NULLS FIRST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW):int>
 -- !query 19 output
-org.apache.spark.sql.AnalysisException
-Window function row_number() requires window to be ordered, please add ORDER BY clause. For example SELECT row_number()(value_expr) OVER (PARTITION BY window_partition ORDER BY window_ordering) from table;
+NULL   NULL    1
+3      NULL    2
+NULL   a       1
+1      a       2
+1      a       3
+2      a       4
+1      b       1
+2      b       2
+3      b       3

@HyukjinKwon The reason is that I specified the "ORDER BY val"?:
Before:
SELECT val, cate, row_number() OVER(PARTITION BY cate) FROM testData ORDER BY cate, val;
After:
SELECT udf(val), cate, row_number() OVER(PARTITION BY cate ORDER BY val) FROM testData ORDER BY cate, udf(val)
I guess we shouldn't specify "ORDER BY val" here right?

Can you elaborate why and confirm if Scala UDF works or not as well? We should better match the test cases with the original file if there isn't a specific reason to differentiate.
The point of why we're doing it is to detect non-working cases and file the issue (not making the test cases working with UDFs).

Test build #107874 has finished for PR 25195 at commit 195bcbd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107874/
Test PASSed.

Merged build finished. Test PASSed.

I thought we need to make the test cases work with UDFs. The original SQL spits out the following errors and I added 'ORDER BY' clause on it:
Window function row_number() requires window to be ordered, please add ORDER BY clause. For example SELECT row_number()(value_expr) OVER (PARTITION BY window_partition ORDER BY window_ordering) from table;
But I now understand the purpose of this work so I think we should put it back to the original one.

Yup, let's put it back.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13125/
Test PASSed.

Done. Thank you @HyukjinKwon

Test build #108015 has started for PR 25195 at commit d376e95.

Test build #108015 has finished for PR 25195 at commit d376e95.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108015/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13230/
Test PASSed.

Test build #108126 has started for PR 25195 at commit e6ce51a.

Test build #108126 has finished for PR 25195 at commit e6ce51a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class CannotReplaceMissingTableException(
case class MakeDate(year: Expression, month: Expression, day: Expression)
case class ReplaceTable(
case class ReplaceTableAsSelect(
case class ReplaceTableStatement(
case class ReplaceTableAsSelectStatement(
case class ReplaceTableExec(
case class AtomicReplaceTableExec(
case class AtomicCreateTableAsSelectExec(
case class ReplaceTableAsSelectExec(
case class AtomicReplaceTableAsSelectExec(

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108126/
Test PASSed.

Thanks, @younggyuchun. Do you mind updating PR description (the diff) too?

Thanks, @younggyuchun. Do you mind updating PR description (the diff) too?

Sure done

Merged to master.
Thanks @younggyuchun. Wellcome to Spark contributors :D.

My pleasure and thank you for reviewing my work. Have a great night!


Conic Sections

LO 13.5.1 requires conic sections but the math topic did not have a subtopic specifically about conics.  I have added this LO to the graph section temporarily but would adding a conics subtopic be better?

Yes I think adding a conics subtopic would be better


Feature request: Check for classes extending AnyVal that allocate

Would be nice to have something prove that a value class didn't allocate. I guess you could either check for situations outlined that are known to cause allocation, or some kind of inspection on class files to check that all you have are references to static method calls. Or something. I'd be happy to have a go implementing it if you were willing to give some guidance on how it should be done.

I was looking into similar stuff recently, and I concluded that it's pointless to check whether value classes allocate or not because the perf difference is not measurable.
See https://groups.google.com/forum/#!topic/scala-user/XdQnbcs2SRM
On our Scala projects I've changed our coding standards to suggest not adding "extends AnyVal" to extension method implicit classes because it's distracting and doesn't have any measurable benefit.
HTH

some kind of inspection on class files to check that all you have are references to static method calls

Scapegoat operates at the Scala language level; it doesn't currently have access to the compiled bytecode.
You'd have to pattern match on the "situations outlined that are known to cause allocation"; you couldn't check for the actual allocation bytecode calls as things currently stand.

The groups discussion you linked to says that it is benefiical, but usually
your other computations are order of magnitude slower so it doesn't matter.
Maybe worth having it, but off by default?
On 1 September 2015 at 13:52, Richard Bradley<EMAIL_ADDRESS>wrote:

some kind of inspection on class files to check that all you have are
references to static method calls
Scapegoat operates at the Scala language level; it doesn't currently have
access to the compiled bytecode.
You'd have to pattern match on the "situations outlined that are known to
cause allocation"; you couldn't check for the actual allocation bytecode
calls as things currently stand.
—
Reply to this email directly or view it on GitHub
https://github.com/sksamuel/scalac-scapegoat-plugin/issues/96#issuecomment-136706066
.

In my opinion, the distraction of worrying about whether your classes are declared properly for AnyVal allocation avoidance is more expensive than the tiny perf benefit. Also, hopefully future compilers will get better at optimising this sort of thing (not to mention the effects of the JITing optimiser), so it's not clear that any time spent optimising your code for this will survive future Scala changes.
I think there is a cost to having this kind of warning turned on: developers will spend time fixing up their classes / traits and their usage to avoid AnyVal allocations.
I can't see any harm in having it available but off by default, if other users have different opinions on the cost / benefits here.

I agree off by default. But there is a use for it so i think it should be
an option?
On 2 Sep 2015 09:39, "Richard Bradley"<EMAIL_ADDRESS>wrote:

In my opinion, the distraction of worrying about whether your classes are
declared properly for AnyVal allocation avoidance is more expensive than
the tiny perf benefit. Also, hopefully future compilers will get better at
optimising this sort of thing (not to mention the effects of the JITing
optimiser), so it's not clear that any time spent optimising your code for
this will survive future Scala changes.
I think there is a cost to having this kind of warning turned on:
developers will spend time fixing up their classes / traits and their usage
to avoid AnyVal allocations.
I can't see any harm in having it available but off by default, if other
users have different opinions on the cost / benefits here.
—
Reply to this email directly or view it on GitHub
https://github.com/sksamuel/scalac-scapegoat-plugin/issues/96#issuecomment-136977950
.

Sure, I would have no objections to that (not that you need my permission anyway :-)


Distribute packages as flatpaks

Hello,
I don't know how hard it could be, if you have the resources to do it or if it even makes since, but I wanted to ask this for a long time. Is it possible to package everything as flatpaks?
Flatpak packages can be safely set to auto-update with a systemd timer or via gnome-software. They would also work in all distributions.
One downside is that it may increase the size on disk and R will probably need to be installed and run as a flatpak too.
Please don't waste time on this. Close it immediately if it makes no since to you.

Thanks for the suggestion, but it's too much trouble and I don't think it makes sense for R packages. :)


Keyboard gets dismissed when first character is typed in

Describe the bug
I have the otp_pin_field with autofocus enabled, on an AlertDialog in my app. On android, when I start typing and enter the first digit of my OTP, the keyboard closes itself.
To Reproduce
Visit the page with the pin-field
Focus it (or even if it is auto focused)
Type one character
The keyboard gets dismissed immediately.
When you click on the second box and continue typing, that part doesn't have this issue
Expected behavior
The second box should get focused as usual
Screenshots
N/A
Smartphone (please complete the following information):

Device: Android Phone
OS: Xiaomi HyperOs <IP_ADDRESS> - android 14

Additional context
We test on four different phones (with different android versions). only this one has this issue.
There is a "autofill from SMS" button appearing above the keyboard (only on the first character is focused) would that cause this issue?

Hi @Jayasubin ,
Thank you all for your feedback and for reporting the issues. I’m happy to inform you that the following issues have been resolved:
SMS hint autofill issue when autoFillEnabled == false has been fixed.
Autofocus issue on the web is now resolved, and autofocus works as expected.
New hint text feature has been added, allowing you to customize showHintText, hintText, and hintTextColor.
You can now update to version 1.3.0 of the package, available here: otp_pin_field v1.3.0.
Thanks again for your contributions and patience! Let me know if you encounter any further issues.
Best regards,

Hi  @Jayasubin ,
We’ve addressed the keyboard dismissal issue for now and are marking this as resolved. If you encounter any similar issues or if the problem persists, feel free to reach out, and we’ll be happy to investigate further.
Thank you for your patience and for helping us improve the package!

Hi @Jayasubin ,
We’ve addressed the keyboard dismissal issue for now and are marking this as resolved. If you encounter any similar issues or if the problem persists, feel free to reach out, and we’ll be happy to investigate further.
Thank you for your patience and for helping us improve the package!


Python 3.13 Support

Pillow added support for Python 3.13 in v11.0, but livekit-agents[images] requires Pillow v10. Manually pinning the resolution at 11 doesn't seem to cause issues for me, but I'm just using the openai package without any actual image functionality.

@montasaurus I tried updating earlier this week and it looked like silero VAD wasn't supporting 3.13 yet due to onxy support not being there yet:
https://github.com/onnx/onnx/issues/6339

Looks like a fix for https://github.com/onnx/onnx/issues/6339 was was closed. onnx 1.18 will have python 3.13 support.


Add GlobalSemaphore, SubDevice pybindings

Ticket
Link to Github Issue
Problem description
Need to add pybindings for some new metal features.
What's changed
Added pybindings for some new metal features.
Checklist

[x] Post commit CI passes (https://github.com/tenstorrent/tt-metal/actions/runs/12092222303)
[ ] Blackhole Post commit (if applicable)
[ ] Model regression CI testing passes (if applicable)
[ ] Device performance regression CI testing passes (if applicable)
[x] New/Existing tests provide coverage for changes

Thanks @tt-aho, looks good!


Improve changelog updates

The current process for updating the changelog has some issues.

Updates in multiple PRs cause merge conflicts.
Adding links to PRs requires another commit after opening a PR.

It works ok with a low volume of PRs, but merge conflicts become more of a problem with more PRs.
One alternative would be to update the changelog only on releases. However,

PyPI releases are so infrequent that attempting to gather a list of changes and create a changelog at release time would be difficult.
We use the latest git revision instead of the latest PyPI release in gnomad_qc so it's useful to have the changelog always up to date instead of only updated on PyPI releases.

More frequent releases could resolve those issues.
Another option would be storing the changelog outside of the repository content. https://github.com/marketplace/actions/release-drafter or something similar looks promising. Each release has its changelog and a draft release has a changelog for the latest development version. The changelog file could also be updated from the release notes on each release.

Perhaps the easiest thing to do is add changelog entries just before merging instead of when creating the PR.

Agree that is a fine way to do it, but I do often forget, so I guess it's good to also encourage reviewers to hold on an approval without the changelog addition when it is needed so there are at least two checks on it.

Good point. We could also add a PR check that blocks merging unless the PR modifies the changelog or has something like "Changelog not needed" in the description or a comment.


Move this repository to the zarr-org

I think having this reference data is fairly useful. However, I think it's a bit hard to discover right now, so I think it would be good to move it to the zarr org to improve visibility (and also encourage more contributions ;)).
Any thoughts on this @joshmoore @alimanfoo @jakirham.

:+1: cc: @jakirkham

Fine by me

Ok, done :).

Ah great was going to ask if there was anything we needed to do to help with this. If anything else comes up, please let us know 🙂


fix: should return the whole message if the OpenAI return the content and the function call/tools at the same time.

Should return the whole message if the OpenAI return the content and the function call/tools at the same time
The return json will be like this
 "choices": [     {       "finish_reason": "function_call",       "index": 0,       "logprobs": null,       "message": {         "content": "Perfect! I'll go ahead and deal with your request, Just a moment while I confirm that for you.",         "function_call": {           "arguments": "{\"start_at\":\"2024-05-16T15:30:00Z\",\"end_at\":\"2024-05-16T16:00:00Z\",\"time_zone\":\"UTC\"}",           "name": "arrange_meeting"         },         "role": "assistant"       }     }   ] }

Thanks @qwlong!


VRApi support + refactoring

List of notable changes:

attempted to separate out the oculus-specifics at the java level (see VrapiActivityHandler)
the framebufferobject class is ported over from the sample but the depth handling is ported from vrappfw
broke down the C++ activity to few pieces, the main class is no more a template
added few gl-specific logging macros and put them to use
should fall back to monoscopic if oculus is not present but have not tested that; need to find a proper device
brings up the oculus confirm-to-exit and global menu screens on back and long-press back; we may choose not to do it now
decided to use the "showLoadingIcon" config option to disable/enable our logo; good idea/bad idea?; it is not applicable currently to the other branch and is only available to vrappfw apps as the loading icon is implemented in that library
updated the xml parser to recognize xyz-parm and xyz-param tags so it can take config files from any branch

Fixed known issues:

force switching to monoscopic mode
model-viewer rendering incorrectly models due to not picking the depth texture format correctly

Known issues:

vrtop; docking/undocking frequently leads to crashes in vrapi
gvrsimplegallery; shows black screen after the logo

-1.  Can't explain why, but the model-viewer sample app is broken.  Looks fine intially, but when you click on each of the models, they are each missing parts.  like maybe the first mesh is being rendered??  can't say.
i haven't tested with a non-VR phone yet.  i used to test with my S5, but the company installed KNOX on it and disabled USB debugging :(.  I'll have to find another device.

There is something wrong with the depth texture. Working on it.

I should have gone through each and every sample. Will do so after I fix this problem.

Fixed the model-viewer; root cause was not picking the right depth texture format.

Found a problem with gvrsimplegallery; will keep updating the list of known issues in the first comment from now on to minimize spamming.

if you've tested against all the samples, then you can also test against: https://github.com/dbrant/GearVR_planetarium

cool, was just able to run in monoscopic mode on a nexus 5.  first time that's been possible in a while.

GearVR_planetarium works fine. Couldn't spot any difference vs master.

 The effect described is similar to what I am seeing. If I don't move the phone at all I can still make out the darkest areas.

I've sunk enough time in this. My suggestion is to deal with this separately from this pull request (as time permits) and first update to the latest Vuforia to hopefully pick up bug fixes. If it still doesn't work, continue the investigation.

Vuforia sample fixed too. No known to me issues with this pull request anymore.

anyone else have comments on this?

Tested on S6 edge+. Mono and fullscreen work. But it still requires developer mode. Is there a way to remove this requirement? In mono mode, we don't need timewarp because it is for developer only.

I think on devices without VR service mono will just work.

Btw, we can finally debug Java code properly with this change in.

I have no problem for merging this patch if others agree.

needs rebase

Rebased; merged a bunch of stuff in renderer.cpp. Shadowmap sample and few others work fine.


Add Update Drops Entitlements

Hey,
I'm currently working with Drops. I figured out, that update drops are missed.

Outdated


distutils is deprecated in Python 3.10

Ref : https://www.python.org/dev/peps/pep-0632/#migration-advice
setup from setuptools can be used.
https://github.com/ram-jayapalan/filesplit/blob/591258a82e9b86e612e4d69bdc11ad0ad00b50ed/setup.py#L1

The latest version is using setuptools instead of distutils. Thanks for bringing this up.


add OVH "cloud instances" maybe ?

There is an other kind of "vps" here: https://www.ovh.com/us/public-cloud/instances/prices/
with an other manager, openstack api access, with recovery mode, allowing changing the kernel, "projects" to group instances, additionnal storage :)

I cover this in general in the issue #22, please go there if you have any though or idea to share. Thanks!


If possible could you fix the index.html so you can upload files and they actually load the roms

please

You seem have to been doing it incorrectly again. Go to Issue: #37, right-click the blue "Here" link, and it should save as "webretro-standalone.html" I have personally tested it, and it works.

oh ok

thanks @HaxsMaster


enable guest_hook_path in remote config

@snir911 this needs to be updated as well - https://github.com/openshift/sandboxed-containers-operator/blob/devel/config/peerpods/mc-40-kata-remote-config.yaml
ex
butane -d ./local mc-remote-butane.conf > mc-40-kata-remote-config.yaml

This is where it gets used - https://github.com/openshift/sandboxed-containers-operator/blob/devel/controllers/openshift_controller.go#L2155

@snir911 you'll need to create a cherry-pick of this into release-1.6 branch
git cherry-pick -sx ..


Add Sublime editor integration to readme

This PR adds a link to the SublimeLinter-contrib-tlint package on packagecontrol.io to the readme under Editor Integrations.

🎉


fix(tooltip): prevent tooltip to create horizontal scrollbars

Description
When the window is resized, occasionally the tooltip creates horizontal or vertical scrolling. This PR sets the left and top attributes to 0 when the tooltip is closed.
As an alternative, we could set display: none when hidden instead of visibility: hidden.
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?
Tested locally by use
Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules
[x] I have checked my code and corrected any misspellings

Can we drop this one in favor of #683? @multiwebinc @tulup-conner

@rluders I'm off for the weekend. I'll take a look at it next week unless someone else wants to. But I think they are probably complimantary because I didn't alter the styles much in #683.

@rluders I'm off for the weekend. I'll take a look at it next week unless someone else wants to. But I think they are probably complimantary because I didn't alter the styles much in #683.

Not being pushy on this one, but... any update?

@rluders Sorry for the delay. Yes, I see that this can just be closed now in favor of #683.


Nxlog dns logs solution

Package a solution for normalizing NXLog-parsed DNS Server analytical events using the DNS ASim schema
Proposed Changes

Provide a large enough sample of DNS Server analytical events to demo aggregation queries
Create a Kusto function for parsing DNS server analytical events using the DNS ASim schema
Write some aggregation queries to show how to create charts and graphs

@techwriter-dev : i am on vacation and it would probably get to it only when i am back.

@aprakash13 Please review the analytical rules.

Since the parser is an ASIM compliant parser, we will add it to the ASIM DNS package. To accelerate, I will do this. In addition, the connector json file needs a few updates which I will list in my review.


chore(main): release 0.6.0

:robot: I have created a release beep boop
0.6.0 (2022-10-02)
Features

add custom notification summary and body (#138) (3012e90)
add Google Analytics (#139) (5f3ff72)
refactor notification module (#136) (1c3dead)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/kucera-lukas/rusty-battery/releases/tag/v0.6.0 :sunflower:


Can I rework docs using Nuxt UI Pro

🆒 Your use case
Hello,
Current docs is build with docus which is currently deprecated. Nitro and H3 will use Nuxt UI Pro in a near future. Can I do the same for the current lib?
🔍 Alternatives you've considered
No response
ℹ Additional info
No response

Oh, I would love that! ❤

@Barbapapazes and @danielroe - had there been any progress with this. Don't mind doing it. Lmk

@Barbapapazes and @danielroe - had there been any progress with this. Don't mind doing it. Lmk

Hey @moshetanze, no progress since I was waiting for unjs/undocs but now it's released, we can move to it! 😄 If you want to make the PR, feel free!

great will try open one soon!


remove hardcode prewarm nodejs:6 allow nodejs:default

currently working on nodejs:8 this allow me to select which is prewarm based on ansible setting.

The indirection is dangerous here since you can end up affecting the setting for a majority of actions unintentially. Did you consider factoring out the prewarm kind(s) to ansible instead?

@rabbah yes I considered moving to ansible.
Was thinking maybe an explicit kind selection, and not assume default for example a top level runtimeManifest property:
runtimesManifest:
  prewarmKind:"nodejs:6"

what you think?

Instead of a new top level property you could add a property prewarm as part of the kind declaration. There is a nuance you also need to potentially declare the memory bound as well.


Mission control abort behavior

Description
This PR adds an abort behavior instead of using attitude behavior for aborting routines. This change was implemented because
the attitude behavior doesn't check if the thruster velocity is 0 RPM to complete the mission.
The changes introduces are:

New abort behavior.
The tests now check if the abort behavior is complete, (e.g. fins are set to surface and thruster velocity is 0 rpm)
Add altitude heading time out test.

Type of change
Please mark options that are relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] The code builds clean without any errors or warnings
[x] Linter tests are passing

How To Test
rostest mission_control test_if_mission_control_aborts_when_behavior_returns_time_out_failure.test

rostest mission_control test_if_mission_control_aborts_when_health_monitor_reports_fault.test

To summarize my thoughts, I think we don't need a custom behavior. A proper abort tree with generic behaviors should be sufficient. But if we do end up with a custom behavior, I'd seize it i.e. drop unnecessary parameterizations, make it feature complete.


Web - Account Settings-'Domain Card tests - DO NOT DELETE' is not an option

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: v9.0.20-0
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail: https://expensify.testrail.io/index.php?/tests/view/4859388
Email or phone of affected tester (no customers<EMAIL_ADDRESS>Issue reported by: Applause Internal Team
Action Performed:

In OldDot, login with<EMAIL_ADDRESS>Navigate to domain settings
In company cards - click on import card
Select Discover Bank
Enter username Expensifydemo / Password: anything
After the card has been imported - click on the settings tab in the discover card
Select the "Domain Card tests - DO NOT DELETE" as the preferred workspace

Expected Result:
The Domain Card Tests - DO NOT DELETE workspace is available for selection as the preferred workspace.
Actual Result:
The Domain Card Tests - DO NOT DELETE workspace is not displayed in the 'Preferred workspace' dropdown list.
Workaround:
Unknown
Platforms:

[ ] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
https://github.com/user-attachments/assets/bf9b045d-0bf1-458e-926e-3e1efd33f14a

View all open jobs on GitHub

@muttmuure  FYI I haven't added the External label as I wasn't 100% sure about this issue. Please take a look and add the label if you agree it's a bug and can be handled by external contributors

Can't reproduce.

Remember: you need to enable Scheduled Submit for options to appear in that preferred workspace dropdown


Add Support for mDNS

With this patch, the CO2Ampel will register a _co2ampel mdns service
On a mac, run
dns-sd -B _co2ampel._tcp
to see which instances are around. To learn more about a particular instance, run
dns-sd -L my-co2ampel  _co2ampel._tcp
in this way you learn on which port you can query the ampel ... it will now return the sensor data in json form.
This PR is related to issue #26

@AlfMueller, what do you think? I like the idea of having MDNS support for the standard firmware.

If mdns support became part of the standard firmware I might also be tempted to finish that IOS/Android App I have written which allows to set an alarm

Regarding network configuration, the behavior when no wlan network is available may have to be verified ...

Any plans to merge this to stable?


Added CMDLET to delete workitem

PR Summary
I added a new cmdlet to delete work items by IDs. I
So the following things were added:

cmdlet Remove-VSTeamWiql

added also the switch parameter "Destroy" to also permanently delete the work item instead on putting it to the recycle bin

added a format file for the cmdelet
added a type file for the cmdlet

PR Checklist

[X] Write Help
[X] Write Unit Test
[X] Update VSTeam.psd1
[X] Update CHANGELOG.md

@DarqueWarrior I accidentally merged the other topic branch into this one in the beginning. So it shows the history of the not yet merged PR and also the comments above. I don't know if that will be a problem.

Make sure you run
.\Build-Module.ps1 -analyzeScript -ipmo

And clear any messages.
I am a big unclear why we need _applyTypesToWorkItemDeleted and can't use _applyTypesToWorkItem

I will take care of these changes.

Just for the explanation. I created _applyTypesToWorkItemDeleted because the return JSON object is not consistent with a normal work item object. Strange enough, even the property names are different.
for example, the name of the work item when you query it is "title". But the object returned from the API when you delete it has the property named "name" and not "title"
This is for several properties. So I thought a new method is needed because I also created a new format.
Unless you tell me, that this can be done differently, then I am happy to know how it can be done simpler.

Can you think of a scenario where you are going to save the returned deleted item? In all the rest of the remove calls we don't return the object. I think to stay consistent I will just not return anything here either.  I am almost done with all the changes. There is nothing for you to do at this point.

Changes merged into PR #217


Settings search stopped working for me

VSCode Version: Code - Insiders 1.9.0-insider (2cdc2b3da8f0fd5322b48747241584cf9dd98f8f, 2017-01-13T07:03:18.620Z)
OS Version: Linux x64 4.8.0-32-generic
Extensions:

Extension
Author
Version

EditorConfig
EditorConfig
0.3.4

lorem-ipsum
Tyriar
1.0.0

theme-glacier
Tyriar
0.0.10

vscode-svgviewer
cssho
1.3.0

python
donjayamanne
0.5.5

gitlens
eamodio
1.4.1

tslint
eg2
0.8.1

git-project-manager
felipecaputo
1.0.1

Theme-Wombat
gerane
0.0.2

md-navigate
jrieken
0.0.1

Go
lukehoban
0.6.50

vscode-scss
mrmlnc
0.6.0

csharp
ms-vscode
1.6.2

seti-icons
qinjia
0.1.3

java
redhat
0.0.8

annotator
ryu1kn
0.10.1

In console:
shell.ts:432t.toLowerCase is not a function: TypeError: t.toLowerCase is not a function
    at Object.a [as matchesContiguousSubString] (file:////usr/share/code-insiders/resources/app/out/vs/workbench/electron-browser/workbench.main.js:30:9716)
    at f (file:////usr/share/code-insiders/resources/app/out/vs/workbench/electron-browser/workbench.main.js:90:3319)
    at t._findMatchesInSetting (file:////usr/share/code-insiders/resources/app/out/vs/workbench/electron-browser/workbench.main.js:90:3541)
    at t.doFilterSettings (file:////usr/share/code-insiders/resources/app/out/vs/workbench/electron-browser/workbench.main.js:90:2054)
    at t.filterSettings (file:////usr/share/code-insiders/resources/app/out/vs/workbench/electron-browser/workbench.main.js:90:8764)
    at t.filterPreferences (file:////usr/share/code-insiders/resources/app/out/vs/workbench/electron-browser/workbench.main.js:114:8733)
    at t.filterPreferences (file:////usr/share/code-insiders/resources/app/out/vs/workbench/electron-browser/workbench.main.js:114:1765)
    at file:////usr/share/code-insiders/resources/app/out/vs/workbench/electron-browser/workbench.main.js:114:281
    at e.invoke (file:////usr/share/code-insiders/resources/app/out/vs/workbench/electron-browser/workbench.main.js:30:26461)
    at e.fire (file:////usr/share/code-insiders/resources/app/out/vs/workbench/electron-browser/workbench.main.js:30:29773)

I was not expecting null values in default settings and there was a new setting introduced yesterday with null value. Fixed the search to be null value aware.


Peer to peer - code review 2 day

It's better just to use one h1 for semantic purposes.
You can use flex-wrap property to set the lists's value to wrap in order to have just one list on two lines..
It can help you to use variables for properties that you'll need to use often in your code such as flexbox's properties.
You could use another container to organize social media's icons list inside of it.}
Adding some space in the form will make to look more like the template.

Hello, Edgard! Your code looks great! 👍
I believe there are some minor issues, that might be fixed, to make your code look even better!

Many CSS lines here are repeating and unnecessary, and it is against the DRY rules of CSS. kindly consider avoiding using unnecessary CSS code.
Try to make your project have a responsive design up until the desktop screen starts from the 768px breakpoint. as you have given a fixed width of 375px in your header (.welcome-container) and headline section (.greetingsandsocial), this makes your design unresponsive. Kindly make sure that your design is responsive. 😄


Doesn't work with node_modules

Not sure why, but this does not work when watching something in node_modules ... Any idea why?

@kewp The Vite server watcher skips node_modules/ by default. They have docs on how to enable this, but it still doesn't seem to work for me. handleFileChange is called for local stuff, but no changes in node_modules. Looking at the source code for this plugin, I don't see anything that would get in the way, but who knows. Would love to get this working!

After a bit more investigation I think I figured it out! 🎉
Because vite-plugin-restart uses path.posix.join(root, i)) when initializing the restart/reload globs, it appends the config.root to the path passed in. This means any paths passed in can be relative to the project/config root with no issue (normally). It does not check if the path is absolute, so relative paths are required for this plugin's config
I put this relative path in a variable and used it for both ViteRestart and the Vite config mentioned above. This was my mistake. Vite requires the server.watch.ignored and optimizeDeps.exclude paths to be absolute.
Working example:
import { defineConfig } from "vite";
import react from "@vitejs/plugin-react";
import path from "path";
import ViteRestart from "vite-plugin-restart";

// Change to any package in your node_modules
const myModule = "node_modules/pick-any-module/*";

export default defineConfig({
  plugins: [
    react(),
    ViteRestart({
      reload: [myModule],
    }),
  ],
  server: {
    watch: {
      ignored: [`!${path.join(process.cwd(), myModule)}`],
    },
  },
  optimizeDeps: {
    exclude: [path.join(process.cwd(), myModule)],
  },
});


[CU-2vtgc9d] Fix replay validation service failure if player mapping fails.

Task linked: CU-2vtgc9d Submitting replays that contain users not in the database breaks


Error if %s is not present in prompt string format

Hey,
I was trying to hide the prompt by setting an empty string to nbsphinx_input_prompt. This results in a TypeError on:
https://github.com/spatialaudio/nbsphinx/blob/77f305b071b2045444602167510c1184dd3e6e21/src/nbsphinx.py#L1065
The blame is the % format. Would you consider switching to format asking user to set the string as "In [{}]" rather than "In [%s]". This will be more robust, see:
In [6]: "hello" % "dummy"
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-a2bd27e7e4d9> in <module>
----> 1 "hello" % "dummy"

TypeError: not all arguments converted during string formatting

In [7]: "hello".format("dummy")
Out[7]: 'hello'

In [8]: "hello {}".format(12)
Out[8]: 'hello 12'

Another possibility would be to test for %s.
Note: I use that workaround for my initial purpose.

Thanks for the suggestion!
This would be a breaking change though, and would make many users angry, so I would need a really good reason to do it.

I was trying to hide the prompt by setting an empty string to nbsphinx_input_prompt.

I wouldn't recommend hiding the prompts in the first place, but if you insist, you still shouldn't do it with nbsphinx_input_prompt.
For a more reliable method, see https://nbsphinx.readthedocs.io/en/latest/custom-css.html.

This would be a breaking change though, and would make many users angry, so I would need a really good reason to do it.

Ya I thought so. This is why I suggested testing for %s in the string to make it more robust without breaking change 😉
Thanks for the additional information. You rock at supporting users 🥇

Thanks!
I don't fully understand your proposal.
Do you mean nbsphinx should check whether nbsphinx_input_prompt contains '%s' and then, depending on that use either % or .format()?
Can you please elaborate?

Thanks, #446 clears up what you mean!


?? Logstash runs only on Windows machines. ??

This is an odd statement on the page. Logstash is also available for Linux and MacOS. In fact, the page with detailed instructions covers Linux...

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 742ec139-d425-cc82-67d0-0cc886253ac4
Version Independent ID: df24c1ac-aa86-288a-f52d-ff0889586717
Content: Microsoft Sentinel migration: Export ArcSight data to target platform
Content Source: articles/sentinel/migration-arcsight-historical-data.md
Service: microsoft-sentinel
Sub-service: microsoft-sentinel
GitHub Login: @limwainstein
Microsoft Alias: lwainstein

@andrePKI
Thanks for your feedback! We will investigate and update as appropriate.

Hi @andrePKI
Can you please provide the doc link where you want changes to be made. I am unable to find the line that you need changes was present in the given doc link.

Hi, I somehow mixed up my comments for two pages. This one was meant for this page: https://learn.microsoft.com/en-us/azure/sentinel/migration-ingestion-tool


CaseTrigger: execution of AfterInsert caused by: System.SObjectException: Invalid field Account for Case (dlrs)

Hi, I'm running into this error when I aggregate Cases to Account.
CaseTrigger: execution of AfterInsert caused by: System.SObjectException: Invalid field Account for Case (dlrs)
The error occurs for System Administrators as well as end users. This is in a Summer 19 sandbox
Here's what I have set:
Parent Object: Account
Child Object: Case
Relationship Field: AccountId
Relationship Criteria: RecordType.name = "Support" and isclosed = False
Field to Aggregate: Id
Aggregate Operation: Count
Any ideas?

It looks like I have the same problem, trying to count how many Cases a Contact has:
Parent Object: Contact
Child Object: Case
Relationship Field:  Contact
Apex trigger dlrs_caseTrigger caused an unexpected exception, contact your administrator: dlrs_caseTrigger: execution of AfterInsert caused by: System.SObjectException: Invalid field Contact for Case: Class.dlrs.RollupService.handleRollups: line 908, column 1

This could have been an error in the platform at the time. Let me know if your still getting it.


clipping of rotated container inside stage bounds

Hi - I posted a question on stack overflow about my canvas graphics being clipped when I rotate the container they are in, and the response made me think that perhaps there is an issue within easeljs.
The stack overflow issue is here, but the gist is that I have a stage equal to browser dimensions, a container inside the stage that I rotate over time, and graphics within the container that move around on their own - but they are getting clipped, almost as if the stage bounds are acting as a rotating mask, beyond which the container graphics are not showing up.
Any ideas on how to fix this?

Why was this closed? Did you find the issue?


adios_pair c2s/s2c

@cwsmith As I'm setting up the rdv usage in the coupler. I came across the following question for you. I'm wondering if the client and server are actually different from the adios perspective.
My current workflow is to add a getter to Redev to get the process type (client/server) and then use this to determine which of c2s and s2c are the sending and receiving comms.
Now, if the server and client are essentially the same on the rendezvous side it may make sense to refactor those to be called something like sending comm and receiving comm (since rdv knows which is which upon construction). If we do this, then the user no longer needs to get access to the rdv process type.

The only difference between the client and server from the adios perspective is the read/write mode of each connection.
The idea is to rename the variables in the CommPair here?
https://github.com/SCOREC/redev/blob/655bfeebb93ec31503a9d0beaa11cae2f374f595/redev.h#L305-L318

closed by #15


[Membership Request] RealVishy

Promotion
I'd like to nominate @RealVishy as an Aurora maintainer.  They have been sending in pull requests, marking down known issues, and supporting others in the Discord. As a minimum, they should have permission to review other pull requests.  If you are an Aurora or Universal Blue member/approver, please acknowledge this issue by posting a +1 if you are in favor of this.
Want to be an Aurora maintainer / active contributor?
There is also this issue/announcement in regards to scouting for other Aurora maintainers. If you are interested in helping out with the project, then your contributions will be noticed and you may be granted maintainer status. We will be having similar membership posts for other active contributors soon.

+1!

+1!  Very helpful in the Discord and created various PRs

+1

+1


Botframework publishing review issue

System Information
SDK Language: NODE
SDK Version: 3.8.4
Development Environment: server
Issue Description
Hi
I have an issue with publishing my bot.
I am trying to submit for review,but system says that my server sent 403 error.
I have looked at my server logs and saw that microsoft sends request and within body of request there is property  channelId: "test", but ChatConnector of botbuilder package don't accept such channelId
and I got this error

ChatConnector: receive - endorsements validation failure. channelId in req.body: test didn't match the endorsements: bing,directline,email,facebook,groupme,kik,skype,slack,sms,telegram,webchat.

this is request from dev.botframework.com
{ type: 'ping',
  id: '6rQ0LAG7X2Q',
  timestamp: '0001-01-01T00:00:00',
  serviceUrl: 'https://dev.botframework.com/',
  channelId: 'test',
  from: { id: 'portal' },
  conversation: { id: 'ping' },
  recipient: { id: 'bot' } }

What I have missed or how can I submit my app fro review?

@Cartman720 Can you post a screenshot of the error you are seeing in the Bot Portal?

Sure, here it is.

Relates to #2979

There's a problem with the publishing workflow. We are actively working to fix the issue.

@Cartman720 Should be fixed now. Can you verify if its working for you?

Fix is deployed

Hi @nwhitmont
I think it works!
Thanks!


Complete bin.zip distribution for download

Suggestion:
Earlier versions of UCanAccess made a complete bin.zip file available for download from SourceForge, e.g., UCanAccess-5.0.1.bin.zip contained
UCanAccess-5.0.1.bin
    lib
        commons-lang3-3.8.1.jar
        commons-logging-1.2.jar
        hsqldb-2.5.0.jar
        jackcess-3.0.1.jar
    licenses
        apache-license-2.0.txt
        hsqldb_lic.txt
        hypersonic_lic.txt
        lgpl-2.1.txt
        licenses.txt
    loader
        ucanload.jar
    console.bat
    console.sh
    copyright.txt
    ucanaccess-5.0.1.jar
    version.txt

This was very convenient for people who wanted to use UCanAccess without necessarily having a Java project that could take advantage of Maven. One example would be a Python project that could use UCanAccess along with JayDeBeApi as described here. The console.bat and console.sh utilities were also very useful for troubleshooting.

Thanks for the suggestions, @gordthompson.
This should not be too much effort I believe.
I will look into it, or if you have a pull request for this, please send it in.

I've added an assembly to create the binary distribution and script files.
@gordthompson please have a look

I think I understand what it is doing, although it's all new to me. Will it include loader/ucanload.jar ?

Unpacking zip:
unzip ucanaccess-5.1.2-SNAPSHOT-bin.zip
rm ucanaccess-5.1.2-SNAPSHOT-bin.zip
tree

Currently looking like this:
.
├── console.bat
├── console.sh
├── lib
│   ├── hsqldb-2.7.1.jar
│   └── jackcess-5.0.0-SNAPSHOT.jar
├── LICENSE.txt
└── ucanaccess-5.1.2-SNAPSHOT.jar

Will it include loader/ucanload.jar?

Not yet at least. What's this for and still needed? Can't remember seeing those at time of fork.

Currently looking like this:

Does the package no longer depend on commons-lang3 and commons-logging?

Will it include loader/ucanload.jar?

Not yet at least. What's this for and still needed? Can't remember seeing those at time of fork.

It's for configuring other apps, e.g.,
https://askubuntu.com/a/519571
https://stackoverflow.com/a/36608661/2144390

Does the package no longer depend on commons-lang3 and commons-logging?

Correct, they're out. UCanAccess now utilizes Java Platform/System Logger, which is part of the JDK since Java 9.
Can you tell whether UcanloadDriver in ucanload.jar is still required today?

It looks like Jackcess still depends on commons-lang3 and commons-logging, though
https://github.com/jahlborn/jackcess/blob/22bf8a8642c84fc500f5c66ef59de1f7211f93a3/pom.xml#L247C1-L257C1

Can you tell whether UcanloadDriver in ucanload.jar is still required today?

It would still be useful for the cases I cited above. It discovers the specific versions of the dependencies so a version change for one of those dependencies won't break the configuration of the consuming app (e.g., LibreOffice Base).

Hi Gord,

It looks like Jackcess still depends on commons-lang3 and commons-logging, though

Not for much longer. I've forked jackcess - it has no dependencies other than Apache POI
I'll see whether I can play with LibreOffice and UCanAcces.

Hi @gordthompson,
good news, I am able to read/write Microsoft Access databases from LibreOffice Base using the latest UCanAccess snapshot.
Simply add three jars to the JVM classpath in LibreOffice. Screenshot:

To make this a little more straight-forward I have added a fat-jar assembly to the project, a single jar with all dependencies unpacked i.e. ucanaccess-<version>-fat.jar.
Using this fat jar, there is only one jar to add to the LibreOffice JVM classpath in order to make JDBC connections to MS Access.

Link to Sonatype snapshot repo
Look for *-fat.jar and *-distrib.zip

If the new fat jar were also shaded it would make my life slightly easier :wink:
(We currently build a shaded version of ucanaccess in order to eliminate the possibility of jar conflicts.)

Hi @ravenAtSafe,
I am sure we can find a good solution. Note that dependencies have been reduced to a minimum (in the current snapshot which will be released soonish). Possibly potential conflicts are already resolved?
Please open open a new issue and provide a  few details. Thanks!

Hi there!
I have implemented changes to create a shaded executable uber jar. The intention is to have this one jar that fulfills all requirements from requests #7 and #8.
The jar is included in the binary distrib. Everything except UCanAccess itself is shaded (prefix shaded.) The start scripts (console.*) are actually no longer required but continue to exist for the time being. This is still work-in-progress (snapshot) and not released yet.
Usage instructions

Download ucanaccess-5.1.2-SNAPSHOT-distrib.zip
or add to project by Maven dependency:

<dependency>
  <groupId>io.github.spannm</groupId>
  <artifactId>ucanaccess</artifactId>
  <version>5.1.2-SNAPSHOT</version>
  <classifier>distrib</classifier>
  <type>zip</type>
</dependency>

Unpack to desired location on your machine

To enter the UCanAccess console

Run console.sh or console.bat -or-
double-click the jar -or-
invoke the software by running java -jar <jarName>

LibreOffice MS Access database access
LibreOffice

Options > LibreOffice > Advanced  > Java options
Select desired JRE
Edit classpath by clicking 'Class Path...'
Add ucanaccess-5.1.2-SNAPSHOT-uber.jar to classpath from your download/install location
Enter advanced options if needed, confirm and exit dialog

LibreOffice Base

In wizard select to connect to an existing database (JDBC)
Enter database url without jdbc: prefix: ucanaccess://full_path_to_database e.g. jdbc:ucanaccess:///home/spannm/projects/github/spannm/ucanaccess/src/test/resources/testdbs/accessLike.mdb
Enter fully qualified UCanAccess driver class name net.ucanaccess.jdbc.UcanaccessDriver
Test by clicking 'Test class'
Proceed in wizard, confirm and save file
Ready to use your MS Access database in Base

Hi @gordthompson, could you review the solution whether it covers your requirements? Thx

Hi @spannm . I just tested Python/JayDeBeApi using code adapted from here and it works. I did get a couple of warnings
$ python3 jdb_test.py
Apr. 23, 2024 8:34:18 A.M. net.ucanaccess.jdbc.UcanaccessConnection logStatementWarning
WARNING: Please use a simple statement (not a PreparedStatement)to execute DDL (e.g.,CREATE TABLE)
Apr. 23, 2024 8:34:19 A.M. net.ucanaccess.jdbc.UcanaccessConnection logStatementWarning
WARNING: Please use a simple statement (not a PreparedStatement)to execute DDL (e.g.,CREATE TABLE)
(1, 'Gord')

but they are probably just a JayDeBeApi thing, so I'm satisfied. Thanks!

You could then investigate where JayDeBeApi issues the calls in question and send them a patch.

Someone did that over 7 years ago, but nothing came of it. :(
https://github.com/baztian/jaydebeapi/pull/29
I'm not too concerned since it's just a warning. (And personally I no longer officially support Access, anyway.)

Hello,
I am interested in trying UCanAccess with Libreoffice base.
Are you still planning to release a binary distribution?

This seems to be the latest build availabke:
https://s01.oss.sonatype.org/content/repositories/snapshots/io/github/spannm/ucanaccess/5.1.2-SNAPSHOT/ucanaccess-5.1.2-20241029.072454-27-distrib.zip
but I am not sure whether this what we are after.
An oficial github release would be excellent :-)

@ghomem - Did you notice the instructions here? https://github.com/spannm/ucanaccess/issues/7#issuecomment-2054059717
@spannm - I agree that an official release would be helpful. I have a number of old SO posts that could be updated to include a new official download location.

@ghomem - Did you notice the instructions here? #7 (comment)

Yes, I did but the mentioned file does not exist anymore. And the file I managed to cherry pick (two comments above) does not work.

@spannm - I agree that an official release would be helpful. I have a number of old SO posts that could be updated to include a new official download location.

I can test on Ubuntu 24.04 + Libreoffice Base.

I just downloaded
https://s01.oss.sonatype.org/content/repositories/snapshots/io/github/spannm/ucanaccess/5.1.2-SNAPSHOT/ucanaccess-5.1.2-20241029.072454-27-distrib.zip
and I can see "ucanaccess-5.1.2-SNAPSHOT-uber.jar" in there.

I just downloaded
https://s01.oss.sonatype.org/content/repositories/snapshots/io/github/spannm/ucanaccess/5.1.2-SNAPSHOT/ucanaccess-5.1.2-20241029.072454-27-distrib.zip
and I can see "ucanaccess-5.1.2-SNAPSHOT-uber.jar" in there.

With that particular file you linked and adding "ucanaccess-5.1.2-SNAPSHOT-uber.jar" to the classpath I was able to load the tables on Libreoffice.

It seems to work, at lest for this use case. Would you consider oficializing a release to assign a more formal nature to this file?
Thank you for the support.

Hi @ghomem, hi @gordthompson
a release is overdue, I've been wanting to release the software for weeks but didn't get around to it yet.
I'll get to it very soon - promised.

Hi @ghomem, hi @gordthompson
a release is overdue, I've been wanting to release the software for weeks but didn't get around to it yet. You know how it is :- I'll get to it very soon - promised.

Thank you for your work!

Fixed in release 5.1.2

Hi @spannm ,
I don't find in
https://github.com/spannm/ucanaccess/releases/tag/ucanaccess-5.1.2
a link to the distribution binary.
What am I missing?

Hi @ghomem,

I don't find in
https://github.com/spannm/ucanaccess/releases/tag/ucanaccess-5.1.2
a link to the distribution binary.

You can now find the release artifacts in Maven Central.
I've also added them to the release page for convenience (I don't have an automated workflow for this (yet)).
cheers & happy halloween

@spannm
Thank you for the efforts. Now I see the files in the release page, which is very convinient.
Best regards and happy halloween


Java: Update Vectorized and Vectorizable interfaces to have one search method

Fixed by microsoft/semantic-kernel-java#226


SOLR-16340 Provide a cluster_id label for all metrics exposed by the Solr Prometheus Exporter

https://issues.apache.org/jira/browse/SOLR-16340

Added label cluster_id on all metrics from exporter
Added filter (template) for cluster_id in grafana board
Defaults to base_url or zk_host, but can be overridden by --cluster-id foo or --id foo command-line or env.var CLUSTER_ID.

~Lacks a test for the actual setting of clusterId and argument parsing, since existing junit tests do not invoke the exporter through shell~. Edit: Docker test now validates env.var, and thus also cmdline arg parsing.
I keep clusterId as a static var on SolrExporter.java to avoid passing it through all kinds of intermediate method calls to SolrScraper, but there is probably a more elegant and more testable way..

@janhoy I added a PR that removed the use of the static variable, and also added zk_host and base_url labels to the scrapingErrors metric.
Feel free to revert if you don't like it!

Nice improvement. Good point about labeling exporter’s own metrics! Should grafana board display some exporter metrics ?

Should grafana board display some exporter metrics ?

Yeah probably. I haven't looked over the grafana changes here, but will hopefully have time to help with those later this week!

Committed quoting of ClusterID in windows script, and resolved CHANGES conflict. Looking forward to final reviews.

Looking forward to final reviews.

Going to spin this up today with a few clusters and poke around!

Ok the dashboard looks good! Just a few changes from my end, mainly that "Shard Leaders", "Replica State History" and "QPS (Collection)" Did not specify the cluster_id, and "QPS (Collection)" did not group by it either. I will push a change with this, but the only problem is the default cluster_id. If it's a long zk-connection-string, then this will probably become problematic and make the UI somewhat unusable. (since the cluster_id will be extremely long).
Maybe instead we use a default such as "main", and require users to override it? I get the convenience of using a zk-connection-string, since that is by-default a 1-1 mapping with cluster_id, but maybe we require users to provide the id?
Would appreciate your opinion here.
Also sidenote, in the future maybe we can change this to a graph for multiple clusters? (Not in this PR though)

Here is an example of using zkConnectionString as cluster_id:

So another option could perhaps be defaulting to a hash of zkHost or solrUrl, e.g. the first 10 chars of a sha hash

I love it.
I fixed a small error, but I think we should be good to go now!

Ah, that was why the Docker test failed. Nice to have end-to-end tests this way.
I’ll proceed with merge tomorrow..


tweaked the option

Should resolve https://github.com/smcnally/office_quiz/issues/2
note the image is moved outside and then "under" the label
Added a click handler for the image itself
Further color tweaks should hopefully be easy.

Looks good. Testing locally. Will report back and merge -

This is merged.
answer labels and buttons look good.
new constructor looks good, thank you.
answerOptionContainer looks good, thank you.


[common] ability to specify the mountPropagation key in a volumeMount entry in common library

Details
Helm chart name:
common
Describe the solution you'd like:
Ability to specify the mountPropagation key in a volumeMount entry.
Anything else you would like to add:
Backstory: For some volumes, we need to specify mountPropagation in addition to mountPath, readOnly, subPath, etc.
For example, an rclone container can mount a google drive or other remote into a shared emptyDir volume, and the other container (the main one from the helm chart, for example, sonarr, radarr, plex) should be able to see that sub-mount in that emptyDir volumeMount.
Referring to https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation, this usecase needs me to set Bidirectional mountPropagation for the custom/emptyDir volume on the rclone container (which I can do right now since the additionalContainers yaml is passed as-is to the rendered resource), but the volumeMount in the other main container still needs mountPropagation set to either Bidirectional or HostToContainer. This is currently not possible in the persistence/storage of the common-library since the template specifically sets the known-so-far keys (mountPath, subPath handling, readOnly).
The resulting volumeMount I want to be able to render is (assuming the example of plex):
      containers:
      - name: rclone # whole entry added as-is via additionalContainers yaml
        image: rclone/rclone:1.55
        args:
          - mount
          - "remote:location/media"
          - "/media/"
        volumeMounts:
        - name: rclone-shared-mount
          mountPath: /media
          mountPropagation: Bidirectional
      [...]
      - name: plex # rendered within helm chart
      [...]
        volumeMounts:
        - name: rclone-shared-mount
          mountPath: /media
          mountPropagation: HostToContainer # need to be able to specify this from the persistence.<name>.mountPropagation values
      volumes:
      - name: rclone-shared-mount # rendered from persistence.rclone-shared-mount.*
        emptyDir: {}

Additional Information:
This isn't limited to only emptyDir volumes specified inline/in-helm, since we should be able to mount an existngclaim PVC or a hostPath with a specified mountPropagation.
Adding something like
          {{- with $persistenceItem.mountPropagation }}
  mountPropagation: {{ . }}
          {{- end }}

to both sections of https://github.com/k8s-at-home/library-charts/blob/main/charts/stable/common/templates/lib/controller/_volumemounts.tpl should do it.
Thanks!

Thanks for creating this issue! I'll try to add a solution for this in our next common version.


question

which mining pool website should i chose im really confused because i cant seem to find one without a bad reputation which one do you recommend

I am currently mining on https://monero.hashvault.pro/

Probably 2miners or nanopool. HashVault has a ban policy that you can find on their start page, where if you have too many low hashrate miners without hosting your own xmrig-proxy you can get banned.