Networking: Add scaffold of tunnel EDS builder

Add Tunnel parameter:

EndpointBuilder,
Add tunnelType. This type is unique since we assume an eds cluster can only have exactly 1 type.

IstioEndpoint
For each pod/workload, the registry tracks what tunnel types can be supported by this workload.

ApplyTunnel is the stage generating the endpoints on top of the direct connected endpoint.
Part of BTS #24540
The endpoints address are updated according to the tunnel.

Tunnel Ability is not yet defined at ingress sidecar.
Need a few changes at envoy both egress proxy and ingress proxy. So the above kill switches should only be enabled in test case.
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Pull Request Attributes
Please check any characteristics that apply to this pull request.
[ ] Does not have any changes that may affect Istio users.

@lambdai: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

release-notes_istio
2d9c84e90f190bb4d77681cd884304ae4c4d5083
link
/test release-notes_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@lambdai: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

release-notes_istio
2d9c84e90f190bb4d77681cd884304ae4c4d5083
link
/test release-notes_istio

gencheck_istio
2d9c84e90f190bb4d77681cd884304ae4c4d5083
link
/test gencheck_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@lambdai: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

unit-tests_istio
670b64d41571a340b405df9866e2ee3c477293fd
link
/test unit-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test unit-tests_istio

Please link a  related issue this pr want to resolve

@lambdai: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

gencheck_istio
37492ee85d42d43d666111c7e97af7fa35d87fc4
link
/test gencheck_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@lambdai: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

gencheck_istio
37492ee85d42d43d666111c7e97af7fa35d87fc4
link
/test gencheck_istio

unit-tests_istio
37492ee85d42d43d666111c7e97af7fa35d87fc4
link
/test unit-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu Updated: the goal is to move on with better transport security.
This PR start with defining tunnel ability. The future PRs will expose the ability to istio.io annotation.

@lambdai: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

unit-tests_istio
37492ee85d42d43d666111c7e97af7fa35d87fc4
link
/test unit-tests_istio

gencheck_istio
4447764388afc6f0791b245481b6fdf5cb85705b
link
/test gencheck_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@lambdai: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

gencheck_istio
4447764388afc6f0791b245481b6fdf5cb85705b
link
/test gencheck_istio

unit-tests_istio
4447764388afc6f0791b245481b6fdf5cb85705b
link
/test unit-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@lambdai: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

unit-tests_istio
4447764388afc6f0791b245481b6fdf5cb85705b
link
/test unit-tests_istio

gencheck_istio
9195e4e117b339092433d2e0c62025686243ebc1
link
/test gencheck_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@lambdai: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

gencheck_istio
0f59a5f7fedf1435f4ec4af8730988b665137a9c
link
/test gencheck_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test gencheck_istio

@lambdai: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

gencheck_istio
0f59a5f7fedf1435f4ec4af8730988b665137a9c
link
/test gencheck_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@lambdai: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-pilot-k8s-tests_istio
4b2e499e300072993c17a5c39786afce5db1cd68
link
/test integ-pilot-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-pilot-k8s-tests_istio

@lambdai: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-pilot-k8s-tests_istio
4b2e499e300072993c17a5c39786afce5db1cd68
link
/test integ-pilot-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-pilot-k8s-tests_istio

/test benchmark_istio

benchmark suggest large increase in memory allocation: EndpointGeneration/1000/1-8 36.9kB ± 0% 53.3kB ± 0% +44.57% (p=0.008 n=5+5). I would have expected this to be pretty small change since we do have anything with h2 tunnel (which would call expensive proto.Clone)

A little bit more during the endpoint build due to an applier per endpoint. But these should be ephemeral.
Another source of the added allocation is copy on write. I may too eager to copy.
Where is the report?

benchmark suggest large increase in memory allocation: EndpointGeneration/1000/1-8 36.9kB ± 0% 53.3kB ± 0% +44.57% (p=0.008 n=5+5). I would have expected this to be pretty small change since we do have anything with h2 tunnel (which would call expensive proto.Clone)

A little bit more during the endpoint build due to an applier per endpoint. But these should be ephemeral.
Another source of the added allocation is copy on write. I may too eager to copy.
Where is the report?

https://prow.istio.io/view/gs/istio-prow/pr-logs/pull/istio_istio/28244/benchmark_istio/34

EndpointGeneration/1000/1-8                  36.9kB ± 0%        53.3kB ± 0%  +44.57%  (p=0.008 n=5+5)
EndpointGeneration/1000/1-8                    26.0 ± 0%          28.0 ± 0%   +7.69%  (p=0.008 n=5+5)
I bet the top one is the allocated ram and the bottom is the allocated count?
Maybe the gc doesn't decrease the counter?
Anyway there might be expensive copy at slice expansion. Let me see if there is low hanging fruit

@howardjohn It should be good to go. This is the known cost.
base
				locLbEps = &endpoint.LocalityLbEndpoints{
					Locality:    util.ConvertLocality(ep.Locality.Label),
					LbEndpoints: make([]*endpoint.LbEndpoint, 0, len(endpoints)),
				}

PR
				locLbEps = &LocLbEndpointsAndOptions{
					endpoint.LocalityLbEndpoints{
						Locality:    util.ConvertLocality(ep.Locality.Label),
						LbEndpoints: make([]*endpoint.LbEndpoint, 0, len(endpoints)),
					},
					make([]EndpointTunnelApplier, 0, len(endpoints)),

For 1000 endpoint, a 1000-interface array is introduced make([]EndpointTunnelApplier, 0, len(endpoints)),
The benchmark shows 18.79MB on 1173 iteration. It's roughly 16 bytes per endpoint.
Regardless what the base is, the increment is about 16 bytes per endpoint and 2 alloc per endpoint.
EndpointGeneration/1/100-8                   94.3kB ± 0%        99.9kB ± 0%   +5.93%  (p=0.008 n=5+5)
EndpointGeneration/10/10-8                   12.6kB ± 0%        14.6kB ± 0%  +15.89%  (p=0.016 n=4+5)
EndpointGeneration/100/10-8                  45.1kB ± 0%        63.5kB ± 0%  +40.62%  (p=0.008 n=5+5)
EndpointGeneration/1000/1-8                  36.9kB ± 0%        53.3kB ± 0%  +44.57%  (p=0.008 n=5+5)

EndpointGeneration/1/100-8                    1.51k ± 0%         1.71k ± 0%  +13.24%  (p=0.016 n=4+5)
EndpointGeneration/10/10-8                      158 ± 0%           178 ± 0%  +12.66%  (p=0.016 n=4+5)
EndpointGeneration/100/10-8                     165 ± 0%           185 ± 0%  +12.12%  (p=0.016 n=5+4)
EndpointGeneration/1000/1-8                    26.0 ± 0%          28.0 ± 0%   +7.69%  (p=0.008 n=5+5)

@lambdai: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

lint_istio
fc0e56a7132acab1ecebb81d0826aaf5b1cc3fda
link
/test lint_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@lambdai: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

lint_istio
fc0e56a7132acab1ecebb81d0826aaf5b1cc3fda
link
/test lint_istio

gencheck_istio
fc0e56a7132acab1ecebb81d0826aaf5b1cc3fda
link
/test gencheck_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@lambdai: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

gencheck_istio
bcf7fa4bfac6a407a6d73e91a3e9dac2ca7527f3
link
/test gencheck_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@lambdai: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

gencheck_istio
bcf7fa4bfac6a407a6d73e91a3e9dac2ca7527f3
link
/test gencheck_istio

integ-pilot-k8s-tests_istio
bcf7fa4bfac6a407a6d73e91a3e9dac2ca7527f3
link
/test integ-pilot-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn Thank you for the comment. Let me address it in the next PR adding metadata


Script show styles

@jalletto LEt's work on this together today to make sure the styling syncs up.  Thanks!


:bug: I91 fix importer issues

TODO: Update bulkrax for submodule, then update submodule for pals ✅
This corrects the reason why we were unable to import.
split-embargo-pals.zip
basic-pals-import.csv

Are the spec failures valid? Or are they known flaky specs on this repo?

Are the spec failures valid? Or are they known flaky specs on this repo?

they're known flakes. feature spec


README.md -> link to v2 instead of the v1 "main thread"

Citing the current README.md: "If you have questions or issues, either post it in the AutoHotkey forums' main thread"
I was mislead by this link and posted a question related to this UIA-v2 repository version in the above v1 thread.
Proposal: Change the link to this thread which I think is the the "v2 main thread"?
https://www.autohotkey.com/boards/viewtopic.php?f=83&t=113065
Hoping this helps & keep up the GREAT work!! Loving it!

Thanks, fixed. :)


Missing support for Workona workspace sections

Workona supports the ability to create sections in order to organize workspaces. In the Workona export files, the sections are represented as top-level workspaces, which then in turn have a workspaces section under them, listing each workspace. The current code sees the workspace sections as workspaces themselves, but then doesn't does not drill down, so it finds no resources to export.

Hi, feel free to open a PR and I will merge it. It looks like this is a new change, I just downloaded a. new export and it has a different structure.
What would a bookmarks.html file look like in the case of sections?  Would it match the structure of the output, like section --> workspace --> bookmark, and then workspaces lacking a section would be in the "Ungrouped" section like in the JSON?  Or would they be in the top level?

This wouldnt be such a tough change to make.

@kaveh-ahmadian I have pushed branch "sections", which correctly parses the new export format.  The HTML produced creates folders for each section, including one for "Ungrouped", and these folders contain folders for the Workspaces.  lmn if it works and I will merge to master (https://github.com/mikewaters/workona-import/pull/2)

Here is an example of what it looks like in my personal export:

btw I had to deprecate the YAML ("Text") export, its a PITA and with the JSON option there's really no reason to use it.  If the output was correctly quoted, I wouldnt have to bother.  So if it fails spectacularly with a message about YAML, just redo the export using JSON.


Wrong Chainsymbol used in total spending on profile page

What happened?
Recently someone purchased polkadot punk #2 and I viewed the buyer profile and also checking some stats then I found a little bug that I was using AHP and chain symbol was Ksm unless of Dot  in total spend

Please reproduce in steps
change chainsymbol with current url prefix
Expected Behavior
chain symbol should be dot
What browsers are you seeing the problem on?
Chrome
At which address did you encounter bug?
kodadot.xyz
Are you logged in?
Yes
Which wallet you are using?
PolkadotJS
Screenshots

Relevant log output
No response
Payment link for reward
https://kodadot.xyz/ahp/transfer?target=1HzwKkNGv4gdWq4ds1C5k63u8hvmjC6ULneAaZbwUBZEauF&usdamount=0
Code of Conduct

[X] I agree to follow this project's Code of Conduct

👋


Emails custom domain

Describe the issue
Not receiving emails no more
Code of Conduct

[X] I agree to follow the IETF's Code of Conduct

Please provide detail about issues you are having with mail from ietf.org. Note that you have agreed to follow the IETF code of conduct.


Decode webP CGBitmapContextCreate fail error

New Pull Request Checklist

[x] I have read and understood the CONTRIBUTING guide

[x] I have read the Documentation

[x] I have searched for a similar pull request in the project and found none

[x] I have updated this branch with the latest master to avoid conflicts (via merge from master or rebase)

[ ] I have added the required tests to prove the fix/feature I am adding

[x] I have updated the documentation (if necessary)

[ ] I have run the tests and they pass

[ ] I have run the lint and it passes (pod lib lint)

This merge request fixes / reffers to the following issues: ...
Pull Request Description
...

Fixed in #1952


Encryption sender secretbox payload is the sender KID (not raw key bytes)

The encryption mode was assuming that the sender KID is the raw box key bytes. This is usually the case, but not always. If you don't use the raw box public key bytes as the KID, then decryption fails.
In other words, the sender secretbox payload is actually the sender KID. We rename senderKey to senderKID and remove the rawBoxKeyFromSlice call. LookupBoxPublicKey is called with KID bytes as before to get the BoxPublicKey.
In the test, where the sender KID bytes were being corrupted, this should result in the sender key not being found since the KID was changed and the subsequent call to LookupBoxPublicKey with the wrong bytes, wouldn't find a key.
This should be a pretty safe change, but lmk if you need any more details or clarifications.

@AMarcedone can you take a look at this PR?

Hi, thanks for the PR.
I am not super familiar with this part of the codebase, so let me know if you disagree with me below, I am pretty open to feedback.
I agree that reading the code it seems that the LookupBoxPublicKey call should take a keyID and not the key itself.
However, from the spec (https://saltpack.org/encryption-format-v2) it seems that:
The sender secretbox is a crypto_secretbox containing the sender's long-term public key, encrypted with the payload key from below..
It mentions the key itself, not the key ID. So accepting this change would require editing the spec as well.
I think we also do not impose any strict requirements on key IDs (they could not be unique for example), so it seems that encoding the public key itself is more robust than the ID, which makes me a bit reluctant to accept the change as is.
If you want to use a keyring where the keyID is not the key itself, the alternative solution I see would be to substitute the call to LookupBoxPublicKey with a reverse lookup call which ensures that the public key is present in the keyring.
What do you think?

Ah right, the spec is clear that it should be the public key bytes, I should have checked out the spec more closely.
Thanks for looking at this!


Mention that the setter returned by useState has no return value

This PR fixes #2232 by mentioning that the setter returned by useState(...) function does not return any value (return type is declared as void in the React source).

Closing as comment in the issue.
Thank you, @joanrieu, for taking the time to contribute!


extend UsePrometheus() with optional bool parameter collectAspMetrics

true: collect ASP request URLs and timings (as before)
false: don't collect anything (only what you explicitly log)
Default value true to be backward compatible.

Can we use an options class rather than a flag? E.g.
  CollectHttpMetrics = true
})

Using an options class allows us to gracefully add new options without them being breaking provided we have reasonable defaults.

Looks like my code was cut out, I meant:
UsePrometheus(new PrometheusOptions()
{
 CollectHttpMetrics = true
});

I don't think that it's future proof enough.

@ahoka not bullet proof but u could in v2.8 add a new options and I'll just update references without having to change my code. Options are also more readable than a Boolean value.

Hi,
@jbrekle Thanks for the proposed PR, but I'm afraid I agree to @ahoka 's opinion about its limited extensibility. Adding more and more flags in the future may not work: they may not be clean enough when more complex config needs arise.
Regarding default operation: We have made a mistake when we enabled ASP metrics collection by default. We have realized this problem both from the discussion here, on GitHub, and also from our own experiences, especially about the Prometheus metrics explosion (see: #15).
I believe we should rather break backwards compatibility now, but then provide a future proof approach, rather than trying to stick to poorly chosen defaults only for the sake of backwards compatibility.
@gideonkorir The config object pattern is versatile enough indeed, but I think we should stick closer to the ASP.Net Core practice and use more fluent, builder pattern based configuration approach.
For these reasons, and because @adamhoka has come up with a PR about his approach, which fits the preferences above the best I decided that his solution shall be merged, and a new major version will be released with the notice about the breaking change in the API.
I sincerely hope that even though this PR is not accepted, you will still consider to contribute to the project in the future, should you see an opportunity.
Best Regards,
@kodfodrasz

sure, whatever solves the issue is fine for me


Fix Relationship Validation [OSF-6969]

Purpose

Fixes failing relationship validation for preprint payload.
Changes the primary_file relationship type to match the primary_file relationship key.

Ticket

Relates to OSF-6969

I don't know. This seems a liiiittle too complicated of a change. Could you try simplifying a bit?
(This is good if the tests pass. 🎉 🐧 🎉 )

Hmm. I notice that this actually reverts a change made from file to primary_file under the hood- any specific reason? See 38cd2a7cbe0c9f58153a5feb16ced610216c3f76
(ultimately, I thought these were still type:file for the API under the hood)

Okay, looked over this more. Going to revert out the old change and not merge this.

Feel free to close this, someone who has push access to this repo.

@abought, basic relationship validation was added recently (per #6189, to resolve [OSF-])

Hmm. I notice that this actually reverts a change made from primary_file to file previously- any specific reason? See https://github.com/CenterForOpenScience/ember-preprints/commit/38cd2a7cbe0c9f58153a5feb16ced610216c3f76

Reverting that probably isn't for the best, as it would set the relationship_key to be preprint_file, not primary_file. This PR seems like it's the right change

In this case I'm referring to the change to the type field, not the relationship key. (Sorry, wi two things changed,  that could have been more clear)
My understanding was that the type field had to match a type in the API, eg file, rather than a field name.

On Aug 29, 2016, at 8:59 PM, mfraezz<EMAIL_ADDRESS>wrote:
Hmm. I notice that this actually reverts a change made from primary_file to file previously- any specific reason? See 38cd2a7
Reverting that probably isn't for the best, as it would set the relationship_key to be preprint_file, not primary_file. This PR seems like it's the right change
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Ah, okay. That makes more sense.
And so far as I can tell, the type field wasn't being validated as a legitimate type with a view in the API, only to be the same as the relationship key -- local testing with current API develop and this change indicates that it this works (that is, I get a 201 when submitting a preprint with this, or a 400 without, and it does become a valid preprint with a file). ¯\_(ツ)_/¯

The reason this issue is closed and the other one reverted is because that was not how it's supposed to work. We will go down a better path for this in the near-ish future.

Also, the reverted change is this PR, not the primary_file/file change @abought mentioned above.


Add back the InfluxQL REPL

In 1.x, there was an influxql REPL where we could issue influxql commands repeatedly against the same server, without re-typing 'influx' or having annoying shell quoting. We should port the 1.x REPL to the 2.x CLI.
The REPL should give clear error messages when attempting to use 1.x-only InfluxQL features, see https://docs.influxdata.com/influxdb/v2.1/query-data/influxql/?t=InfluxDB+API#influxql-support

See also https://github.com/influxdata/influx-cli/issues/331 if we can't get to this in 2.3


Provisiong burstable worker nodes when added in instance or instance-pool mode

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

New or Affected Resource(s)

Potential Terraform Configuration

When worker nodes are added as instance pool or instance , we need an ability to define burstability. Currently its not possible to add a worker in instance or instance-pool mode , with this flag. I have given a sample below for adding baseline_ocpu_utilization in the shape_config
# Copy-paste any Terraform configurations for how the requested feature may be used.

      dynamic "shape_config" {
        for_each = length(regexall("Flex", each.value.shape)) > 0 ? [1] : []
        content {
          baseline_ocpu_utilization = lookup(each.value, "burst", "BASELINE_1_1")
          ocpus = each.value.ocpus
          memory_in_gbs = ( # If > 64GB memory/core, correct input to exactly 64GB memory/core
            (each.value.memory / each.value.ocpus) > 64 ? each.value.ocpus * 64 : each.value.memory
          )
        }
      }

References

resolved in #970


Connect to different host

I'm unable to connect to a host other than localhost even if the server is on the same local network as the NodeJS server runs within.
I've tried connecting by host name and IP address, but no luck.

Urgh. As soon as I post, it starts working. Sorry.

np ; )

what was the fix??


add constraints

some penalties should really be constraints.
for example, monotonic smoothing and harmonic smoothing should be hard constraints.
perhaps they might also be added as penalties, but a basic application would be as constraints.

there is the projection method for adding these types of subspace constraints:
http://www.stat.colostate.edu/statresearch/stattechreports/Technical Reports/2010/2010_3.pdf
but also the simpler method of adding an extremely large penalty to a violated constraint, a la eilers and marx:
http://www.ibschannel2015.nl/project/userfiles/Crash_course_handout.pdf
i like the simple approach better, because the projection method requires rethinking the linear algebra. Any way, the simple method seems essentially similar.

implemented monotonicity and convexity penalties!!
just need to write the harmonic penalty.

still some problems:

the convexity constraint showed me that the GAM still has edge problems!
i think that the repeated edge knots make the bases not sum as expected.

the line above is the sum of the bases multiplied by a linearly increasing factor. this should produce a straight line, and in fact it does everywhere except the edges!
the convexity looks violated when using many splines.
perhaps need to increase the regularization amount by more?

increasing regularization from lam=1e6 to lam=1e9 helped with the second point.
now i get this:

you can tell by inspecting the second order differences of the GAM coefficients. they are all very small (~1e-8) but NEGATIVE! this means that the model wants to make a concave fit, but is being forced to make an extremely flat one.

with such large regularizations, the numerical methods become unstable.
i got a CholmodNotPositiveDefiniteError: ../Core/t_cholmod_change_factor.c:243: L not positive definite (code 1) doing the same optimization with 50 splines.
need to think this over a bit.

i tried adding .1to the diagonal of the constraint matrix and this fixes things.
it is a bit unsatisfying since we are effectively adding l2 regularization everywhere whenever we use constraints.

note, it is possible to add smaller values to the diagonal, but this increases the number of steps required to reach convergence.

ive also been thinking about how to implement a simple project method to constraint enforcement.
so far just thinking iteratively. for monotonic increasing:

find which coefficients violate constraint
for violated_coef in violated_coeffs:
-- set difference between violated_coef and its prev-neighbor to the difference between the prev-neighbor and its prev-neighbor.

i added a circular constraint.
some notes,

the circular constraint doesnt make complete sense as a penalty because it only enforces periodicity, ie no smoothness.
it is becoming more important to be able to compose penalties :)


Skip fuzzing for cfg in rustdoc tests too

Follow up to:

#2736

Both this PR and the previous emit warnings because latest nightly checks for un-used and undeclared cfg strings. I need to upgrade the rust toolchain in this repo to resolve


fix navigation around split operations

@moveson I think this fixes the odd navigation issues around split operations. There may be some scenarios that I missed.
I don't falling back to splits_url on destroy of a split is the right thing to do but it may be the best option for now.

Looks great. Thanks Dave.


Immersive Engineering && Mekanism

IE page

alright this is probably ready now

Could probably use some better examples in the future but for now this works


Expose pub fn of a re-exported struct to Dart

Describe the bug
I noticed that functions of re-exported types are not generated for accessing in Dart.
As in Rust these are available it would be nice to have the same behavior for Dart access.
For example, if I have:
file myStruct.rs
pub struct MyStruct{
   field: String
}

impl MyStruct{
   pub fn get_content(&self) -> String {
       self.field
   }
}

and another file myApi.rs
pub use domain::MyStruct;

pub fn useMyStruct(myStruct: MyStruct){
   myStruct;
}

and configured flutter_rust_bridge.yaml with
rust_input: 'crate::application::api'

then the function myStruct.get_content() is not generated in FRB.
An easy workaround is to add  to flutter_rust_bridge.yaml myStruct.rs
rust_input: 'crate::myApi, crate::myStruct'

but since other rust code would be able to access the function through a re-import (by use other_project::myApi) it would be nice to have Dart be able doing the same.
Would that be easy to implement - or have unwanted side effects and was thus disabled intentionally?
Steps to reproduce
Implement the above example ... I can do that if needed.
Logs
Nothing specific written into the logs.

Expected behavior
No response
Generated binding code
No response
OS
No response
Version of flutter_rust_bridge_codegen
2.5.0.
Flutter info
No response
Version of clang++
No response
Additional context
No response

Good suggestion!

have unwanted side effects

It seems that frb chooses to scan functions inside rust_input and ignore functions/methods outside that. Therefore, by doing this re-export, it may be a breaking change...
In addition, we use pub use to allow users to expose third party types to be understood by frb_generated.rs, thus giving pub use a second meaning may be a little bit confusing.
Thus, I guess it may not be super great to implement this.

Good suggestion!

Thanks :)

have unwanted side effects

(...) doing this re-export, it may be a breaking change...

Hmmm ... I think this might not break old code: more functions would be exposed, but they would not be used (if I don't overlook something). Surely on the rust side nothing changes, on the dart side there should be just more functions accessible but not automatically called.
Maybe there would be a conflict if a struct is re-exported and part of rust input - potentially duplicated code. But we might be able to detect this and warn when generating code.

In addition, we use pub use to allow users to expose third party types to be understood by frb_generated.rs, thus giving pub use a second meaning may be a little bit confusing.

Oh, yes, that is a good point. The question is then is a user would or would not like to have the functions of these types exported as well. I would assume that I might want this functions - if I deal with a specific struct, these functions would give me tools do deal with it. But maybe I am thinking to object-oriented about that. In reality I would probably not have a need for 3rd party structs in Dart and deal with them in rust ...
Maybe I misunderstood you - but isn't the scenario nearly the same? pub use to export structs of own, not rust input included structs and for 3rd party structs? In both cases currently without functions?

Looks reasonable! I will think about this proposal a bit later.

Sure, take your time! There is a very easy workaround, so this is more of a "clean-up" :) Worth doing only is easy todo and side-effect free :)
Though a bit too much working with the inner core that I feel comfortable doing.

I think this might not break old code

I was thinking about if the newly scanned function has something that frb does not understand. But yes, most of the time frb will happily skip functions it does not understand, so it should be OK.

Maybe there would be a conflict if a struct is re-exported and part of rust input

Yes that also look like a problem.

Maybe I misunderstood you - but isn't the scenario nearly the same? pub use to export structs of own, not rust input included structs and for 3rd party structs? In both cases currently without functions?

The problem is, pub use currently has semantics that "that type is exposed s.t. frb_generated.rs can use it"; but we are now introducing a second semantics that "that type is exposed to the Dart side". That's why I say it can be confusing...

I think this might not break old code

I was thinking about if the newly scanned function has something that frb does not understand. But yes, most of the time frb will happily skip functions it does not understand, so it should be OK.

Maybe there would be a conflict if a struct is re-exported and part of rust input

Yes that also look like a problem.

Yes - it sounds like this could happen - but I assume that the chance is very low.
And probably the fix very easy: If we have, for example, a name clash, the formally good build would break, but a good error message would make it trivial to fix.

Maybe I misunderstood you - but isn't the scenario nearly the same? pub use to export structs of own, not rust input included structs and for 3rd party structs? In both cases currently without functions?

The problem is, pub use currently has semantics that "that type is exposed s.t. frb_generated.rs can use it"; but we are now introducing a second semantics that "that type is exposed to the Dart side". That's why I say it can be confusing...

Aha. Currently the type is exposed to Dart as well, not only to frb_generated.rs - but yes, it is the difference between „use a rust structure“ vs „use a function“.
Maybe my use case can show why it would be good to expose functions as well - however, it is not a big deal to not do that:
I have a api-class which offers a „process_command“ function. Here I modify data structures, which are passed around, but are not used in Dart directly. This data structures (domain objects) have inner fields, which I don’t want to expose - so I use ‚pub use‘ for exposing the main structures only.
Now I want to display some attributes in Dart. For that I implemented a function which computes a String that is moved to Dart.
However, this function is coded in the domain file, as coming from OOP I want to have it close to the type definition.
Now I actually implemented it in the api file, which ‚pub use‘es the main struct and ‚use‘es the inner fields to implement that String producing function.
The „workaround“ is so easy and not necessary wrong, that it is ok as well.
I was just wondering, if it wouldn’t be less confusing if ‚pub use“ behaves the same to rust and dart code.

Hmm I see. I personally am still not sure whether we should add it; but if you have the interest in doing so, feel free to PR!

I had an unexpected twist, which is a good argument against implementing this!
I implemented some generic traits, using associated types and implies, etc.
These constructs are not supported by FRB at the moment, so the generated code was faulty.
Thus, I actually removed my "domain::*" code from rust_inputs, so that only the types are scanned, not the functions. These are accessed through proxy functions in my API (wich do a bit more).
So, taking into account that more syntactical constructs could be added in future, to avoid FRB having to implement those all, I thing we should not implement this issue!

That looks like also a good reason!


CSHTML file not refreshing

I make a change in a .cshtml file. I refresh the browser. I don't see the change. I rebuild the application. I see the change. It worked in asp net core 2.2.
How to solve?
https://stackoverflow.com/questions/54106058/why-does-page-not-update-after-refresh-when-cshtml-changes

This issue was moved to aspnet/AspNetCore#14430


Reconnect if IP is not accessible

From discord
I’m not a web developer, so I haven’t really dug into the codebase to fully understand the implementation yet, but I definitely plan to in the future to be able to contribute more effectively.
Right now, I’m testing mavlink-server in one of our applications and noticed some behavior I wanted to clarify. I’m not sure if I’m missing something, so any insights would be appreciated!
Here’s my setup:

CubeOrange connected to an Raspberry Pi via serial.
The Pi is connected via Ethernet to two 4G routers (running VPN).
I want mavlink-server to forward the MAVLink data to 4 separate UDP endpoints.

I’ve configured the Pi to start up mavlink-server on boot with this command:
mavlink-server serial:/dev/ttyS0:115200 udpclient:<IP_ADDRESS>:11999 udpclient:<IP_ADDRESS>:10001 udpclient:<IP_ADDRESS>:10002 udpclient:<IP_ADDRESS>:21999

The issue:

When I power cycle the device, the Cube and Pi boot up quickly, but the 4G/VPN routers take a minute or two to get online.
Once the routers are online (confirmed they’re up), mavlink-server doesn’t reconnect to my GCS everytime sometimes one endpoint connects and other doesn't.
If I manually restart mavlink-server, it immediately connects all endpoints and works fine. I have attached 2 logs 00 was on power cycle where one connection was successful and other wasn't, and log 01 was after restart mavlin-server service which worked fine.

I understand I can write a script to wait for the 4G to come online before starting mavlink-server, but I’m curious if there’s any built-in behavior for retrying connections that I might be missing. Or is there a setting I should configure to handle this better?
Also, I’ve tested the exact same setup with MAVProxy using the following command:
mavproxy.py --master=/dev/ttyS0:115200 --out <IP_ADDRESS>:11999 --out <IP_ADDRESS>:10001 --out <IP_ADDRESS>:10002 --out <IP_ADDRESS>:21999

The same behavior persists with MAVProxy.

@patrickelectric did you had time to look into this issue?

I just did a simple test and it appears to be working, I connected to another mavlink-server running on the vehicle, and disconnected my ethernet cable and reconnected back again, and it works.
@loki077 can you provide an example between two machines where the issues is replicable ? Also, try the master version if possible.

maybe try setting the network interface down?

Hey @patrickelectric,
I’ve set up a Mavlink server on an RPi where the Cube is connected over serial. The RPi is connected to two routers via a switch.
RPi IP: <IP_ADDRESS>
I just installed the Mavlink server from the master branch and used the binaries from CI. However, I’m running into CRC failures when I execute the following:
mavlink-server serial:/dev/ttyS0:115200 udpclient:<IP_ADDRESS>:14550

Here’s the output log for reference:
 mavlink-server serial:/dev/ttyS0:115200 udpclient:<IP_ADDRESS>:14550
2024-11-13T07:01:23.132009Z  INFO main ThreadId(01) src/lib/logger.rs:172: mavlink-server, version: 0.1.0-9aca3175d3c313d7eab417b2506691bdecc740ea, build date: 2024-12-20
2024-11-13T07:01:23.132404Z  INFO main ThreadId(01) src/lib/logger.rs:179: Starting at 2024-11-13T07:01:23
2024-11-13T07:01:23.132592Z DEBUG main ThreadId(01) src/lib/logger.rs:183: Command line call: mavlink-server serial:/dev/ttyS0:115200 udpclient:<IP_ADDRESS>:14550
2024-11-13T07:01:23.132840Z DEBUG main ThreadId(01) src/lib/logger.rs:184: Command line input struct call: Args {
    endpoints: [
        Serial {
            name: ArcSwapAny(
                "Serial",
            ),
            uuid: 74201462-a8c7-5349-aa0b-d3e641a0149b,
            port_name: "/dev/ttyS0",
            baud_rate: 115200,
            on_message_input: Callback,
            on_message_output: Callback,
            stats: RwLock {
                data: AccumulatedDriverStats {
                    name: "Serial",
                    driver_type: "Serial",
                    stats: AccumulatedDriverStatsInner {
                        input: None,
                        output: None,
                    },
                },
            },
        },
        UdpClient {
            remote_addr: "<IP_ADDRESS>:14550",
            name: ArcSwapAny(
                "UdpClient",
            ),
            uuid: 2a8a8950-e52e-5201-8587-ab14f30eef90,
            on_message_input: Callback,
            on_message_output: Callback,
            stats: RwLock {
                data: AccumulatedDriverStats {
                    name: "UdpClient",
                    driver_type: "UdpClient",
                    stats: AccumulatedDriverStatsInner {
                        input: None,
                        output: None,
                    },
                },
            },
        },
    ],
    verbose: false,
    web_server: <IP_ADDRESS>:8080,
    enable_tracing_level_log_file: false,
    log_path: Some(
        "./logs",
    ),
    streamreq_disable: true,
    udp_server_timeout: 10,
    mavlink_system_id: 1,
    mavlink_component_id: 191,
    mavlink_heartbeat_frequency: 1.0,
    send_initial_heartbeats: false,
    mavlink_version: 2,
    default_api_version: 1,
}
2024-11-13T07:01:23.138368Z  INFO                 main ThreadId(01) src/lib/web/mod.rs:32: Running web server on address <IP_ADDRESS>:8080
2024-11-13T07:01:23.143455Z ERROR tokio-runtime-worker ThreadId(02) src/lib/drivers/generic_tasks.rs:64: Failed to decode packet: InvalidCRC { expected_crc: 9760, calculated_crc: 40210 }
2024-11-13T07:01:23.153157Z ERROR tokio-runtime-worker ThreadId(02) src/lib/drivers/generic_tasks.rs:64: Failed to decode packet: InvalidCRC { expected_crc: 3693, calculated_crc: 29848 }
2024-11-13T07:01:23.156370Z ERROR tokio-runtime-worker ThreadId(02) src/lib/drivers/generic_tasks.rs:64: Failed to decode packet: InvalidCRC { expected_crc: 39807, calculated_crc: 28336 }
2024-11-13T07:01:23.242804Z ERROR tokio-runtime-worker ThreadId(02) src/lib/drivers/generic_tasks.rs:64: Failed to decode packet: InvalidCRC { expected_crc: 41820, calculated_crc: 60403 }
2024-11-13T07:01:23.274552Z ERROR tokio-runtime-worker ThreadId(02) src/lib/drivers/generic_tasks.rs:64: Failed to decode packet: InvalidCRC { expected_crc: 10073, calculated_crc: 64807 }
2024-11-13T07:01:23.274629Z ERROR tokio-runtime-worker ThreadId(02) src/lib/drivers/generic_tasks.rs:64: Failed to decode packet: InvalidCRC { expected_crc: 22909, calculated_crc: 22472 }
2024-11-13T07:01:23.279414Z ERROR tokio-runtime-worker ThreadId(02) src/lib/drivers/generic_tasks.rs:64: Failed to decode packet: InvalidCRC { expected_crc: 25087, calculated_crc: 43792 }
2024-11-13T07:01:23.282018Z ERROR tokio-runtime-worker ThreadId(02) src/lib/drivers/generic_tasks.rs:64: Failed to decode packet: InvalidCRC { expected_crc: 19585, calculated_crc: 57929 }

Interestingly, when I use MavProxy, everything works fine. Command:
mavproxy.py --master=/dev/ttyS0:115200 --out <IP_ADDRESS>:14550

Output from MavProxy:
mavproxy.py --master=/dev/ttyS0:115200 --out <IP_ADDRESS>:14550
Connect /dev/ttyS0 source_system=255
Log Directory:
Telemetry log: mav.tlog
Waiting for heartbeat from /dev/ttyS0
MAV> Detected vehicle 1:1 on link 0
online system 1
RTL> Mode RTL
AP: GCS Failsafe Off

Any ideas why the Mavlink server setup is failing while MavProxy works perfectly? Appreciate any pointers!
Rpi Os:
uname -m
armv7l

Installed from CI
'mavlink-server-armv7-unknown-linux-musleabihf'

Hi @loki077 can you get us the following log?
RUST_LOG=debug,mavlink_codec=trace mavlink-server serial:/dev/ttyS0:115200 udpclient:<IP_ADDRESS>:14550 --enable-tracing-level-log-file

This should allow us to probe the protocol decoding, handled by mavlink-codec.
Also, could you upload the .tlog file from mavproxy? If mavproxy succeeded in decoding those messages, we can then use that as mavlink-server input and reproduce your problem.
Another way of reproducing the problem would be to use mavproxy to connect to serial, make mavlink-server connect to mavproxy via TCP or udp, and check if the errors happen.
Thanks


https://developer.mozilla.org/en-US/docs/Learn/JavaScript/First_steps/What_is_JavaScript

Request type

[ ] Please close this issue, I accidentally submitted it without adding any details
[ ] New documentation
[x] Correction or update
Details

The example code in the section External JavaScript is not correct. The script is loaded and executed asychronously (with async). Therefore the button might not have been added to the DOM when the following line is exsecuted: const buttons = document.querySelectorAll('button');
It is easy to force this situation if the DOM loading is blocked by an alert message:
  <body>
    <script>
      alert("Delay parsing HTML...");
    </script>
    <button>Click me</button>
  </body>

In my opinion, the correct solution would be to use defer instead of async.

Might be an issue, yes. I've updated it. Thanks!


Trigger a reconfiguration, on all cmake files change

This change addresses item #1911
This changes visible behavior
The purpose of this change
When any CMakeLists.txt or **.cmake file is changed in the project, it triggers a reconfiguration

this will trigger configure even when the toolchain files are modified.


Airdrop issue

Hello! After I have registered, confirmed and everything I just open my wallet and start to add the paper keys, after that I just pressed the bottom "become a lucky airdropee" but I still receive the message "you are not qualified" I added a lot of info, got all the three point ("paper keys, verified phone number and beautiful smile") but still nothing 😔😔 someone can help??

Das gleiche Problem keiner kann helfen was können wir machen

Ich meinte keiner kann helfen

keiner kann helfen

Ps: name of account: dariopascale

hey my account  name danialafridi i add everything but still red cross on mobile phone number  my number is verified but still not qualified and red cross iam seeing

Your country is not qualified as Canada for example, I have all the three cross but still i"m not able to join my account name is dariociccio

Hello, my account name is bashkeybase, I did everything, I did not use any not or even try to cheat system, but I was not lucky
Does it mean that I cannot never join or earn?
My country is Nigeria


End of life Python SDK

There are more than 10M Python developers in the world.
I am one of them and would like to use the Python SDK to build a Teams bot.
Could you please reassess the decision to stop support for the Python SDK by Nov 2023?

Thank you for your feedback, we will share the feedback internally.

@zurferr - We have the following update from the engineering team - "Bot Framework SDK is not currently reconsidering the Python SDK deprecation".

We are closing this issue for now, please feel free to reopen it, if required.


[SPARK-18278] [Scheduler] Spark on Kubernetes - Basic Scheduler Backend

What changes were proposed in this pull request?
This is a stripped down version of the KubernetesClusterSchedulerBackend for Spark with the following components:

Static Allocation of Executors
Executor Pod Factory
Executor Recovery Semantics

It's step 1 from the step-wise plan documented here.
This addition is covered by the SPIP vote which passed on Aug 31 .
How was this patch tested?

The patch contains unit tests which are passing.
Manual testing: ./build/mvn -Pkubernetes clean package succeeded.
It is a subset of the entire changelist hosted in http://github.com/apache-spark-on-k8s/spark which is in active use in several organizations.
There is integration testing enabled in the fork currently hosted by PepperData which is being moved over to RiseLAB CI.
Detailed documentation on trying out the patch in its entirety is in: https://apache-spark-on-k8s.github.io/userdocs/running-on-kubernetes.html

cc @rxin @felixcheung @mateiz (shepherd)
k8s-big-data SIG members & contributors: @mccheah @ash211 @ssuchter @varunkatta @kimoonkim @erikerlandson @liyinan926 @tnachen @ifilonenko

Can one of the admins verify this patch?

Jenkins, ok to test

Test build #82614 has started for PR 19468 at commit f6fdd6a.

Test build #82614 has finished for PR 19468 at commit f6fdd6a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82614/
Test PASSed.

@vanzin @srowen @mridulm @mgummelt

+ @susanxhuynh @ArtRand (both from Mesosphere). I will also have a look.

@skonto, there was some discussion about this on the SPIP. We see them as separate and independent issues with the pluggable API being a long term goal.  It would involve a working group of people drawn from all cluster manager integration maintainers - because the changes are extensive and require scalability/performance testing in a variety of environments - probably over 2-3 releases before we can iron out the kinks.
In the short term however, adding the package here enables supporting the growing number of K8s users of Spark that currently rely on our fork, and the integration testing makes us confident about not impacting the release process or adding complexity to the maintainers workflow. The K8s community will set up all the requisite testing, and ensure health of this package. The integration testing would also carry forward to when we have pluggable APIs in the future.

iron out the kinks

A large chunk of the difficulty in identifying and ironing out kinks in such a project is the difficulty of writing adequate tests of the scheduler code. I'd expect test coverage to take roughly the same amount of effort as all of the rest of the scheduler plug-in effort.

Test build #82854 has started for PR 19468 at commit 75e31a9.

Test build #82854 has finished for PR 19468 at commit 75e31a9.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82854/
Test FAILed.

Test build #82856 has started for PR 19468 at commit cf82b21.

cc @JoshRosen @rxin @cloud-fan @sameeragarwal @jiangxb1987

Test build #82856 has finished for PR 19468 at commit cf82b21.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82856/
Test PASSed.

Test build #82886 has started for PR 19468 at commit 82b79a7.

@vanzin, thank you for reviewing - I've addressed some of the style and organization related issues. You're right in that the use of YARN internal constants may not make sense here - we'll be discussing this in our community meeting and addressing that shortly.

Test build #82886 has finished for PR 19468 at commit 82b79a7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82886/
Test PASSed.

Test build #82938 has started for PR 19468 at commit c052212.

Test build #82940 has started for PR 19468 at commit c565c9f.

@vanzin, you were right, the YARN constants were left overs and made no sense wrt k8s. We discussed it in our weekly meeting - it was simply dead code. I've addressed most of the style comments and the major concern about the constants. It's ready for a more in-depth review.

Test build #82938 has finished for PR 19468 at commit c052212.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82938/
Test PASSed.

Test build #82940 has finished for PR 19468 at commit c565c9f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82940/
Test PASSed.

Merged build finished. Test PASSed.

Test build #83057 has started for PR 19468 at commit 2fb596d.

Test build #83060 has started for PR 19468 at commit 992acbe.

Test build #83060 has finished for PR 19468 at commit 992acbe.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83060/
Test FAILed.

Test build #83061 has started for PR 19468 at commit b0a5839.

Test build #83061 has finished for PR 19468 at commit b0a5839.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83061/
Test FAILed.

Test build #83062 has started for PR 19468 at commit a4f9797.

Test build #83057 has finished for PR 19468 at commit 2fb596d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83057/
Test FAILed.

Test build #83062 has finished for PR 19468 at commit a4f9797.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83062/
Test FAILed.

Test build #83100 has started for PR 19468 at commit 2b5dcac.

Test build #83100 has finished for PR 19468 at commit 2b5dcac.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83100/
Test PASSed.

Test build #83309 has started for PR 19468 at commit 4b32134.

@foxish @mridulm Heads up - since the last review iteration, I wrote an extra test in KubernetesClusterSchedulerBackend that exposed a bug where if executors never register with the driver but end up in the error state, the driver doesn't attempt to replace them in subsequent batches. Test case is here. The fix is to ensure that all executors that hit an error state in our watch are accounted as "disconnected" executors, regardless if they were ever marked as disconnected by the driver endpoint otherwise - see here.
Additionally, I was able to remove one of the data structures which mapped executor pod names to executor IDs. Instead, whenever we get a Pod object, we can look up its ID via the executor's ID label. This makes it such that we don't have to worry about as much state.

Test build #83425 has started for PR 19468 at commit 6cf4ed7.

Test build #83425 has finished for PR 19468 at commit 6cf4ed7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83425/
Test PASSed.

@mridulm @jiangxb1987 @vanzin thanks for taking the time to review this. Is there another round of reviews that you guys would want to do here? We're hoping to get this and just one other PR (the client logic and sparksubmit changes) associated with this k8s mode into 2.3 (the other PR is being worked on, and will be sent upstream shortly).
A modest goal for 2.3 would be to enable just static executors on Kubernetes - no R or Python yet, or dynamic allocation.
It would help get some usage and feedback as well. WDYT? Does that seem feasible to everyone?
Lastly, on the integration testing, there has been a lot of progress and with a little more time, we should have them sorted and enabled on RiseLabCI, (thanks to @shaneknapp!).
cc/ @rxin @mateiz @felixcheung

The plan looks good, I'm going to review another round recently, thanks!

ping @jiangxb1987

@vanzin @mridulm @jiangxb1987 https://github.com/apache/spark/pull/19717 is the second part of this PR which has the client-side and spark-submit changes. The two in combination should make a lot more sense, especially wrt some points that came up above. PTAL.
Also, let us know if it would make sense to get this merged after the next round of reviews that @jiangxb1987 was planning to do - and then move to the other PR, where the overall picture should be clearer. These two PRs represent almost all of what we planned to make available for Spark 2.3 (modulo documentation and minor enhancements). Again, thanks for the reviews!

Test build #83789 has started for PR 19468 at commit 1f271be.

Test build #83800 has started for PR 19468 at commit 71a971f.

Test build #83789 has finished for PR 19468 at commit 1f271be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83789/
Test PASSed.

Test build #83800 has finished for PR 19468 at commit 71a971f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83800/
Test PASSed.

Test build #83830 has started for PR 19468 at commit df03462.

Test build #83830 has finished for PR 19468 at commit df03462.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83830/
Test FAILed.

Test build #83851 has started for PR 19468 at commit 0ab9ca7.

Test build #83851 has finished for PR 19468 at commit 0ab9ca7.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83851/
Test FAILed.

@mridulm It would be great if you could look at this again, thanks!

Test build #83903 has started for PR 19468 at commit 7f14b71.

Test build #83903 has finished for PR 19468 at commit 7f14b71.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

@vanzin @jiangxb1987 @mridulm We have addressed all the comments so far. Is this good to merge?

I'll let others take a look first (also because I'll be out traveling real soon...).

I am actually quite swamped right now, hence the delay in getting to this PR.
I will try to take a pass over the weekend; but if it does get merged if @vanzin is ok with it, fine by me !

@mridulm @vanzin @mateiz @rxin This is waiting for another LGTM I think (aside from the one by @jiangxb1987). If all parties have no major changes to suggest, may I propose we merge this before thanksgiving and revisit it during the review of the other PR for K8s support in 2.3?

Sounds good to me, also cc @sameeragarwal @ueshin

Test build #84083 has started for PR 19468 at commit 7afce3f.

Test build #84083 has finished for PR 19468 at commit 7afce3f.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84083/
Test FAILed.

Test build #84084 has started for PR 19468 at commit b75b413.

Test build #84084 has finished for PR 19468 at commit b75b413.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84084/
Test FAILed.

retest this please
Failure looks unrelated.

retest this please

Test build #84088 has started for PR 19468 at commit b75b413.

Test build #84088 has finished for PR 19468 at commit b75b413.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84088/
Test PASSed.

Test build #84096 has started for PR 19468 at commit 3b587b4.

Test build #84096 has finished for PR 19468 at commit 3b587b4.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84096/
Test FAILed.

Merged build finished. Test FAILed.

Test build #84114 has started for PR 19468 at commit e5a6a67.

Test build #84116 has started for PR 19468 at commit cb12fec.

@reviewers, just as a PSA we have to rebase the second PR (containing the submission client) after this one merges, and then we can submit the second one against this upstream. This is a series of large PRs and so review needs to be thorough; but if/when reviewer comments are addressed, this one blocks the following one.

Test build #84114 has finished for PR 19468 at commit e5a6a67.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84114/
Test PASSed.

Merged build finished. Test PASSed.

Test build #84116 has finished for PR 19468 at commit cb12fec.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84116/
Test PASSed.

Hi! I think we have quorum at this point.
Not sure if folks are around to review during the holiday, but I plan to do another final round within a day or so, and merge unless anyone has any concern then.

I went through the changes to make sure the non-k8s changes are ok. They do look ok to me. From that perspective, LGTM.

Test build #84227 has started for PR 19468 at commit ae396cf.

Test build #84228 has started for PR 19468 at commit f8e3249.

Test build #84230 has started for PR 19468 at commit a44c29e.

Test build #84227 has finished for PR 19468 at commit ae396cf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84227/
Test FAILed.

Test build #84228 has finished for PR 19468 at commit f8e3249.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84228/
Test FAILed.

Merged build finished. Test FAILed.

Test build #84233 has started for PR 19468 at commit 4bed817.

Test build #84234 has started for PR 19468 at commit c386186.

Test build #84230 has finished for PR 19468 at commit a44c29e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84230/
Test PASSed.

Test build #84233 has finished for PR 19468 at commit 4bed817.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84233/
Test PASSed.

Merged build finished. Test PASSed.

Test build #84234 has finished for PR 19468 at commit c386186.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84234/
Test PASSed.

I see the latest changes are backported back to the fork - thanks!  it's useful to do since I think we get more comprehensive test coverage there, for now.

I think the other cases, mesos, yarn/docker are quite different as they are one of the possible environment/setup, whereas in this case there is no other alternative - docker image must be there.
Anyway I think we should proceed as-is and revisit if it makes sense.

Test build #84265 has started for PR 19468 at commit b85cfc4.

@felixcheung @mridulm All the comments have been addressed. Is this good to merge?

Test build #84265 has finished for PR 19468 at commit b85cfc4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84265/
Test PASSed.

Merged build finished. Test PASSed.

Thanks for the changes, and all the great work !
The pr LGTM for merge.
@rxin and @felixcheung have already +1'ed it.
We can leave it open for a day or so in case others want to do a last pass over it before merging it in.
+CC @vanzin @ jiangxb1987 @markhamstra - any additional comments ?

LGTM, thanks for the awesome work!

Thanks - merging in master!

For future pull requests, can you create subtasks under https://issues.apache.org/jira/browse/SPARK-18278 ?

Done, created subtasks.
Thanks everyone for helping review this PR!

Thanks everyone and congrats!


Move env! to env::var_os for buck determinism

When building pyo3 with buck2 and remote execution, we're especially sensitive to deterministic builds and being able to compile different rust libraries on different hosts in parallel.
The issue with resolve_cross_compile_config_path was that we were embedding the full path of OUT_DIR into the compiled rlib and we are unable to guarantee the path is identical across all invocations of rustc. The other instances of env!("OUT_DIR") are no issue as they are used as part of the larger expression include_str!(concat!(env!("OUT_DIR"), "/pyo3-build-config-file.txt")); and we are able to guarantee that the contents of those files are deterministic.
By switching to env::var_os("OUT_DIR"), we are able to avoid embedding absolute paths into the compiled rust library and maintain backwards compatibility with the rest of the cargo ecosystem.

Looks like I need to update a changelog https://github.com/PyO3/pyo3/actions/runs/11042056229/job/30673632736?pr=4578

Related: https://github.com/PyO3/pyo3/pull/4555

Thanks very much for the PR! I think you're right that there's a problem here, but I worry that this isn't the complete solution. I'll write up my observations in an issue now...

See #4579

Given that there's a more complex problem here, which I don't think this solves, I will close this PR. Let's please continue figuring out what to do on https://github.com/PyO3/pyo3/issues/4579.


File permissions changing inside the sandbox

See original issue on GitLab
In GitLab by [Gitlab user @jennis] on Mar 15, 2018, 18:09
The issue
When writing the integration test (in bst-external) for the dpkg-build plugin, in order for the test to pass, debian/rules must be built.
Now, locally executing bst build dpkg-build/dpkg-build-test.bst resulted in a successful build. However, when this command is launched in the sandbox, using the following test:
https://gitlab.com/BuildStream/bst-external/blob/convert-to-new-style-integration-tests/tests/dpkg-build.py#L14
The test fails due to a permissions change of the debian/rules file, see failed job here:
https://gitlab.com/BuildStream/bst-external/-/jobs/57647277
A quick fix
This can be fixed by using chmod +x <file> to explicitly change the permission of the file and prepending this as a build command, like so:
https://gitlab.com/BuildStream/bst-external/blob/convert-to-new-style-integration-tests/tests/project/elements/dpkg-build/dpkg-build-test.bst#L15
The problem
Obviously this solution is not practical for a project with many .bst files.

In GitLab by [Gitlab user @jennis] on Mar 16, 2018, 11:03
mentioned in merge request bst-external!22

In GitLab by [Gitlab user @tristanvb] on Mar 22, 2018, 09:09

https://gitlab.com/BuildStream/bst-external/blob/convert-to-new-style-integration-tests/tests/dpkg-build.py#L14

Broken link.

https://gitlab.com/BuildStream/bst-external/blob/convert-to-new-style-integration-tests/tests/project/elements/dpkg-build/dpkg-build-test.bst#L15

Broken link.

When writing the integration test (in bst-external) for the dpkg-build plugin, in order for the test to pass, debian/rules must be built.

Ummm, okaaaay, what do you mean "Must be built" ? As I vaguely recall, the dpkg-deploy plugin takes care of building some debian metadata out of existing public data, but this is not the dpkg-build plugin, which expects a repository with an existing debian/ subdir.
So... are we talking about an integration test which has to create a "something with a debian/ subdir" in order to pass a test ?
If so, isn't it exactly the right expectation that the creator of the debian/rules file is responsible for setting the executable bit on it ?

Now, locally executing bst build dpkg-build/dpkg-build-test.bst resulted in a successful build. However, when this command is launched in the sandbox, using the following test:

This indicates that there must be a problem somewhere, possibly in CI, with these broken links I dont think this report succeeds in explaining exactly what the problem is.
Closing this with NEEDINFO.

In GitLab by [Gitlab user @tristanvb] on Mar 22, 2018, 09:09
closed


Add topics and visibility to Repo Stream

Adding a few fields to the repo stream:

topics
language
visibility

Ready to merge :) @edgarrmondragon or @aaronsteers could you restart the tests please?


Bryan: Moved fun page to about me; fixed title

Fun page gone (migrated to about me)
Title in projects looks better
Commented out settings options that weren’t implemented
TEST=manual
Change-Id: I8490b21915f1229c493586ba870b1cab377edf7d

Code Review comments can be found at https://gerrithub.io/q/I8490b21915f1229c493586ba870b1cab377edf7d


Using bitwise operators to print binary of a number in C++

🚀 Feature
Using bitwise operators to print binary of a number in C++
Have you read the Contributing Guidelines on Pull Requests?
YES

I would like to contribute to the above problem in CPP. am a GSSoC'21 participant. Kindly assign this to me.

/assign


chore(doc): remove redundant punctuation marks

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?

Checklist

[ ] I have written necessary rustdoc comments
[ ] I have added necessary unit tests and integration tests
[ ] I have added test labels as necessary. See details.
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[ ] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Thanks for your contribution.

In British English, “i.e.” and “e.g.” are not followed by a comma, so the first example above would be:

They sell computer components, e.g. motherboards, graphic cards, CPUs.

Virtually all American style guides recommend to follow both “i.e.” and “e.g.” with a comma (just like if “that is” and “for example” were used instead), so the very same sentence in American English would become:

They sell computer components, e.g., motherboards, graphic cards, CPUs.

https://jakubmarian.com/comma-after-i-e-and-e-g/
Therefore, I think it's find to put a punctuation mark here.

Thank you for your patient explanation, which helped me learn new knowledge.
closing


Signed in with details and updated index.html

@wiljnich   thanks for making your first contribution, I will be merging your Pull Request, also look at the issues so that you can get a chance to win a limited community Swag from me.
If you like the project and Repo please give it a star and Follow me on Github (would be appreciated)


how to set php.validate.executablePath if I want to use php on Windows Subsystem for Linux (WSL)

I am trying WSL and vscode. I would like to use php that I installed on wsl  with vs code. I used this solution to add php into windows path but I am not sure how set php.validate.executablePath in this scenario. is this possible? can anyone help please?
Thanks.

If it doesn't work with that fix to add bash php to your path, then I don't think it will work. But that workaround is basically what I think you'll have to do - tell bash to invoke php with some parameters. I don't really know what those parameters are though, I'm not fluent in batch script.

Closing as possibly resolved / no recent followup


Do not pass lambdas to the backend in case of basic GroupBy aggregation

Current GroupBy implementation is still passing python lambdas to the backend to do simple aggregation (for example mean):
https://github.com/modin-project/modin/blob/89f6cdde56abf5dcf9561ea6721fb31f84a855d5/modin/pandas/groupby.py#L128-L129
These lambdas are not processable by non-python engines, and so even if the backend does support particular aggregation it can't be executed because it's passed in a python function format. At the moment, all of those lambdas are being passed into a single method QueryCompiler.groupby_agg.

@devin-petersohn what do you think would be the most appropriate way to resolve the lambdas problem:

We may continue to pass the aggregation method into a single groupby_agg backend function but as a string function name.
We may create separate query compiler methods for all of them, like we already did for the GroupBy reduce aggregations (groupby_sum, groupby_count... and we'd also add groupby_mean, groupby_std, groupby_median...)

+1 thanks @dchigarev, this is not efficient or ideal. I prefer to have some more explicit control in the query compiler. The hard part is that we end up with a huge bloat in the query compiler API if we explicitly have each API defined individually.
Maybe we need a better GroupByFunction to avoid the bloat? It is probably complementary with option 2 above.

@devin-petersohn
Not sure that I understand what you mean under "better GroupByFunction", is it about those template functions that we store in data_management/functions? If so, then I don't get how we could avoid extraordinary size of the query compiler API with a better GroupByFunction. The way I see your wish is the following:
groupby_mean = GroupByFunction.register(pandas.GroupBy.mean)
groupby_nunique = GroupByFunction.register(pandas.GroupBy.nunique)
groupby_std = GroupByFunction.register(pandas.GroupBy.std)
...

In this approach, the problem is still unsolved: query compilers have a huge mixed API and potentially enormous size. The problem of having a huge group of methods related to the different high-level objects, but located in a single query compiler class is not groupby specific: we have the same issue with the resample_*, rolling_*, str_*, and dt_* methods. Bringing groupby_* methods to the query compiler will no create any new issues, it'll just highlight the existed one.
In a long term, I think, we should start thinking about splitting query compiler somehow. I think we could move the logic of actual object_* functions into different object-specific classes, something like: QueryCompilerGroupByMethods, QueryCompilerWindowMethods, and so on. Old query compiler object_* methods may be squeezed into just groupby, resample, window ..., that would be dispatchers of the QueryCompiler$ObjectMethods methods.
As a solution particularly for the #3197 issue I suggest expanding the query compiler's API with such methods:
def groupby_mean(self, **kwargs):
    return self.groupby_agg(agg_func="mean", **kwargs)

def groupby_nunique(self, **kwargs):
    return self.groupby_agg(agg_func="nunique", **kwargs)
...

adding them (about 15 new methods) to the BaseQueryCompiler would allow us to not modify other query compilers at all. All of the groupby methods continue to work naturally via groupby_agg. At the same time, if a backend needs more control for some specific method it could just overwrite it.

We likely want to keep QC API as clean as possible and all the implementation should go to a different location. We could move those implementation to kernels folder after #2957 is merged. Something like this:
.pandas/
   query_compiler.py
   .kernels/
      .dataframe.py
      .groupby.py
      .resample.py
      .etc.

Firstly I would like to see #3537 and #3562 merged into master. After that we can move forward more gradually and introduce more granular changes in the code and docs.

I think the problem we're discussing here (slice QC into several objects) is a bit different from the original one, IMO they could and should be solved independently.
Splitting the Query Compiler is a right thing, but I think it's much more complicated, and what's important, much longer to implement, rather than passing backend-agnostic information about groupby aggregation (via function's string name or by calling a specific QC method) in the existing paradigm. The fact of defaulting to pandas for most of the OmniSci's groupby aggregation just because the type of aggregation is passed as python lambdas is hurting, and we would like to fix this as soon as possible. I propose we shake this issue a bit and try to come up with some solution that would conform to everyone.
Just reminding the current status:
There was proposed two options of how we could provide to a backend non-python and non-pandas dependent information about an aggregation function:

Pass string function names to the QC.groupby_agg instead of python lambdas. The con of this approach is that it's hard to specialize the implementation of some concrete aggregation at the QC level (it would require a decent if-else tree to determine whether the passed operation is specialized)
Introduce new 15 QC methods named like QC.groupby_{fn_aggregation_name} and invoke aggregation through them (the same we do for groupby.size, groupby.sum, groupby.count ... and other MapReduce implemented fns). The con of this approach is that it will bloat Query Compiler's API.

Currently, there is an open PR (#3373) implementing the second approach. The PR was postponed until we will figure out how to avoid QC API bloating, however, we still haven't progressed much in it.
What I want from @modin-project/modin-core, @modin-project/modin-contributors, and @modin-project/modin-omnisci is to help to pick one of two proposed approaches or propose a new one. If we would agree on the second of the proposed approaches, I will make an effort to revive #3373 and make it ready for review (currently it's much outdated).

Many of our current groupby aggregations follow mapreduce or treereduce style of computation patterns. Many of the aggregations not yet implemented for groupby cannot follow this pattern. The new algebraic operator definitions in #3717 will help here a bit. groupby takes an operator (reduce, map, treereduce, mask, etc.). This is sufficient to support all of the groupby operators.
My vote is for (2) in @dchigarev's list here. The query compiler needs to be split, completely agree, but also agree with @dchigarev that it's orthogonal to the current discussion and can be done separately. Option (2) would allow us to individually define the groupby + operator pairing and let us optimize each independently. Other thoughts?

Yes @vnlitvinov let's do that.


Support for a bug report or sysdump in calicoctl

We currently have a troubleshooting guide here that requires us to run several commands separately. It would be great if we could have a feature that allows us to run a single command that can generate a bug report or sysdump for offline troubleshooting. For example, something similar to what cilium offers here

So something like this?  https://docs.tigera.io/calico/latest/reference/calicoctl/node/diags

I've added the enhancement label, because it sounds like you'd want it to do more than it currently does.  Also "good first issue", because it feels like adding more diags to this existing function in calicoctl is a nice starter task.

Thank you. Exactly something like on top of node diag.
I actually played with node diag its good but its limited to a node. It will be great if we have a package which is cluster level e.g. cluster diag and that has below features.

accept array of nodes to collect node diags from all the nodes
run all the kubernetes commands from here as well https://docs.tigera.io/calico/latest/operations/troubleshoot/
some level of possible filtering to limit the logs
tldr, the cluster diag package will contain multiple node diag plus useful kubernetes commands to check the overall health of cluster.


Replace moment.js by the standard Date API of Javascript

resolves #151

The version breakdown is not correct.
If you notice, without these changes the midnight tick shows just the date in a short way:


bind to port failed (Cannot assign requested address)

I read item #298 and it says this issue is fixed in version 79.
I am running a Raspberry pi 4b 2GB with Raspian lOS lite, and I still get the issue with the daemon not starting. I've checked but nothing is running on port 8888 already, and I manually downloaded and installed the latest version (79), several reboots and manual attempts at starting the service and running it on the command line and this is what I get (the simplest script to reproduce)
pi@asciipi:~ $ /usr/bin/pigpiod -v
79
pi@asciipi:~ $ sudo /usr/bin/pigpiod -l
pi@asciipi:~ $ 2023-01-29 12:36:42 initInitialise: bind to port 8888 failed (Cannot assign requested address)
Can't initialise pigpio library
pi@asciipi:~ $ netstat | grep 8888
pi@asciipi:~ $

What am I missing or what have I not read?

Should work.  Can you provide details on the OS, how you installed it.  Hardware revision too.

Have you tried leaving off -l

I was originally excited there wasa lock file... but removing it didn't help :(

pi@asciipi:~ $ sudo /usr/bin/pigpiod
2023-01-29 19:55:14 initInitialise: Can't lock /var/run/pigpio.pid
Can't initialise pigpio library
pi@asciipi:~ $ ls -als  /var/run/pigpio.pid
4 -rw-r--r-- 1 root root 5 Jan 29 19:55 /var/run/pigpio.pid
pi@asciipi:~ $ sudo rm /var/run/pigpio.pid
pi@asciipi:~ $  sudo /usr/bin/pigpiod
pi@asciipi:~ $ 2023-01-29 19:56:19 initInitialise: bind to port 8888 failed (Address already in use)
Can't initialise pigpio library
pi@asciipi:~ $ !ls
ls -als  /var/run/pigpio.pid
ls: cannot access '/var/run/pigpio.pid': No such file or directory
pi@asciipi:~ $

OS is the latest Raspian lite installed through the Raspberry PI imager. This is the general first login path I take:

sudo apt update -y
sudo apt full-upgrade -y
sudo apt autoremove -y
sudo rpi-eeprom-update -a -d
cat >/tmp/boot.conf <<EOF
[all]
BOOT_UART=0
WAKE_ON_GPIO=1
ENABLE_SELF_UPDATE=1
BOOT_ORDER=$BOOT
EOF
sudo rpi-eeprom-config --apply /tmp/boot.conf
sudo apt install -y lsb-release apt-transport-https ca-certificates git python3-dev python3-pip python3-pil automake autoconf pkg-config libcurl4-openssl-dev libjansson-dev libssl-dev libgmp-dev make g++ tor screen xserver-xorg x11-xserver-utils xinit openbox chromium-browser xserver-xorg-input-evdev libhdf5-dev libhdf5-serial-dev python3-pyqt5 libatlas-base-dev libjasper-dev libgtk2.0-dev pkg-config samba samba-common-bin cups build-essential libcups2-dev libcupsimage2-dev python3-serial pigpio python3-pigpio
sudo pip install --upgrade pip setuptools wheel opencv-python qrcode[pil] pigpio

I have also installed the zj-58 printer drivers for a thermal printer I have attached

I have not played with rpi-eeprom-update.   Maybe something new  there.
So you start with lite OS but set it up as a desktop?  Are you using Systemd?  If a fresh install, pigpiod should be disabled.

There is no desktop, I launch chromium browser though. I do not actively use systemd, but it may be there and doing it's thing. I believe I used it to enable the service at boot as per the instructions.

Which service, pigpiod?

The instructions I followed said to enable the daemon on startup, so I did that. It fails.
When I check the status, it's failed, and then all the rest above
pi@asciipi:~ $ sudo service pigpiod status
● pigpiod.service - Daemon required to control GPIO pins via pigpio
     Loaded: loaded (/lib/systemd/system/pigpiod.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2023-01-29 12:35:53 GMT; 18h ago
    Process: 1064 ExecStart=/usr/bin/pigpiod -l (code=exited, status=0/SUCCESS)
   Main PID: 1065 (code=exited, status=1/FAILURE)
        CPU: 62ms

Jan 29 12:35:53 asciipi systemd[1]: Starting Daemon required to control GPIO pins via pigpio...
Jan 29 12:35:53 asciipi systemd[1]: Started Daemon required to control GPIO pins via pigpio.
Jan 29 12:35:53 asciipi pigpiod[1065]: 2023-01-29 12:35:53 initInitialise: bind to port 8888 failed (Cannot assign requested address)
Jan 29 12:35:53 asciipi pigpiod[1065]: Can't initialise pigpio library
Jan 29 12:35:53 asciipi systemd[1]: pigpiod.service: Main process exited, code=exited, status=1/FAILURE
Jan 29 12:35:53 asciipi systemd[1]: pigpiod.service: Failed with result 'exit-code'.
pi@asciipi:~ $

Yep, pigs returns the current version
pi@asciipi:~ $ pigs pigpv
79
pi@asciipi:~ $

Try disabling the service, reboot, then run the daemon from command line.  The fact that pigs returned the version indicates the service was already running, in which case trying to run the daemon again will fail.

So I have done another few clean installs of Raspian OS lite 32 bit, update, upgrade etc, and I still got a failure of the service. and my code still throws an exception and the service won't start. I was about to give up and then I realised what you actually said, that if the version was returned it was working. So I tried again on the new install and it fails again.
pi@asciipi:/webroot/asciipi $ pigs pigpv
socket connect failed
pi@asciipi:/webroot/asciipi $ pigs HWVER
socket connect failed

Due to it being port thing, I thought I'd give a change of port a go...
pi@asciipi:/webroot/asciipi $ sudo /usr/bin/pigpiod -l -p 8887
pi@asciipi:/webroot/asciipi $ 2023-01-31 07:09:05 initInitialise: bind to port 8887 failed (Cannot assign requested address)
Can't initialise pigpio library

pi@asciipi:/webroot/asciipi $ sudo /usr/bin/pigpiod -l -p 8886
pi@asciipi:/webroot/asciipi $ 2023-01-31 07:09:11 initInitialise: bind to port 8886 failed (Cannot assign requested address)
Can't initialise pigpio library

pi@asciipi:/webroot/asciipi $ sudo /usr/bin/pigpiod -l -p 8885
pi@asciipi:/webroot/asciipi $ 2023-01-31 07:09:16 initInitialise: bind to port 8885 failed (Cannot assign requested address)
Can't initialise pigpio library

pi@asciipi:/webroot/asciipi $

The reason I'm using the -l is because the EXECSTART in the service definition says it uses it
pi@asciipi:/webroot/asciipi $ sudo service pigpiod status
● pigpiod.service - Daemon required to control GPIO pins via pigpio
     Loaded: loaded (/lib/systemd/system/pigpiod.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2023-01-31 06:58:55 GMT; 14min ago
    Process: 2136 ExecStart=/usr/bin/pigpiod -l (code=exited, status=0/SUCCESS)
   Main PID: 2137 (code=exited, status=1/FAILURE)
        CPU: 71ms

Jan 31 06:58:55 asciipi systemd[1]: Starting Daemon required to control GPIO pins via pigpio...
Jan 31 06:58:55 asciipi systemd[1]: Started Daemon required to control GPIO pins via pigpio.
Jan 31 06:58:55 asciipi pigpiod[2137]: 2023-01-31 06:58:55 initInitialise: bind to port 8888 failed (Cannot assign requested address)
Jan 31 06:58:55 asciipi pigpiod[2137]: Can't initialise pigpio library
Jan 31 06:58:55 asciipi systemd[1]: pigpiod.service: Main process exited, code=exited, status=1/FAILURE
Jan 31 06:58:55 asciipi systemd[1]: pigpiod.service: Failed with result 'exit-code'.

As above (I had to clean up the lock file), I tried a different port but without the -l and it didn't throw an error... and it works
pi@asciipi:/webroot/asciipi $ sudo /usr/bin/pigpiod -p 8885
pi@asciipi:/webroot/asciipi $
pi@asciipi:/webroot/asciipi $ export PIGPIO_PORT=8885
pi@asciipi:/webroot/asciipi $ pigs pigpv
79
pi@asciipi:/webroot/asciipi $ pigs HWVER
11546898
pi@asciipi:/webroot/asciipi $

Since I can't get my software to change its port, I updated the service file to take off the -l and rebooted without a problem and all working now. The process I had to follow to getthis to work out of the box was this:
    sudo apt install -y pigpio python3-pigpio python3-gpiozero
    sudo find /lib/systemd/system/pigpiod.service -exec sed -i "s/\/usr\/bin\/pigpiod -l/\/usr\/bin\/pigpiod/g" '{}' \;
    sudo systemctl enable pigpiod
    sudo rm -f /var/run/pigpio.pid
    sudo systemctl start pigpiod
    sudo service pigpiod status


Can you please provide a sample on how to use OSDP.Net with secure sessions. I am able to connect to the device when secure channel is disabled. But I want to set up a secure session with the device for communication. Please provide a sample for encryptionkeyset method and Adddevice method with a securechannelkey

Can you please provide a sample on how to use OSDP.Net with secure sessions. I am able to connect to the device when secure channel is disabled. But I want to set up a secure session with the device for communication.
I enabled secure mode on the device but not able to use EncryptionKeySet on the device.
Please provide a sample on how to use encryptionkeyset method and Adddevice method with a securechannelkey

Good idea to update the Readme with a secure channel example. In your specific instance, did you establish a secure connection with the default key first? The KEYSET command can only be accepted if by the PD while the connection is secure.

Hi,
I have set the device to secure mode. Then started the connection as below:
_connectionId = panel.StartConnection(new TcpClientOsdpConnection(IPAddr, port, baudRate));
Next, I am trying to set the Encryption key set using some key as below:
byte[] bytes = Encoding.ASCII.GetBytes("Deepa123");
await panel.EncryptionKeySet(_connectionId, deviceAddress, new EncryptionKeyConfiguration(KeyType.SecureChannelBaseKey, bytes));
I am getting the exception from the above call as "Sequence contains no matching element"
Can you please let me know if the above sequence is correct? Is there anything as default key?

Two things

You need to establish a secure connection first. Use the default key if the key hasn't been set before.
The key needs to be 16 bytes in length.

Hi,
I am trying to use the default key since the key is not set before. I have used 16 bytes key as below. The device is also set to Secure Mode in the settings. When I try to use EncryptionKeySet command, I get the error - The operation has timed out. Any other hints on what could be the reason?
Below is the sample code used
byte[] bytes = new byte[]
{
0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07
};
panel.AddDevice(_connectionId, deviceAddress, true, true);
bool successfulSet = await panel.EncryptionKeySet(_connectionId, deviceAddress, new EncryptionKeyConfiguration(KeyType.SecureChannelBaseKey, bytes));

Closing because of no response


Update dependencies; notably flask 1.x

Rebased to fix conflict with requirements.txt given the addition of Flask-Limiter from the other PR.
Side-note, do you have a preference for explicit versions in requirements.txt?

do you have a preference for explicit versions

Absolutely --  otherwise you'll never know what you're gonna get when you update your packages.


REST API

Hi there, how are you going handle Ajax requests in core module?

You can use Fetch as in any other ReactNative and React application.


results don't match!

I ran link prediction (run_LinkPrediction_with_validation) on the train/valid/test set of ppi data that I had generated before and did not get the same result as before! I get AUC around 0.5 after 150,000 num_steps with Skipgram and CBOW. I do not think that i have made a stupid mistake but I am going to read the word2vec and find what caused this difference.

@vidarmehr  Should we do a code review at 2:30PM -- hopefully there is just some simple mistake that we can fix

@vidarmehr Should we do a code review at 2:30PM -- hopefully there is just some simple mistake that we can fix

Yes, Peter. Thank you very much for your help.

Hopefully it was the fact that RaggedTensor was not getting recognized as a list of lists
https://github.com/monarch-initiative/N2V/compare/develop...detectRagged?expand=1

Still the AUC is around 0.5. I used num_steps=5000,  50,000 and 150,000.

@vidarmehr could you possibly post the exact command you ran, and also links to the edge files we should use to try and reproduce this error? (Sorry, you've probably already posted this somewhere...)

@justaddcoffee
Hi Justin, Thanks a lot for your help.
I just share the google drive folder containing the edge files that you need to run link prediction.
https://drive.google.com/drive/u/0/folders/1g61minrM8DYqnqQifbErsbbrMUPekRMm
Here is the command to run the code:
python  runLinkPrediction_with_validation_ppi.py
It runs the link prediction code with validation on train/validation/test sets with some default parameters like:
p=1, q=1, skipgram model for word2vec, logistic regression as the classifier, num_steps=150000
Please remember to set the paths to the input edge files by adding:
--pos_train path-to-the-pos-train-edges   --pos_valid  path-to-the-pos-validation-edges  --pos_test pth_to_the_pos_test_edges
similarly for negative edges:
--neg_train path-to-the-neg-train-edges   --neg_valid  path-to-the-neg-validation-edges  --neg_test path_to_the_neg_test_edges
examples:
python runLinkPrediction_with_validation_ppi.py  --p 2 --q 3 --num_steps 10000  --classifier RF
python runLinkPrediction_with_validation_ppi.py  --p 2  --num_steps 10000  --embedding_size 100 --w2v_model CBOW
Please let me know if you have any questions. Thanks.

hi @vidarmehr @realmarcin - I just merged the fixes the group suggested (#141)
Could you guys let us know if this fixes/improved the ROC score for you?

@justaddcoffee Thank you very much, Justin! I will run the code on the cluster.

Hi everyone,
Good news! I ran link prediction (with validation) on the ppi data set with default parameters p=1,q=1, Skipgram model for word2vec, Logistic regression classifier and the results are:
n_epochs=1:
AUC score (training): 0.7826864458285363
AUC score (validation): 0.7804941668369281
AUC score (test): 0.7797414042943122
n_epochs=2:
AUC score (training): 0.8196919169001615
AUC score (validation): 0.8203164357063712
AUC score (test): 0.8178015448505055
I am going to test it with different parameters and also with CBOW and  random forest classifier.

Excellent Vida!
The hackathon quashed that bug pretty quickly - Linus's Law in action!

@pnrobinson @justaddcoffee @callahantiff @LucaCappelletti94 @realmarcin @deepakunni3 Thank you everyone for your help!


[KF-5958] Upgrade to Python 3.10+

Same as in https://github.com/canonical/charmed-kubeflow-uats/pull/137
Upgrading dependencies to Python 3.10, as reported in KF-5958 and https://github.com/canonical/charmed-kubeflow-uats/issues/79
To be decided whether this or the other PR should be merged, depending whether we want to use poetry or not.

Can we also link successfull runs from ci ?

Closing as we decided to move forward with Poetry based dependency management