Fabio and Vault Tokens

I'm investigating the new integration options for fabio with cert stores, esp. vault (btw. great work, really looking forward to using it!) and I'm struggling with Vault Tokens.
I've created an orphan token with a limited policy but when I try to use it with Fabio I'm getting the following:

2016/07/26 12:35:57 [FATAL] vault: client: Error making API request.
URL: POST http://vault-server:8200/v1/auth/token/create
Code: 400. Errors:

root or sudo privileges required to create orphan token

This would suggest Fabio needs a token with a root policy. I'm not sure I'm happy about having a token with root-level access floating about my config files and live environments.
Is there a reason fabio needs a root token? Am I doing something wrong?

I'll have a look.

I don't think this is a hard requirement. I've created the fabio token as an orphaned token which apparently requires these privileges. I'll check what the best approach is and make this configurable if necessary. If you have suggestions on how to set this up feel free to comment.

Maybe it would help to understand why you are trying to create an orphan token?
My assumption was that I would be responsible for supplying a token and keeping it renewed and all Fabio would do was use the token it was given. If you prefer to generate short-lived tokens based on the supplied token on startup and then cycle them internally for your own reasons then I think that will still work.
Orphaning the token would mean it wasn't revoked when the parent token was revoked which is probably not the behaviour you want - and, of course, orphaning can only be done by root.
In fact, the token I'm supplying is itself orphaned as I didn't want it related to my login token but I think that's something for the Vault admin to deal with rather than Fabio :)

The rationale is that you provide a short-lived (e.g. 30s) token to fabio during startup so that fabio could generate its own token which it continues to use and extend as long as it is running. Otherwise, anybody who can inspect the environment via the /proc filesystem or the command line arguments would be able to re-use the token.
However, it is perfectly possible that I misunderstood something and that this is achievable without creating an orphan token. I remember having a discussion on the Vault approach fabio takes with @jeffrai - the Vault maintainer. I'll dig into the docs again.

To combat the problem of the token being visible you could make use of the new 'wrapping' capability in Vault. With this facility, the token supplied to Fabio would be a one-use token and fabio would query Vault using the unwrap endpoint to fetch the actual token to use. Other than this extra step the token will behave as normal.
The problem with this approach is that a new wrapped token would be needed every time Fabio is started meaning some form of automated script would prob. be needed to do it and that script would then need its own token in order to generate the wrapped token. I guess the advantage here is that this script would be short-lived and while you'd need to protect the token on disk you'd not see it in memory. After all, we are used to storing private cert keys on disk and if the token was stolen then it would only give access to the same private certs that could have been stolen from disk with the same level of access as the token - and the token can at least be revoked easily!
The other problem is one of renewing the token because this is another task prob. best automated and so, again, another script needing a token.
I'd be interested in what @jeffrai suggests :)

I think I prefer the approach where fabio gets the ability to create and manage its own token since you then don't have to worry about renewals and visibility. In your approach you'd still need a short-lived token to bootstrap fabio only for fabio to re-use the same token and move token renewal outside of fabio.
One question is whether that requires the creation of an orphaned token. Another is what happens when the parent token expires? This should invalidate the child tokens as well. And I also need to check whether I deal with the expired token the right way, i.e. invalidate the cert store instead of treating it as "connection error" which means that the certs remain valid.
In our environment we don't restart fabio for days, weeks or even months. It just runs and we forget about it.
As far as I recall the conversation with @jeffrai during HashiConf EU he suggested the approach of the short-lived startup token. But at that time Vault 0.6.0 was just out and the guy was quite popular. Not sure if we had enough time to talk about it in-depth.

Hmm, this won't work. If you pass a short-lived token to fabio and create a renewable child token with it then the child will expire when the parent expires unless I'm missing something.
I've used this ACL:
path "secret/fabio/*" {
  capabilities = ["read"]
}

// do we need this?
path "auth/token/lookup-self" {
  capabilities = ["read"]
}

path "auth/token/create" {
  capabilities = ["create", "update"]
}

path "auth/token/revoke" {
  capabilities = ["delete"]
}

I don't think fabio needs to create child tokens and I don't think it should create orphan tokens. Fabio's token is basically a password or an oauth type key and so it would make sense for an external system to create a token with specific permissions (i.e. policy) and hand it to fabio to use.
Firstly, it makes sense for the token to expire if not used. But it looks like there is support for self-renewal of a token (assuming the token, when created, was set to allow renewal and the policy is configured to allow it as well).  See the '/auth/token/renew-self' POST endpoint in the docs (link at the end of the comment). I'm not sure how you can find out the expiration of the token given to fabio but when you renew-self you get back info on the lease duration which you could log in consul so as to renew again in good time (and rinse/repeat).
I don't think it's a good idea to create orphan child tokens because whoever / whatever gave the token to fabio should retain the ability to revoke access. If orphan child tokens are created the revocation process is significantly harder or possibly even impossible.
To cover your concerns about the token being provided to fabio in a way that is sniffable, I suggest support for wrapped tokens. The token provided to fabio on startup would be a one-shot wrapped token which fabio would use to fetch the actual token. Pretty much anything in Vault starting v0.6 can be wrapped and this functionality makes use of the cubbyhole secret backend.
A final and somewhat different approach would be to skip the tokens entirely and use the AppID auth backend to authenticate. Here, you could accept the appId via an env var and calculate the userId through some machine-specific value (mac address is suggested but not sure that's sensible in a container). You can also lock the appId/userId combinations down to a certain CIDR. Authentication is linked to a policy and so you still end up with a token with ACLs which you can, I assume, renew as needed as long as fabio is running. Arguably, however, this authentication could be the job of a pre-launch helper script which does the login and fetches a wrapped token which is then supplied to fabio as in the previous paragraph.
https://www.vaultproject.io/docs/auth/token.html
https://www.vaultproject.io/docs/secrets/cubbyhole/index.html
https://www.vaultproject.io/docs/auth/app-id.html

Thinking about it, maybe @jeffrai was referring to these 'wrapped' tokens when he talked about the short-lived startup tokens.

Possibly. OK, then fabio would get a token, unwrap it and renew it on a regular basis. I think that makes sense.

I think I have something working but this needs a bit more refactoring since the current code tries to unwrap the token twice which won't work.

@far-blue I've pushed a patch that changes the way fabio integrates with vault. It will use the provided token instead of creating a new one and will transparently unwrap a wrapped token. fabio still tries to refresh the token on every refresh cycle but it will no longer fail if it can't. However, it logs a warning on every failure which has the potential of filling up the logs quickly if provided with a non-renewable token. I've also updated the integration test to use a policy to limit the required capabilities.
I think this is starting to look better than the original approach but I need to test the edge cases a bit more. It would be great if you could test this setup.
The required policies are (assuming that you configured your cert store with secret/fabio/cert):
  path "secret/fabio/cert" {
    capabilities = ["list"]
  }

  path "secret/fabio/cert/*" {
    capabilities = ["read"]
  }

That's great. I'm on holiday for a couple of days but I'll try and test the patch out and let you know :)
I assume if Fabio tries to refresh the token it will need permissions to do this so does the policy also need to include auth/token/renew-self and possibly auth/token/lookup-self?

I would have assumed that as well but the test registers this policy and refreshes the token on every cert refresh cycle. Since that didn't cause an error maybe that is included in the default policy. Enjoy your holiday :)

@far-blue any chance looking at this?

Hello :) Yes, been catching up on work after holiday but have had a chance this morning to test things. The log doesn't mention much about vault and tokens but I can confirm that when I provide a non-root token with the policy you mentioned above then fabio does indeed start for me without error and can access the specified vault path. Changing the path in the config causes an error and fabio will quit - so the policy is working and the token is working.
The one thing I'm still not sure about is when and whether the token will renew. I'm not sure what the "Refresh":<PHONE_NUMBER>, means in the vault section of the config output on startup - is it 30 seconds, 5 mins, 34 days etc. I've tried querying the token via vault but either Fabio is renewing the token expiry every few seconds or querying the token with vault changes its last-renewal time because every time I query I get effectively the current date/time.
I'll keep experimenting and I'm yet to see any log output suggesting fabio tried and failed to renew the token - but I've also yet to see any log output suggesting fabio tried and succeeded either. Some additional log output might be quite useful to confirm to users on startup that accessing Vault was successful and when tokens have renewed.
I've yet to test and confirm wrapped tokens so I'll try that next.

Ok, I can confirm unwrapping and I can confirm correct termination of fabio if the unwrapping token itself has expired or already been accessed.

That's good news. I'll add some more logging to the code. The only concern is to flood the logs but I'll check what can be done about that. The Refresh values are a time.Duration which means they are in ns. Default is 3s. Might be a good idea to or render them as string.

I think really 3s is a little too often for my needs. The default expiry for a vault token is 1 hour although you can set it when you create the token. It might be good to fetch the token TTL when fabio first starts (as I assume it will refresh the token as soon as it starts anyhow) and then set the refresh period to something related to the TTL - such as 75%. That way the creator of the token can control the renewal rate.

I've left fabio running far longer than the TTL on the token I gave it and it has not complained so I assume it is correctly refreshing the token. However, as I've not got much fabio might be fetching from vault at the moment I don't know whether it might fail to fetch if it tried. Overall, looks good to me :)

Regarding the refresh, are you supplying a renewal increment? Because it looks like although you are refreshing every few seconds the lease is actually being renewed for an hour (the default lease length in Vault). Again, this could be based on the creation TTL from the token info instead.

The refresh interval and token extension interval solve different problems:
The refresh interval is the time it takes fabio to detect a certificate for a domain it did not have before. It also determines how quickly fabio will act upon a deleted certificate or a revoked token.
The token extension interval protects (or is supposed to protect) fabio from intermittent failures of vault. fabio could fetch new certs lazily but at this moment I'm concerned about thundering herd issues during startup. Also, I don't want fabio to shut down because it cannot reach vault.
I think the latter part could use some more thought since both the cert and the token have a TTL which fabio could honor. But even then you'd want fabio to honor a revoke operation quickly.
Regarding refresh: as long as vault does not offer a way to notify a watcher for changes (like consul) then I don't see an alternative to constant polling. Configuring certificates in an external store becomes interesting if you have too many or if they change too often to configure them manually. But then you want fabio to react as soon as the cert is there/gone just like with the services.

I can see your concerns regarding the time it takes to pick up a new certificate for a domain and your concerns about stampeding. My gut reaction to such a problem would be to do an initial fetch of all certs during startup and then an on-demand fetch for certs for domains where no cert currently exists in order to pick up new ones. You could combine this with a longer periodic refetch to handle updates and revocations and I believe there is already support for signals for force-reloading the config in case of emergency.
You are correct in saying Vault doesn't have events and watch capabilities but I think rapid polling is not really appropriate and maybe something 'out-of-band' using a better trigger such as a consul gossip event, kv change or os signal would be worth considering. I think I'll also suggest to the Vault developers the option to configure consul events on actions on definable paths. Technically they already have the framework because they have audit backends.
Whatever option(s) you choose for checking and refreshing certs, I don't think the vault auth token needs to be renewed every few seconds. I think it needs decoupling from any fetching of data from vault and run on a longer period - possibly, as I suggest, based on a percentage of the TTL of the token when it was created. You don't need to renew the token on every use.

fyi: https://github.com/hashicorp/vault/issues/1712

yes, I agree. That would be a saner implementation. The question is whether the added complexity will have any benefits. The current implementation is straightforward, simple and robust. Since it always fetches everything there is no doubt on what the current state is. Since it always refreshes the token I don't have to have another timer for this.
Also, I don't consider polling every 3s rapid. Even with 10 concurrent fabio instances you're only generating 30 req/sec which should be cheap and come from memory (but may not now that I think about it since you may be using an external storage like S3). We're easily handling 1000 req/sec on a single service for a cached response with < 1ms latency and not a lot of CPU load.
Preferably, I'd like to have the lazy fetching implemented at a higher level so that this doesn't become something vault specific. I'll think about this and check on how to refactor this maybe for a 1.3 release. Lets first make sure that the behavior is correct, is generally usable and doesn't kill your infrastructure. Then I can make it more efficient in the next iteration.

Are those 1000 req/sec fetching data from vault backed by consul? Just curious as I've really not tested the performance of my setup :)
I guess the only problem that might be caused by the frequent renewing of the token is if you have auditing turned on as each renewal will be logged. Maybe for the moment it could just be a configuration option so people can tune it like they can the consul healthcheck.

No, this is a service that is delivering its own cached data. At any rate 30 req/sec don't seem a lot but that doesn't change the validity of your argument. I'll see what I can do to reduce the impact. For now, just increase the refresh time or set it to 0 to disable it.

I've not seen a config option to change the refresh time for vault - does it support the same 'refresh' option as the http cert store?

Well, that's emberrassing :). yes it does. I'll update the docs.

That makes sense and will certainly do for me for now :)

Regardless of the discussion on refresh rates etc. do you feel the improvements for handling vault tokens will be released soon? The fixes will at least allow me to use Vault without needing to supply a root token :)

yes. It's next on the list. Time is a scarce right now. I'll merge it soon.

Merged to master. Lets deal with the fallout in subsequent tickets. @far-blue really appreciate your help and dedication on this one!

Updated the wiki with the new behavior


⚠ Bofrost.es has degraded performance

In f0e6219, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1622 ms

Resolved: Bofrost.es performance has improved in 7553238 after 6 minutes.


Suggestion: scrollable/shorter recent brews dropdown menu

Suggestion: recent brews dropdown with it's own scroll function
Why? because i am experiencing a chrome bug, that when i put my mouse over the right part of the recent brews button, the dropdown opens, and it is larger than the page, so it creates another scroll bar at the right of the page, therefore moving everything, including the bottom, which suddently i'm not over.
All that activates and deactivates the hover effect, and weirds the page until you move your mouse.
video
it is a minor inconvenience, but easily sorted by either decreasing the number of recent brews, or making the dropdown menu scrollable by itself.

Easy CSS fix
homebrew nav .recent.navItem .dropdown {
  overflow: scroll;
  height: 100vh;
}

Add those two CSS declarations to this file:
https://github.com/naturalcrit/homebrewery/blob/dd93c4cdd4e4183d340188ae570bed3ac78e4542/client/homebrew/navbar/navbar.less#L70-L77


Allow specifying a reason for annotation deprecation

This assists https://github.com/istio/api/pull/1997
Example:
Info [IST0135] (Pod shell-74bc57ff88-ptb6d.default) Annotation "sidecar.istio.io/discoveryAddress" has been deprecated and may not work in future Istio versions.
Info [IST0135] (Pod shell-74bc57ff88-ptb6d.default) Annotation "sidecar.istio.io/inject" has been deprecated in favor of the "sidecar.istio.io/inject" label and may not work in future Istio versions.

This has been replaced with a label of the same name. The label is
strictly more powerful since selection is done in Kubernetes, rather
than in Istio.
See
https://preliminary.istio.io/latest/docs/setup/additional-setup/sidecar-injection/#controlling-the-injection-policy

@howardjohn: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-pilot-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-pilot-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-pilot-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-pilot-k8s-tests_istio

release-test_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test release-test_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-pilot-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-pilot-k8s-tests_istio

release-test_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test release-test_istio

integ-distroless-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-distroless-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-pilot-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-pilot-k8s-tests_istio

release-test_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test release-test_istio

integ-distroless-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-distroless-k8s-tests_istio

integ-security-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-security-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-pilot-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-pilot-k8s-tests_istio

release-test_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test release-test_istio

integ-distroless-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-distroless-k8s-tests_istio

integ-security-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-security-k8s-tests_istio

integ-pilot-multicluster-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-pilot-multicluster-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@howardjohn: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-pilot-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-pilot-k8s-tests_istio

integ-security-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-security-k8s-tests_istio

integ-pilot-multicluster-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-pilot-multicluster-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@howardjohn: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-telemetry-mc-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-telemetry-mc-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-telemetry-mc-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-telemetry-mc-k8s-tests_istio

integ-pilot-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-pilot-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-telemetry-mc-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-telemetry-mc-k8s-tests_istio

integ-pilot-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-pilot-k8s-tests_istio

integ-security-multicluster-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-security-multicluster-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@howardjohn: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-pilot-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-pilot-k8s-tests_istio

integ-security-multicluster-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-security-multicluster-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@howardjohn: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-security-multicluster-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-security-multicluster-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@howardjohn: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-security-multicluster-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-security-multicluster-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@howardjohn: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-security-multicluster-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-security-multicluster-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-security-multicluster-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-security-multicluster-tests_istio

integ-pilot-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-pilot-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@howardjohn: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-security-multicluster-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-security-multicluster-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@howardjohn: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-pilot-k8s-tests_istio
d42ca5e5e720602255c67f26402e665a1809424d
link
/test integ-pilot-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest


TextArea selectors

Implemented a textarea value selector
Added a textarea example that follows the style given in vapor (see demo)

Codecov Report

Merging #1094 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1094      +/-   ##
==========================================
+ Coverage   99.75%   99.76%   +0.01%
==========================================
  Files         330      331       +1
  Lines        8180     8185       +5
  Branches     1954     1955       +1
==========================================
+ Hits         8160     8166       +6
+ Misses         20       19       -1

Impacted Files
Coverage Δ

src/components/textarea/TestAreaSelectors.ts
100% <100%> (ø)

src/components/drop/DropPod.tsx
98.36% <0%> (+1.63%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a93f4ce...11289a9. Read the comment docs.


[Feature Request/Question] Save Interpolated Value, Select .. where

Let's say I have a yaml like this:

user: ${env:USER}

And, I want to save the config to a file with the interpolated values.
Loading a file:

user = OmegaConf.load('user.yaml')

Writing a file (without referencing any attributes)

OmegaConf.save(user, 'cfg.yaml')

Should write:

user:  jsmith

Is this possible?

Is it possible to select a node/list of nodes with a where clause?

users:

type: admin
..
type: guest

I was to select from users where type == 'admin'
Thanks,
Vish

Check the signature of save:
    @staticmethod
    def save(
        config: Container, f: Union[str, pathlib.Path, IO[Any]], resolve: bool = False
    ) -> None:
        """
        Save as configuration object to a file
        :param config: omegaconf.Config object (DictConfig or ListConfig).
        :param f: filename or file object
        :param resolve: True to save a resolved config (defaults to False)
        """

You want to use resolve=True.

Thank you. And, can I select .. where?

Ah, missed that one.
This is not supported. you can select the parent node and do your own filtering.

Thank you @omry


Instructions for protobuf generation inadequate

Current behaviour/situation
We have some instructions for how to generate network protocol in the comments at the top of
software/common/generated_libs/network_protocol/network_protocol.proto.
However they are geared towards macOS, which is not an assumption we want to make.
Expected state/behaviour
Instructions should be in a README.md and should be for linux as well.
How do you know it is wrong
I don't have a mac and I don't want one :man_shrugging:

Better yet, the protocols should be generated in CI with every push, to ensure we are still able to generate them with whatever version of prtobuffers and nanpb are available

Better better yet, the generated code should not even be committed to repo, but generated locally only, prior to building controller or GUI. Probably, an additional command for the common/common.sh script.


CU-270gpt7 Cleanup Android Navigation, avoid adding multiple instance…

…s of the same screen to the back stack

Task linked: CU-270gpt7 Android: Fix Navigation


Hang/Race condition when deleting engine if child of engine is selected

Steps:

Add engine
Select taglist or mapping
Delete engine
Hang :(

You can work around the hang by selecting "exit" from the taskbar but you can't close the error dialog generated for whatever reason, probably its modal but then gets aborted when the taglist ui gets closed/aborted

can we check if current node is a child of node to-be-deleted and close if it so? Are we already doing that?

NO longer hangs now Throws and Unhanded error 1556


Preliminary I2S Support

starting with ESP32-C3 and ESP32-S3
then ESP32 and ESP32-S2
for the start just the TX channel
for the start just standard I2S

Kudos! Is there an active branch I can look at for this work?

Kudos! Is there an active branch I can look at for this work?

I pushed the current WIP here: https://github.com/bjoernQ/esp-hal/tree/feature/i2s ... it's still a lot to do

Thanks!
This work is now tangentially related: https://github.com/embassy-rs/embassy/pull/1049
Since I just noticed https://github.com/esp-rs/esp-hal/pull/225 :)


64-bit Autosplitter

The only "broken" splits are the endings. Any% splits when you hover over a choicer and NG+ splits when Niko leaves the game window. I'm not entirely sure how to fix that quite yet but they're disabled by default so I think it's fine :bailey:

epic


test: Additional info to debug fromCIDR+toPorts flake

I'm hoping this additional information dumped on failures will help debug flake #12596.

retest-net-next

cilium/ci is covered and the only build that covers this code is passing. Marking as ready to merge.


Add validation of forms

Add validation of data in the forms.
Related to #32

Ještě Frekvence a je to hotovo.


[Relay][VM] JIT

Right now relay vm dont support any/symbolic shape with dense, because dense schedule/compute require input shape known as constant beforehand.
This PR add polymorphic inline JIT, so if a function cannot be lowered at compile time, for each new shape we will encounter, we will compile a tvm kernel at runtime, and invoke it.
@wweic @jroesch @junrushao1994 @icemelon9 @vinx13 please review.

I'm under the impression that for operators that accepts/returns dynamic shape tensor, we just need to implement the shape function. Please correct me if I'm wrong. @icemelon9

@wweic yes, however some schedule does not accept dynamic shape tensor, such as nn.dense

How do we pick a descent dense schedule at runtime? My feeling is that JIT compilation done in this PR can be covered by AOT compilation, like in https://github.com/dmlc/tvm/issues/4118.

@MarisaKirisame Got it. I think it might be better to fix the schedule for nn.dense, since the design for any support is by adding shaping function to each operator. We should try to be consistent in the implementation. What do you think?

@wweic I had talked with @eqy, and he say generic schedule is slow. This also allow us to tune schedule for each specific input shape (so dense with batch size of 16 vs 32 need not share the same schedule at all).
The shape function is still needed no matter what, I dont get what you mean by that.

@MarisaKirisame I don't think tune a dynamic shape kernel is related to JIT.

@kevinthesun they both have their own merits.
For JIT, the problem is the compiler need to live in the runtime - so it can never be used on embedded device. Also if new shapes are constantly encounter (which I suspect wont happend), compilation will be a major cost of runtime.
For Dispatch, it is the complexity, and, if the dynamism fall into a few predefined case, JIT will get the utmost precision.
Both approach can definitely coexist and we can have some LoweringStrategy option which pick which to use.

@kevinthesun we cant tune symbolic shape, but in JIT, there is no symbolic shape/any shape/dynamic shape. All that's left is purely concrete shape, and we can tune them individually.

@kevinthesun we cant tune symbolic shape, but in JIT, there is no symbolic shape/any shape/dynamic shape. All that's left is purely concrete shape, and we can tune them individually.

But how do we know which shapes will come in? In AOT, we can split a symbolic axis into several buckets and AOT tune/compile kernel for each bucket. I don't see how we can achieve this in JIT.

@kevinthesun there will be two run of the same program.
In the zeroth run, u do not tune and use the default schedule. whenever you does this, tvm will complain that it has to use the default schedule (by outputting to the stdout). You record all of them.
In the first run, you tune the schedule beforehand according to the stdout output.

@MarisaKirisame The difficulty for dynamic shape kernel is that we cannot predict which concrete shape will come in at runtime. In the zero run, only a sample of concrete shape comes in, and in the first run only this sample is tuned. There can be a lot of different shapes coming in and a lot of tuning have to be done in runtime. We cannot tune every concrete shape because both of time and memory complexity. Also if we want to tune kernels, we should do this even before compilation.

@kevinthesun it depend on the work load. For my current work load, only the batch_size is dynamic, and it is either 32/16, so the problem is easily solved.
One can also imagine converting from torchscript to relay (or something similar), and all of the 'dynamic' shape only have 1 single value at run time.
I agree that this is no a universal solution, however it will always work (instead of the current vm implementation which will simply break on dynamic dense).
And it seems like there is a lot of work in topi/tvm/relay to make graph dispatching work, so I think the best plan is to have this for now (which I need right now).
After the graph dispatching is done, we can figure out(by benchmarking) if hybird make sense, or is it just universally better, then we implement a hybird strategy or swap it out, does it sounds good to you?

I have some concern at this PR as it is a temporary solution and is limited to your current workload. It's definitely possible that the workload is fully dynamic, such as batch size, sequence length, output of nms and arange op, etc.
Besides, after this PR, the VM runtime then requires llvm and tvm to be able to jit these kernels. This will increase the runtime size which is not a desirable behavior in many situations.
I think a better solution to your current case is to use cblas in the target. This can support dense with symbolic shape.

@kevinthesun it depend on the work load. For my current work load, only the batch_size is dynamic, and it is either 32/16, so the problem is easily solved.
One can also imagine converting from torchscript to relay (or something similar), and all of the 'dynamic' shape only have 1 single value at run time.
I agree that this is no a universal solution, however it will always work (instead of the current vm implementation which will simply break on dynamic dense).
And it seems like there is a lot of work in topi/tvm/relay to make graph dispatching work, so I think the best plan is to have this for now (which I need right now).
After the graph dispatching is done, we can figure out(by benchmarking) if hybird make sense, or is it just universally better, then we implement a hybird strategy or swap it out, does it sounds good to you?

This method only works for very limited cases, but introduces extra complexity and dependency to VM. I'm not quite sure if this is the way we want to go.

@icemelon9 @kevinthesun the 'your current workload' include all classical vision model (resnet/densenet/vgg), and all classical nlp model (treelstm/lstm, as they already use ADT). That leave only bert/transformer uncovered, which AFAIK did not exist in tvm yet.
batch size is not fully dynamic - for 99% of the time it will be the batch size specified by the user. the remaining 1% is the len(dataset) mod batch_size (which is also static).
I am not sure if 'everything excluding transformers' is very limited. I am not an expert in ML, so I might be wrong. But in this case, please tell me what other popular model beside bert/elmo/transformer am I missing.

@kevinthesun batch size is universal in all vision task.
in virtually every vision task, the batch size is almost fixed (because there is validation/test/leftover).
supporting all vision task is by no mean 'very limited'.

@MarisaKirisame If your

in virtually every vision task, the batch size is almost fixed

It looks to me that your use case is to support some fixed batch sizes. This definitely doesn't cover most actual use cases for dynamic shape in practice.

current fail doesn't mean we cannot support this in the future. We can have better approach to support symbolic shape. I don't think we should merge a half-bake solution which will be replaced or deprecated soon.
and, we should move the discussion to https://discuss.tvm.ai

Agree we should move this discussion to forum or an RFC issue since there are some fundamental issues to be resolved.

close due to inactive status for now


Required changes for merging

Moved into the 'fair' topic with the other RO crate tutorials
Optimized PNGs (we have a bot that would PR it on monday but, why not do it now)
Added the copyright statement next to the license in the tutorial metadata box header

And the footer

Changed figure/figcaption HTML elements to their corresponding GTN-ish syntax. (Using the markdown equivalents enables us to pre-calculate the image dimensions and reduce page layout shifting.)

Thanks! Will check that the SVG fonts render as before.


Typo in Cargo.toml

https://github.com/cristicbz/rust-doom/blob/master/Cargo.toml#L16
Should it be path = "src/wad"?

Thanks for the issue, it's fixed by consolidating all the different projects in one (see #83 for reasoning).


Can i translate it into Chinese

This project is amazing and has been very helpful to me. I'm wondering if I can open another repository and translate it into Chinese. I will definitely credit the source. Or do you have any recommendations on how I can contribute Chinese translation right into to this repository ?

Happy it helped! The project has a pretty open license (MIT) so there would be no issues with virtually any fork, including a translation, so long as there is attribution to the original work.


Added the parent nodes view to all searched glossary term in search bar

Checklist

[ ] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[ ] Links to related issues (if applicable)
[ ] Tests for the changes have been added/updated (if applicable)
[ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
[ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

bringing these changes into another PR of small glossary fixes


fix issue #2677 the signal of client disconnect to server ignored

When constructing a new ghttp.Request, the context should be set as the underlying context of wrapped http.Request.

The failed checks have nothing to do with the commits. How can I re-run the test?

Why should add new context field  to ghttp.Request?
Request has two methods in ghttp/ghttp_request_param_ctx.go, Context() and GetCtx()
If you init context in newRequest directly, the logic in Context() is useless.
So,  we should understand why forbids the context manually done in Context() method before submit PR.

Why should add new context field to ghttp.Request? Request has two methods in ghttp/ghttp_request_param_ctx.go, Context() and GetCtx()
If you init context in newRequest directly, the logic in Context() is useless.
So, we should understand why forbids the context manually done in Context() method before submit PR.

I think neverDoneCtx context did not prevent the underlying context from triggering Done().
The reason of context not triggering Done() is because the real context was not passed as underlying context of the neverDoneCtx.

Codecov Report
Patch coverage: 100.00% and project coverage change: +0.02 :tada:

Comparison is base (15eaac3) 78.88% compared to head (407df64) 78.91%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2710      +/-   ##
==========================================
+ Coverage   78.88%   78.91%   +0.02%
==========================================
  Files         631      629       -2
  Lines       51661    51542     -119
==========================================
- Hits        40753    40673      -80
+ Misses       8894     8861      -33
+ Partials     2014     2008       -6

Flag
Coverage Δ

go-1.16-386
?

go-1.16-amd64
?

go-1.18-386
?

go-1.18-amd64
?

go-1.19-386
?

go-1.19-amd64
78.91% <100.00%> (+0.01%)
:arrow_up:

go-1.20-amd64
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

database/gredis/gredis.go
38.09% <100.00%> (ø)

net/ghttp/ghttp_request.go
93.28% <100.00%> (+0.05%)
:arrow_up:

... and 9 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Readd missing gems

PR #826 updated Gemfile and Gemfile.lock but forgot to include the new gems for boxen-2.9.0 and faraday-0.9.2.
Fixes #827

👍

👍

Thanks! 🍭


Creating this project in react

Hii @07sumit1002 ,
I would like to alter this project using react
So it would be more dynamically used and it would be easily handled by other peoples
Would u please assign this issue to me under VSOC'24

@Khanishsuresh Go ahead and work on it.
for the time being I am giving it a medium tag but depending upon your scope of work, i will give bonus as well

@Khanishsuresh Also write about this issue in discord channel so that it will be easier for me to mention it in Spreadsheet.

https://discord.com/channels/1234877876382208082/1252619076522475530

@07sumit1002 Done

@07sumit1002
Hii bro, I am working on this issue. I request you not to accept any PR for 1 day
So i will create the app in react, next the users can add there contribution in react
I am waiting for u reply. Once u say i will start doing it
But i need hardly one day.Since there are many files in ur project

Sure
On Mon, Jul 8, 2024 at 8:47 PM Khanishsuresh @.***>
wrote:

@07sumit1002 https://github.com/07sumit1002
Hii bro, I am working on this issue. I request you not to accept any PR
for 1 day
So i will create the app in react, next the users can add there
contribution in react
I am waiting for u reply. Once u say i will start doing it
But i need hardly one day.Since there are many files in ur project
—
Reply to this email directly, view it on GitHub
https://github.com/07sumit1002/CabRental/issues/6#issuecomment-2214420542,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AXPSQMWCYNCIQQQ3YRU5HLLZLKURBAVCNFSM6AAAAABJRXP4GSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJUGQZDANJUGI
.
You are receiving this because you were mentioned.Message ID:
@.***>


Github actions to build & push images (worker, service, tests)

We need to specify SOURCE_BRANCH build arg, which AFAIK can't be done in Quay.io build trigger so we have to build it in Github.

Would it be possible to get rid of some of the duplication by using context variables, for example? Also, is there some kind of "matrix" functionality in GitHub workflows? Maybe that could be used to get away with a single workflow file.

Would it be possible to get rid of some of the duplication by using context variables, for example?

Yes, it is possible to merge them but at a price of much more code.
I decided to go the simpler, but duplicated code way.
We can always merge later (once I finally read something more about actions and know the best practices).

Also, is there some kind of "matrix" functionality

no idea what's that

We can always merge later (once I finally read something more about actions and know the best practices).

I would like to see this work done in this PR, in order to avoid merging something which adds more to the pile of technical dept.

At least let's have a look if using a build matrix could help to remove some of the duplication.


Add ability to stop/pause the pool and wait for all processes to complete

Hi,
As discussed with @mnmkng in the support chat, we would like to be able to stop or pause the pool and to wait for all running processes to complete before going further.
Basically we tried with await pool.abort() but that will resolve immediatly after the pool is destroyed and wont take care of running processes to complete. Ideally we would need for something like await pool.stopGracefully(timeout) that wait either for the processes to terminate or the timeout to be reached.

Implemented in https://github.com/apifytech/apify-js/pull/268


[Feature] get interpolated values at x-Position of axis pointer

What problem does this feature solve?
If user hovers over the series of specific data points. I want the tooltip to display also interpolated values in between of real data points.
What does the proposed API look like?

First allow tooltip to be triggered at any mouse position the user hovers.
Allow tooltip formatter to contain parameters of interpolated value
Allow tooltip formatter to contain parameters to indicates that its an interpolated value
Make this feature be activated via flag. Default inactive.

currently faking this through adding my own interpolated datapoints before passing them to echarts. i wonder if doing it on the tooltip layer via xaxis coordinate transformation would be more performant.


Can't scroll list while dragging item

I have "overflow:auto" property on ul-list with long set of items. I tried to scroll list after I pressing mouse button to drag item but I can't! In this example http://gaearon.github.io/react-dnd/examples-sortable-simple.html just set "overflow: auto" property and fixed height (for example 200px) on list container's div  and you can reproduce the bug.

This is not a bug, this is how HTML5 drag and drop is implemented in (some) browsers.
If you're using react-dnd-html5-backend, that's what powers it.
The solution is to make the container list a drop target too, and scroll the list when the container drop target receives hover in the top or bottom area. When it's hovering over top area, begin scrolling to the top, with the speed depending on how close the cursor is to the top edge. Similarly, when hovering over the bottom area, begin scrolling to the end—the lower the cursor, the faster the speed. You can take card dragging behavior in Trello as an inspiration.
If you implement this, feel free to contribute it back to the example.

I'm looking to do the same thing as even with the above suggestion, having a bit of trouble.
For instance... hover seems to only fire when you "move" where as this type of functionality scrolls even when not moving and is only dependent on how close you are to the bottom/top of the page.
Also found this.... https://gist.github.com/gaearon/150fa1bed09c92abdb46 ... but it appears to be outdated?

Yes, it's very outdated, but it shows the approach. Start animation loop on enter, stop it on leave, and inside the animation tick, consider to scroll if condition matches.

Posting for other people who had this issue. I've built this solution to auto scroll container that works smoothly in every browser I've tried. https://github.com/azuqua/react-dnd-scrollzone

@nickclaw can you give example?

I'll add a demo tonight if it helps. But it's really as easy as the example on the README, just make sure you set the overflow values to allow the container to scroll.

@nickclaw  Please provide a working example. I know it's simple but I don't why it's not working for me. If working example there then I can cross check it.

Hey @erkishorjangir, did you find a way to use scrolling with react-dnd?

For future reference, react-dnd-scrollzone works also on Safari and Firefox. You can see this closed ticket: https://github.com/azuqua/react-dnd-scrollzone/issues/11

Does anyone know why Jquery UI isn't bitten by this issue? I'm considering hacking it into my React app because being able to scroll around while dragging is that important for my use.

I've written a module to work around this; auto-scroll divs at the top and bottom that are hidden unless you drag something. Unfortunately, I'm unable to get the npm version to import properly; the standalone index.js file in the repo works though. This is different from react-dnd-scrollzone, in that it uses the full window with its normal scroll bar, instead of making a sub-window with its own scrollbars.
https://github.com/David-OConnor/dnd-scroll

When I drag my card from one lane to the last lane the scroll-x doesn't get scroll and it's like my viewport is at a fixed width.

I think this works for me.
hooks for long tap
export type TapTimeline = 'pending' | 'start' | 'tapping' | 'end';

export const useLongPressImage = () => {
  const [tapTimeline, setTapTimeline] = useState<TapTimeline>('pending');

  useEffect(() => {
    let timerId: NodeJS.Timeout | undefined;
    if (tapTimeline === 'start') {
      timerId = setTimeout(() => setTapTimeline('tapping'), 200);
    }
    return () => {
      clearTimeout(timerId as NodeJS.Timeout);
    };
  }, [tapTimeline]);

  return {
    tapTimeline,
    ref: tapTimeline === 'tapping',
    onTapStart: () => setTapTimeline('start'),
    onTapEnd: () => setTapTimeline('end'),
  };
};

useDrag(React-DnD API)
export const useDragAndDrop = (tapTimeline: TapTimeline) =>  {
  // ...
  const [{ isDragging }, drag, preview] = useDrag({
    item,
    type,
    canDrag: timeline === 'tapping',
    collect: (monitor) => ({
      isDragging: tapTimeline === 'tapping' ? monitor.isDragging() : false,
    }),
  });
  return {
    dragRef,
    dropRef,
    drag,
    drop,
    preview,
    isDragging,
  };
}

Drag Item Component
const Component = () => {
  const { tapTimeline, onTapStart, onTapEnd } = useLongPressImage();
  const { dragRef, dropRef, drag, drop, preview, isDragging } = useDragAndDrop(tapTimeline)

  return (
  // ...
   <div
       ref={dragRef}
       onTouchStart={onTapStart}
       onTouchEnd={onTapEnd}
     >
  // ...
  )
}

React-dnd-scrollzone is not working with 4+. Using react-dnd-scrolling is working for me https://github.com/TechStark/react-dnd-scrolling

React-dnd-scrollzone is not working with 4+. Using react-dnd-scrolling is working for me https://github.com/TechStark/react-dnd-scrolling

It works! Thanks!


Resize the sprite

Hello,
First thank you for this package, it is easy to implement it.
Will it be possible to use the scale to decrease the size of the Sprite.
var scale = 0.5;
We will need a double for that.
Do you think you could update that?
Also, would be for possible to flip the sprite?
Thank you

@MikaReesu
Thanks for using my library.
Changed scale option to double type and added flipX, flipY options.

Perfect! Thank you for the fast update. All is working fine!


Generate predictions using proposed ensemble

Hi,
In this repo, It seems like the predictions are made by a single trained model rather than using the proposed ensemble technique.
Do you have any plans for uploading the ensemble part of the work?

Hi, unfortunately, we are not supporting this repo anymore. But I'll try to find the weight and upload


fixed HL1 input

XRTK - Mixed Reality Toolkit Pull Request
Overview

The Windows Mixed Reality APIs changed and handedness for the HoloLens One was enabled which broke our controller mappings.
We added a new controller type HololenOneController for the HoloLens 1 device.
We updated the HololensOneController controller to utilize the new handedness
We updated the default controller mappings for the HololensOneController
Breaking Changes
Breaking change mitigation:

Delete the HoloLensControllerProfile asset
Reimport the default controller assets

@FejZa can you test this on HL2 to ensure we don't have a regression issue

@FejZa we had to put the workaround back in to get HoloLens 1 input working correctly again.

I'll test this on HL2 today.


[SPARK-9860][SQL] Join: Determine the join strategy (broadcast join or shuffle join) at runtime

What changes were proposed in this pull request?

add query fragment for execute stage according to exchange.
before running no leaf query fragment, optimize plan using mapOutputStatistics of its child fragments.
if size of one exchange is less than autoBroadcastJoinThreshold, it transforms sort merge join to broadcast join.
I will support optimization for skew join in another PR.

How was this patch tested?
unit tests

Test build #58239 has started for PR 13028 at commit 73de7b2.

Test build #58239 has finished for PR 13028 at commit 73de7b2.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58239/
Test FAILed.

Test build #58240 has started for PR 13028 at commit 6cc187e.

Test build #58240 has finished for PR 13028 at commit 6cc187e.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58240/
Test FAILed.

Merged build finished. Test FAILed.

Test build #58241 has started for PR 13028 at commit 6f1105c.

Test build #58241 has finished for PR 13028 at commit 6f1105c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58241/
Test FAILed.

Test build #58376 has started for PR 13028 at commit 5de46b0.

Test build #58376 has finished for PR 13028 at commit 5de46b0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58376/
Test FAILed.

Test build #58378 has started for PR 13028 at commit a18be55.

Test build #58378 has finished for PR 13028 at commit a18be55.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class QueryFragmentSuite extends QueryTest with SQLTestUtils with SharedSQLContext

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58378/
Test FAILed.

Merged build finished. Test FAILed.

cc @rxin @davies

Test build #58481 has started for PR 13028 at commit d9c66d7.

Test build #58481 has finished for PR 13028 at commit d9c66d7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58481/
Test PASSed.

@lianhuiwang could this regress performance for existing queries? Can you please share some benchmarks with us (if any)?

We are closing it due to inactivity. please do reopen if you want to push it forward. Thanks!

It is great for BI scene. Why not continue?


Fix that mouse-button-up event is parsed twice for drag-and-drop

Currently Input::get_singleton()->parse_input_event(mb); is called twice for mouse-button-up events when dropping in a different window on linuxbsd.
I would expect, that an event is parsed only once.
This PR removes the first location where this happens. The second location, which is also used for mouse-button-down events appears a few lines later:
https://github.com/godotengine/godot/blob/f3e6750a7e4702918e05f42b1376e30e652f2f90/platform/linuxbsd/x11/display_server_x11.cpp#L3958

Thanks!


itemLabel not showing up in homekit apps

Having a sitemap with multiple light switches it is likely that some of them are named the same (e.g. "ceiling light"). So using the label is name is not working as it would not be unique and I want to use itemLabel in the customAttributes.
Example:
Having a item in openHab named "EG_KUE_LichtDecke"
I switched off the label usage ("useLabelForName": false) and added custom attributes for the item, e.g.:
"customAttrs": [ {
"itemName":"EG_KUE_LichtDecke",
"itemLabel":"Ceiling Light",
"itemType":"LightbulbItem",
"itemManufacturer": "MDT",
"itemModel": "KNX Aktor",
"itemSerialNumber":"1.1.1.A"
} ]
In the homekit apps I see this item labeled "EG_KUE_LichtDecke" instead of "Ceiling light" as I´d expect. The name and everything else is considered.
So the questions are:
If I define an item in the items with the name XYZ, add it to the sitemap, switch off using labelsfornames and add custom attributes where itemName=XYZ and itemLabel=ABC

Is the link between customAttribs and the openHAB item done using itemName XYZ ? (Thinks so)
Where is this itemLabel ABC used ?
Is this the lack of the Homekit App (tried "MyTouchHome" as well as "Eve" where "Eve" is even showing less infomation on the item) or of the plugin (label not geting transfered) ?

A bit more documentation details on how to name and link items to custom attributes would be great, currently it is a bit of a lucky guessing.

Hi Thomas, for now it's impossible to do because Siri use accessory's name and these must be unique. So if you want to pronounce the accessories with Siri you should use name as "Garage door" and not as "House1_Garage_door".

Thanks for looking into this.
From my perspective this is a clear gap in apples homekit / siri combination. They mix up technical IDs with everyday language use to control accessories using Siri.
Closing this as it´s an Apple issue that can´t be fixed on your side.


Pass payload through to receive

When a receive callback is called pass back the received messages
payload to the callback.

That 1 comment then LGTM

Thanks for the review @jakecraige!


Incorrect prediction after calling forwad

Hi everyone,
I have a single input, six output system which I am training with the rnn module.
I construct the rnn network as follows:
ninputs = 1
noutputs = 6
  --Recurrent Neural Net Initializations
if opt.model == 'rnn' then
  rho         = 5                           -- the max amount of bacprop steos to take back in time
  start       = 1                          -- the size of the output (excluding the batch dimension)
  rnnInput    = nn.Linear(ninputs, start)     --the size of the output
  feedback    = nn.Linear(start, ninputs)           --module that feeds back prev/output to transfer module
  transfer    = nn.ReLU()                     -- transfer function
end
--[[Set up the network, add layers in place as we add more abstraction]]
function contruct_net()
  if opt.model  == 'mlp' then
          neunet          = nn.Sequential()
          neunet:add(nn.Linear(ninputs, nhiddens))
          neunet:add(transfer)
          neunet:add(nn.Linear(nhiddens, noutputs))

  elseif opt.model == 'rnn' then
    require 'rnn'
    --RNN
    r = nn.Recurrent(start,
                     rnnInput,  --input module from inputs to outs
                     feedback,
                     transfer,
                     rho
                     )

    neunet     = nn.Sequential()
              :add(r)
              :add(nn.Linear(nhiddens_rnn, noutputs))

    neunet    = nn.Sequencer(neunet)

I should mention that I am training in mini-batches of 6 at a time. When I call forward as follows
      for step = 1, rho do
        table.insert(inputs_, inputs[step])
        outputs[step] = neunet:forward(inputs_)
      end

I end up with an output that is a 6 X 6 matrix. I would expect that I should obtain a 6 X 1 table with each element consisting of 6 outputs. I wonder if this is normal. If it is not, can someone please explain to me what I could possibly be doing wrong?
Thanks!

I rearranged the output into a table of the form:
 outputs
{
  1 :
    {
      1 : DoubleTensor - size: 6x1
      2 : DoubleTensor - size: 6x1
      3 : DoubleTensor - size: 6x1
      4 : DoubleTensor - size: 6x1
      5 : DoubleTensor - size: 6x1
      6 : DoubleTensor - size: 6x1
    }
  2 :
    {
      1 : DoubleTensor - size: 6x1
      2 : DoubleTensor - size: 6x1
      3 : DoubleTensor - size: 6x1
      4 : DoubleTensor - size: 6x1
      5 : DoubleTensor - size: 6x1
      6 : DoubleTensor - size: 6x1
    }
  3 :
    {
      1 : DoubleTensor - size: 6x1
      2 : DoubleTensor - size: 6x1
      3 : DoubleTensor - size: 6x1
      4 : DoubleTensor - size: 6x1
      5 : DoubleTensor - size: 6x1
      6 : DoubleTensor - size: 6x1
    }
  4 :
    {
      1 : DoubleTensor - size: 6x1
      2 : DoubleTensor - size: 6x1
      3 : DoubleTensor - size: 6x1
      4 : DoubleTensor - size: 6x1
      5 : DoubleTensor - size: 6x1
      6 : DoubleTensor - size: 6x1
    }
  5 :
    {
      1 : DoubleTensor - size: 6x1
      2 : DoubleTensor - size: 6x1
      3 : DoubleTensor - size: 6x1
      4 : DoubleTensor - size: 6x1
      5 : DoubleTensor - size: 6x1
      6 : DoubleTensor - size: 6x1
    }
}

but still I get
size mismatch, m1: [6 x 1], m2: [6 x 1] at /home/local/ANT/ogunmolu/torch/pkg/torch/lib/TH/generic/THTensorMath.c:706
stack traceback:
	[C]: in function 'addmm'
	...l/ANT/ogunmolu/torch/install/share/lua/5.1/nn/Linear.lua:75: in function 'updateGradInput'
	...T/ogunmolu/torch/install/share/lua/5.1/nn/Sequential.lua:55: in function 'updateGradInput'
	...NT/ogunmolu/torch/install/share/lua/5.1/rnn/Recursor.lua:45: in function '_updateGradInput'
	...lu/torch/install/share/lua/5.1/rnn/AbstractRecurrent.lua:46: in function 'updateGradInput'
	...T/ogunmolu/torch/install/share/lua/5.1/rnn/Sequencer.lua:78: in function 'updateGradInput'
	...l/ANT/ogunmolu/torch/install/share/lua/5.1/nn/Module.lua:30: in function 'backward'
	rnn.lua:475: in function 'train'
	rnn.lua:688: in main chunk
	[C]: in function 'dofile'
	...molu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
	[C]: at 0x00406240

@lakehanne I don't think the above code is implementing a 1 input to 6 output . For that you should decorate your Recurrent module into a Repeater. The Repeater takes a single input of size batchsize x inputsize and outputs a table of tensors of size seqlen x batchsize x outputsize where seqlen would be 6 in your case. Is this what you are looking for?

@nicholas-leonard thanks for the response. I am just hearing about the repeater sequence, to be honest with you. A couple of questions,

Is there a documentation somewhere for the Repeater module?

Also, in the outputs table of tensors of size seqlen X batchSize x outputsize, if seqlen is 6, what would outputsize be?

Thanks in advance!

@lakehanne

Doc can be found here : https://github.com/Element-Research/rnn#rnn.Repeater. It isn't much I admit.
if you are using a Recurrent with an inputSize of 10 and outputSize of 20, and you decorate it with a Repeater having a rho argument of 6 (output sequence length), and your batchsize is 5 such that your input to the Repeater(Recurrent) is 5 x 10, then the output will be 6 x 5 x 20.

Hope that helps.

Thanks for the reply @nicholas-leonard . I tried your suggestion. Here's what I found:
I set up the network as folows:
ninputs     = 1
noutputs    = 6
nhiddens    = 1
  rho         = 5                           -- the max amount of bacprop steos to take back in time
  start       = 1                         -- the size of the output (excluding the batch dimension)
  rnnInput    = nn.Linear(ninputs, start)     --the size of the output
  feedback    = nn.Linear(start, ninputs)           --module that feeds back prev/output to transfer module
  transfer    = nn.ReLU()
    r = nn.Recurrent(start,
                     rnnInput,  --input module from inputs to outs
                     feedback,
                     transfer,
                     rho
                     )

    neunet     = nn.Sequential()
              :add(r)
              :add(nn.Linear(nhiddens_rnn, noutputs))

    neunet    = nn.Repeater(neunet, noutputs)

When I start training in 6 minibatches, as in
      for step = 1, rho do
        outputs[step] = neunet:forward(inputs[step])
        _, outputs[step] = catOut(outputs, step, noutputs, opt)
        print('outputs[step]'); print(outputs[step])
        --reshape output data
        _, targetsTable = catOut(targets, step, noutputs, opt)
        err     = err + cost:forward(outputs[step], targetsTable)
        print('err', err)
      end
      print(string.format("Step %d, Loss error = %f ", iter, err ))

The call  outputs[step] = neunet:forward(inputs[step]) produces a 6  X 6 output matric which I still do not understand. It should be a 6 X 1 table with each table element consisting of 6 X 1 vectors. Anyhoo, I wrote a minimal function that reshapes the output the way I would expect to see it, i.e.   the line _, outputs[step] = catOut(outputs, step, noutputs, opt).
When I start my backward propagation through time,
         local gradOutputs, gradInputs = {}, {}
      for step = rho, 1, -1 do  --we basically reverse order of forward calls
        gradOutputs[step] = cost:backward(outputs[step], targets[step])

        --resize inputs before backward call
        inputs_bkwd = gradInputResize(inputs, step, noutputs, opt)
        --inputs_bkwd = inputs[step]:view(6, 1):expand(6,6)
        print('inputs_bkwd'); print(inputs_bkwd)
        print('gradOutputs'); print(gradOutputs[step])
        print('#inputs_bkwd'); print(#inputs_bkwd)
        print('#gradOutputs'); print(#gradOutputs[step])
        gradInputs[step]  = neunet:backward(inputs_bkwd, gradOutputs[step])
        -- print('gradInputs'); print(gradInputs)
      end

The algorithm gets stuck with a size mismatch error when I call backward. These arer the structure of the data being passed through backward
inputs_bkwd
{
  1 : DoubleTensor - size: 6
  2 : DoubleTensor - size: 6
  3 : DoubleTensor - size: 6
  4 : DoubleTensor - size: 6
  5 : DoubleTensor - size: 6
  6 : DoubleTensor - size: 6
}
gradOutputs
{
  1 : DoubleTensor - size: 6x1
  2 : DoubleTensor - size: 6x1
  3 : DoubleTensor - size: 6x1
  4 : DoubleTensor - size: 6x1
  5 : DoubleTensor - size: 6x1
  6 : DoubleTensor - size: 6x1
}

The error is the same as before viz,
/home/lex/torch/install/bin/lua: /home/lex/torch/install/share/lua/5.1/nn/Linear.lua:75: size mismatch, m1: [6 x 1], m2: [6 x 1] at /home/lex/torch/pkg/torch/lib/TH/generic/THTensorMath.c:766
stack traceback:
	[C]: in function 'addmm'
	/home/lex/torch/install/share/lua/5.1/nn/Linear.lua:75: in function 'updateGradInput'
	...me/lex/torch/install/share/lua/5.1/nn/Sequential.lua:55: in function 'updateGradInput'
	...ome/lex/torch/install/share/lua/5.1/rnn/Recursor.lua:45: in function '_updateGradInput'
	...orch/install/share/lua/5.1/rnn/AbstractRecurrent.lua:46: in function 'updateGradInput'
	...ome/lex/torch/install/share/lua/5.1/rnn/Repeater.lua:40: in function 'updateGradInput'
	/home/lex/torch/install/share/lua/5.1/nn/Module.lua:30: in function 'backward'
	rnn.lua:472: in function 'train'
	rnn.lua:685: in main chunk
	[C]: in function 'dofile'
	.../torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
	[C]: ?

What could I be missing in the way I am setting up my decorator? Would appreciate your input. Thank you for your time in replying earlier.

I tried using the nn.AbstractRecurrent([rho]) class instead. But I find that the module has issues with the interpreter:
/home/lex/torch/install/bin/lua: rnn.lua:239: unexpected symbol near '['
stack traceback:
	[C]: in function 'dofile'
	.../torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
	[C]: ?

@lakehanne You are using noutputs = 6 for both outputsize via nn.Linear(...,outputsize) and as the seqlen in nn.Repeater(...,seqlen). So that is why your output is 6x6.

Thank you very much indeed. That helped.
My network is now constructed properly with
   r = nn.Recurrent(start,
                     rnnInput,  --input module from inputs to outs
                     feedback,
                     transfer,
                     rho
                     )

   --     r      = nn.AbstractRecurrent(rho)
    --we then join the feedforward with the recurrent net
    neunet     = nn.Sequential()
                  :add(ffwd)
                  :add(r)
                  :add(nn.Linear(nhiddens, 1))

    neunet    = nn.Repeater(neunet, noutputs)

and this yields a network graph of the following sort:
rnn
nn.Repeater {
  [  input,    input,  ...,  input  ]
       V         V             V
  nn.Recursor @ nn.Sequential {
    [input -> (1) -> (2) -> (3) -> output]
    (1): nn.Sequential {
      [input -> (1) -> (2) -> output]
      (1): nn.Linear(1 -> 1)
      (2): nn.ReLU
    }
    (2): nn.Recurrent {
      [{input(t), output(t-1)} -> (1) -> (2) -> (3) -> output(t)]
      (1):  {
        input(t)
          |`-> (t==0): nn.Add
          |`-> (t~=0): nn.Linear(1 -> 1)
        output(t-1)
          |`-> nn.Linear(1 -> 1)
      }
      (2): nn.CAddTable
      (3): nn.ReLU
    }
    (3): nn.Linear(1 -> 1)
  }
       V         V             V
  [output(1),output(2),...,output(6)]

But when I do BPTT, the first iteration of the geadInputs gets computed whereas subsequent calls throw errors like
/home/lex/torch/install/bin/lua: ...ome/lex/torch/install/share/lua/5.1/rnn/Recursor.lua:41: assertion failed!
stack traceback:
	[C]: in function 'assert'
	...ome/lex/torch/install/share/lua/5.1/rnn/Recursor.lua:41: in function '_updateGradInput'
	...orch/install/share/lua/5.1/rnn/AbstractRecurrent.lua:46: in function 'updateGradInput'
	...ome/lex/torch/install/share/lua/5.1/rnn/Repeater.lua:40: in function 'updateGradInput'
	/home/lex/torch/install/share/lua/5.1/nn/Module.lua:30: in function 'backward'
	rnn.lua:462: in function 'train'
	rnn.lua:676: in main chunk
	[C]: in function 'dofile'
	.../torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
	[C]: ?

My BPTT code snipper is
      local gradOutputs, gradInputs = {}, {}
      for step = rho, 1, -1 do  --we basically reverse order of forward calls
        gradOutputs[step] = cost:backward(outputs[step], targets[step])
        gradInputs[step]  = neunet:backward(inputs[step], gradOutputs[step])
        neunet:updateParameters(opt.rnnlearningRate)
      end

The whole code is available here. Perhaps I am not correctly passing the right inputs size to backward but I do not seem to be able to  find any thing wrong so far. Maybe a third eye can help out.

@lakehanne You shouldn't need to use a for-loop that is handled internally by the Repeater:
cost = SequencerCriterion(cost) -- or if you have only one target : RepeaterCriterion(cost)
outputs = neunet:forward(input)
loss = cost:forward(outputs, targets)
gradOutputs = cost:backward(outputs, targets)
neunet:zeroGradParameters()
neunet:backward(input, gradOutputs)

Also since you are using Repeater, you should only need one input.

Wow! Couple of questions:

The snippet you have up there doesn't seem to be implementing batches of inputs. I have noticed that most of the examples you have also do not do batches. Are batches internally handled in the repeater module as well?

Also, the targets in the snippet above are not incremented after calling forward steps as some of your examples show. No need for this in the Repeater module too?

Lastly, does this mean I should basically make all targets to be equal to true outputs without worrying about incrementing sequence indices after every forward call?

Thanks for your quick response and I appreciate your help so far.

The snippet uses batches. As far as I can tell, all our examples uses batches. Again the input is a tensor of size batchsize x inputsize. The output has size seqlen x batchsize x outputsize. A batch does not require a for loop. The difference between batch and online is handled internally by modules.

Targets are up to you to determine. When incremented in the examples, it is for language models. In your case, it is up to you to define your dataset. You are using Repeater so I assume it isn't a language model.

Depends. What are the inputs and targets of your dataset?

Thank you very much indeed, @nicholas-leonard !

I had it all mixed up until you explained in your previous post. I have the repeater working now. I simply set up my input as batchSize X inputSize which in my case is 6 X 1, generated an output of size seqlen X batchSize X outputSize where seqlen = 6 and outputSize = 1. Since I am using 5 rho time steps, my predictions after calling forward are of size 5 X 6 X 1  as you rightly noticed. Everything works perfectly and I am so indebted to you for helping out.

2 and 3)  I simply incremented my targets in steps of 1 based on the delay I found in the data. I must say my problem does not fall into the standard machine learning domain. It is the system identification and control of a biomedical robot targeted towards cancer radiotherapy. My input is applied current to a pneumatic valve which controls a soft robot while the output is the corresponding motion on a patient head (6-DOF motion, hence 6 outputs).
Few questions again :-),

I noticed my gradInputs are always zero. I suspect something is wrong.
the degradation in the reduction of the loss error is not consistent. For example, my terminal is spilling out stuff like so:

Step 9718, Loss error = 91287.138996
Step 9719, Loss error = 91287.138996
Step 9720, Loss error = 91287.138996
Step 9721, Loss error = 96812.036729
Step 9722, Loss error = 96812.030329
Step 9723, Loss error = 94386.266278
Step 9724, Loss error = 98474.770919
Step 9725, Loss error = 91302.628445
Step 9726, Loss error = 94854.937780
Step 9727, Loss error =<PHONE_NUMBER>78
Step 9728, Loss error = 91334.555197
Step 9729, Loss error = 91289.035644
Step 9730, Loss error = 91287.214862
Step 9731, Loss error = 91287.142031

i.e., it decreases, increases and then decreases. I suspect this shouldn't be. Here is the code section that is relevant. I am sorry I am taking so much of your time but would definitely appreciate your insight:
    local iter = 1
    for t = 1, train_input:size()[1], 1 do
      offsets = train_input[{ {t} }]
      offsets = torch.LongTensor():resize(offsets:size()[1]):copy(offsets)

      --BPTT
      inputs, targets = {}, {}
        --batch of inputs
        inputs = train_input:index(1, offsets)
        targets = {train_out[1]:index(1, offsets), train_out[2]:index(1, offsets),
                          train_out[3]:index(1, offsets), train_out[4]:index(1, offsets),
                          train_out[5]:index(1, offsets), train_out[6]:index(1, offsets)}
        --increase offsets indices by 1
        offsets = train_input[{ {t+1} }]
        offsets = torch.LongTensor():resize(offsets:size()[1]):copy(offsets)

      --2. Forward sequence through rnn
      neunet:zeroGradParameters()
      neunet:forget()  --forget all past time steps

        outputs, err = {}, 0
        outputs = neunet:forward(inputs)
        err     = err + cost:forward(outputs, targets)
        print(string.format("Step %d, Loss error = %f ", iter, err ))

      --3. do backward propagation through time(Werbos, 1990, Rummelhart, 1986)
      local gradOutputs, gradInputs = {}, {}
        --we basically reverse order of forward calls
        gradOutputs = cost:backward(outputs, targets)
        gradInputs  = neunet:backward(inputs, gradOutputs)
        --print('gradOutputs'); print(gradOutputs)
      --4. update lr
      neunet:updateParameters(opt.rnnlearningRate)
      iter = iter + 1
    end

@lakehanne So given the current which is a single number (base on your inputsize =1), you are trying to predict the motion of a robot along 6 DOF. If your data is sequential, you should use a different model. So for example, assuming your dataset is a sequence of N time-steps where x[t] depends on x[t-1] and so on, which could look like:
dataset = {inputs = torch.Tensor(N), targets = torch.Tensor(N, 6)}

Then you should use something like a Sequencer instead of Repeater. Is this the kind of dataset you have?

Yes this is the sort of dataset I have. I was using the sequencer before but you advised against it as I was getting stuck with a size mismatch error when I call backward. See comment 2 above (posted 11 days ago).

Yes. Obviously there was a lot of confusion in the above thread. What you want is to use Sequencer. The inputs to your model will be of size seqlen x batchsize x inputsize, the outputs will be seqlen x batchsize x outputsize. The seqlen is a hyper-parameters. It should be something like 10, 20, 50 or 100. When you go through your dataset, your chunks seqlen rows where seqlen << N.

Thanks, Leonard! Just to be roundly sure, the gradInputs = 0 that results from my repeater training and irregularity in loss/error is because I did not use the sequencer module?

@lakehanne Corrected in original comment.

What's the purpose of the seqlen hyperparameter? And it's supposed to be randomly chosen?

Yes. Yes. Yes. :)

Thanks! If that is the case, it keeps getting stuck during calls to backward like I originally had in post 1.
gradInputs cannot compute because of inconsistent tensor size. I am not sure why you suggested I abandon the Repeater decorator early on.
Here is a public gist of the code section that's causing World War III in my head :)
Sorry to be taking so much of your time, but I would love to hear what you think regarding what you think might be contributing to these errors.

Fixed in my FARNN SR-AllModels  tag. Thanks for your help @nicholas-leonard !


Copyright infringement in utils.py

Description
A Black Duck scan has highlighted a copyright infringement on https://github.com/zalando/connexion/blame/master/connexion/utils.py#L65.
Expected behaviour
The code is no longer a copyright infringement.
Actual behaviour
The code is a copyright infringement.
Steps to reproduce
... run another Black Duck scan?
Additional info:
None
Output of the commands:

python --version
pip show connexion | grep "^Version\:"

I get it. It's April Fools' Day.

No, really.  Black Duck highlighted this infringement - is it will any other organisation who goes through a due diligence process.  This is no an April Fools.

Did it highlight the comment? I mean the code is really just on line and in no way can be copyrighted.
Can you add a screenshot of the Black Duck finding?

Hi ... I'm part of Malcolm's team and have been reviewing the BlackDuck scan. Yes, I was really hoping it was a joke too, but no, they are serious. Just to show you exactly how it was flagged ...
This is how it was shown on the summary view:

And then in the details tab it was shown as this:

So basically it's saying that on line 65 of utils.py it's detected some unlicensed code.
Obviously we don't know how BlackDuck really works, but from what it's found I think it is mainly looking at headers and comments and looking for anything that suggests cut-and-paste code. I spoke to a lawyer about it who said that you could argue that one line of code is an "excerpt" under copyright law. Also that if you had anything from the author saying you could use this that'd probably be fine. But at the moment anything showing as Red in BlackDuck is of course making legal-types worried.
Apologies since this feels so trivial, but really would appreciate any support that prevents us having to do some kind of patching. Thanks!


[Bug] When I re-render the :fields property

This problem occurs when I have an object in :data: Fields
This doesn't happen the first time you click download
When I reassign (:fields)
Click Download again and the Object will appear

The problem arises:  Json.parse (json.stringify ({' field ': 'value'}))
It was a deep copy pit

He will erase my methods


[NetAppFiles] bump to 2024-03-01-preview

Migrate from msrestazure.tools to azure.mgmt.core.tools
This checklist is used to make sure that common guidelines for a pull request are followed.
Related command

General Guidelines

[x] Have you run azdev style <YOUR_EXT> locally? (pip install azdev required)
[x] Have you run python scripts/ci/test_index.py -q locally? (pip install wheel==0.30.0 required)
[x] My extension version conforms to the Extension version schema

For new extensions:

[x] My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify src/index.json.

NetAppFiles

@AllyW updated

aaz  https://github.com/Azure/aaz/pull/560

@audunn, please verify the above breaking change are expected and write then in the release note. Thanks.

@kairu-ms  This should also satisfy the requrement to drop msrestazure dependency from CLI extensions.  Is there specific confirmation of this needed ?
https://microsoft.sharepoint.com/:x:/t/AzureCLITools237/ETrztIl6PXhMpt22vr_5aSoBl7vdAq8lXRocPJhelkUPxA?e=ETZxaJ&CID=BE6A6F46-D970-4795-BE7C-B8692DB941ED&wdLOR=cA301C305-6C4B-43FB-B162-8B01A95B8FAE


AZLearner: TD vs MSE-to-outcome

The AZLearner  implementation here appears to do TD on transitions.
As far as I can tell from page 2 of the AlphaZero paper, the original AlphaZero optimizes MSE between value-predictions and game-outcomes.
Is this an intentional variation? (Or am I misunderstanding either of the files?)
(Sorry in case this is the wrong place to put this)

Hi, thanks for the question!
Yes, this (and some other details) are variations from the AlphaZero paper -- this is not intended to be a reproduction of that algorithm, but more of a general model-based RL learner, which borrows the main innovation of AZ, which is to integrate model-based planning with learned policy/value.
To answer your question, TD learning is the more general approach of the two; if there is no discounting and the reward signal is +/- 1 at game end and zero otherwise, this boils down to the MSE-to-outcome used in the AZ paper. Using TD allows us to handle environments with rewards on non-terminal transitions, such as e.g. bsuite.
Hope that helps!

Thanks!
One more clarification question:
Are there specific reasons that favor this approach over just replacing the binary outcome in AlphaZero's loss by the sum of discounted future rewards? (This should achieve the same generalizations, right?)
I'd think the latter might be less susceptible to instabilities in the learned value function, but maybe other effects are more relevant in practice, or it just doesn't matter very much?

What you're describing is using the complete Monte Carlo return (e.g. as used in REINFORCE) as the target rather than bootstrapping from the value function. This is known to have higher variance and is generally eschewed for boostrapping. There is however TD(lambda) which essentially allows us to mix between the two. See Ch 5-7 of Sutton & Barto: http://incompleteideas.net/book/RLbook2020.pdf


add C# gRPC doc

cc @joostjager

So it does indeed compile? 🤔

I rebased this onto master and went through the steps and examples again, on Linux.
Because there is no full example code, some modifications are needed. But this is limited to adding using statements and changing paths. With that, all examples work.
@Roasbeef I'd recommend to merge this as it is now and possibly amend a bit if things prove to be unclear when people start using this.


[Backport to 2.x]Catalog to Datasource changes (#1027)

Signed-off-by: vamsi-amazon<EMAIL_ADDRESS>(cherry picked from commit 3e30379903ba596b8565e680af69073a16cdf940)
Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

should 2.4 be merged into 2.x?

Yes, But I am not sure if there are any pending PRs which are not merged into 2.x

should 2.4 be merged into 2.x?

Yes, But I am not sure if there are any pending PRs which are not merged into 2.x

This is the only one I could find.

Codecov Report

Merging #1049 (8a05fe7) into 2.x (1a52511) will decrease coverage by 2.53%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##                2.x    #1049      +/-   ##
============================================
- Coverage     98.21%   95.68%   -2.54%
- Complexity     3268     3326      +58
============================================
  Files           314      334      +20
  Lines          8197     9057     +860
  Branches        533      672     +139
============================================
+ Hits           8051     8666     +615
- Misses          142      334     +192
- Partials          4       57      +53

Flag
Coverage Δ

query-workbench
62.76% <ø> (?)

sql-engine
98.26% <100.00%> (+0.04%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...rch/sql/planner/physical/catalog/CatalogTable.java
100.00% <ø> (ø)

...ensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java
100.00% <ø> (ø)

...metheus/storage/PrometheusMetricDefaultSchema.java
100.00% <ø> (ø)

...ain/java/org/opensearch/sql/analysis/Analyzer.java
100.00% <100.00%> (ø)

...sql/planner/physical/catalog/CatalogTableScan.java
100.00% <100.00%> (ø)

...l/planner/physical/catalog/CatalogTableSchema.java
100.00% <100.00%> (ø)

...sql/opensearch/response/agg/SingleValueParser.java
100.00% <100.00%> (ø)

...earch/sql/opensearch/response/agg/StatsParser.java
100.00% <100.00%> (ø)

.../opensearch/sql/opensearch/response/agg/Utils.java
100.00% <100.00%> (ø)

...java/org/opensearch/sql/ppl/parser/AstBuilder.java
100.00% <100.00%> (ø)

... and 29 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

ok i checked the other missing one is https://github.com/opensearch-project/sql/pull/1050, should be good


Support alertmanager

Just for support the prometheus alertmanager.

Hi @hectorj2f
Now i just add one simple rules, maybe i should add more alert rules ...but the alert feature is OK.
i need some time to learn the prometheus rules.
so you can review and merge it.
Note. some grafana default dashboards have N/A value, need update.

Hi, is this PR ready/finished/tested ? Thanks for your great work :-)

Yes, have finished and Tested OK.

Hi @hectorj2f
can you review this PR again? if no problem, please merge it.

@xiaoping378 really sorry this got sidetracked, your changes look good, sadly it conflicts a bit with  recently merged changes. Do you want to resolve them yourself or should I help a bit.
Basically you could reduce your PR to include only the alert manager yamls. Only change needed there would be adding the namespace inside the yaml. I would maybe also update to latest alert manager (0.5).
I would then add add something to docs/index.md (no need for you to do this in this PR) and also create the manifests-all.yaml if you haven't done so already.

OK, i would make the smallest PR for alertmanager ...

Thanks, sorry again for the additional effort ;)


[Monitor] Add Public Preview APIs of Metric Baseline

These APIs include:

Getting the pre-calculated baseline of a metric
Calculating the baseline of a metric given the metric values

This checklist is used to make sure that common issues in a pull request are addressed. This will expedite the process of getting your pull request merged and avoid extra work on your part to fix issues discovered during the review process.
PR information

[x] The title of the PR is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
[x] Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
[ ] If applicable, the PR references the bug/issue that it fixes.
[x] Swagger files are correctly named (e.g. the api-version in the path should match the api-version in the spec).

Quality of Swagger

[x] I have read the contribution guidelines.
[x] My spec meets the review criteria:

[x] The spec conforms to the Swagger 2.0 specification.
[x] The spec follows the guidelines described in the Swagger checklist document.
[x] Validation tools were run on swagger spec(s) and have all been fixed in this PR.

@simongdavies could you have a look?

Looks good.

No modification for AutorestCI/azure-sdk-for-ruby

 https://github.com/Azure/azure-sdk-for-python/pull/1710


Refactor app

As the app is built on top of vuetify dashboard there is a lot of carry over code that currently is not in use. Review it and clean all the code that is not needed.

Implemented.


queue.pending now contains full message. puppetjs/puppetjs#71

besides the question in code comment, could you add a test for this?

Yeah, it should be easy enough. It's worse with the actual reconnection stuff


Add mayam command

Allows
mayam eye yam eyepatch yam

Also, can I request a change to the input?
I think it would be nice if this command supported resonances in the future (doesn't have to be this PR).
In that case, a command might be mayam resonance battery. So instead of mayam eye yam eyepatch yam, perhaps mayam rings eye yam eyepatch yam?
Alternatively, it could be assumed that if the first parameter isn't "resonance" it's the normal case, but I think most commands with subcommands do them explicitly.


zstack installation should help to set mysql root password.

fixed by https://github.com/zstackorg/zstack-utility/commit/8eeaab032cc7f8e0eee1b75012b856fe4dd59508
ZStack installer will help to set mysql root password.


fix miner download miss block

Codecov Report

Merging #173 (2c78e0b) into master (368b4e6) will increase coverage by 1.53%.
The diff coverage is 11.11%.

@@            Coverage Diff             @@
##           master     #173      +/-   ##
==========================================
+ Coverage   46.15%   47.68%   +1.53%
==========================================
  Files         124      124
  Lines       10394    10427      +33
==========================================
+ Hits         4797     4972     +175
+ Misses       4637     4404     -233
- Partials      960     1051      +91

Impacted Files
Coverage Δ

bcs/network/p2pv2/server.go
53.84% <0.00%> (ø)

kernel/engines/xuperos/common/error.go
0.00% <ø> (ø)

kernel/network/p2p/subscriber.go
40.27% <0.00%> (-1.76%)
:arrow_down:

kernel/engines/xuperos/config/config.go
61.90% <100.00%> (+1.90%)
:arrow_up:

kernel/contract/sandbox/utils.go
66.66% <0.00%> (-22.23%)
:arrow_down:

bcs/ledger/xledger/state/xmodel/xmodel_iterator.go
57.14% <0.00%> (-7.15%)
:arrow_down:

bcs/consensus/pow/pow.go
37.28% <0.00%> (-5.51%)
:arrow_down:

bcs/consensus/tdpos/common.go
62.31% <0.00%> (-2.90%)
:arrow_down:

bcs/contract/xvm/code_manager.go
63.63% <0.00%> (-2.73%)
:arrow_down:

kernel/consensus/base/driver/chained-bft/smr.go
50.30% <0.00%> (-0.93%)
:arrow_down:

... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 368b4e6...2c78e0b. Read the comment docs.

Codecov Report

Merging #173 (2c78e0b) into master (368b4e6) will increase coverage by 1.58%.
The diff coverage is 11.11%.

@@            Coverage Diff             @@
##           master     #173      +/-   ##
==========================================
+ Coverage   46.15%   47.74%   +1.58%
==========================================
  Files         124      124
  Lines       10394    10427      +33
==========================================
+ Hits         4797     4978     +181
+ Misses       4637     4400     -237
- Partials      960     1049      +89

Impacted Files
Coverage Δ

bcs/network/p2pv2/server.go
53.84% <0.00%> (ø)

kernel/engines/xuperos/common/error.go
0.00% <ø> (ø)

kernel/network/p2p/subscriber.go
40.27% <0.00%> (-1.76%)
:arrow_down:

kernel/engines/xuperos/config/config.go
61.90% <100.00%> (+1.90%)
:arrow_up:

kernel/contract/sandbox/utils.go
66.66% <0.00%> (-22.23%)
:arrow_down:

bcs/ledger/xledger/state/xmodel/xmodel_iterator.go
57.14% <0.00%> (-7.15%)
:arrow_down:

bcs/consensus/pow/pow.go
37.28% <0.00%> (-5.51%)
:arrow_down:

bcs/consensus/tdpos/common.go
62.31% <0.00%> (-2.90%)
:arrow_down:

bcs/contract/xvm/code_manager.go
63.63% <0.00%> (-2.73%)
:arrow_down:

kernel/consensus/base/driver/chained-bft/smr.go
50.30% <0.00%> (-0.93%)
:arrow_down:

... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 368b4e6...2c78e0b. Read the comment docs.


[WFLY-8917] Propagate the run-as identity along even if an ejb is uns…

…ecured
WF: https://issues.jboss.org/browse/WFLY-8917

Can one of the admins verify this patch?

This is ok to test


April 4 New Update

The New Update is now up on Steam with new Solo Modes, Leaked cards and items will you implement it on next update? thanks
https://ygorganization.com/master-duel-april-4th-update/

yeah it will be nice if you keep this project alive for a while

? i don't know how Pixeltris work and also i'm interested but i don't know how too

v1.5 adds this content. Please note that I don't have any commitments to updating immediately when a new update is released which I stated previously https://github.com/pixeltris/YgoMaster/issues/19


Main Storyboard segue is nil

I am trying to use KYDrawerController but the main and drawer segue is nil. I tried it a lot but i never found the solution. I am a attaching my storyboard picture. Please help me with this issue. Thanks

me too! please supporst thank you

do u have any project sample. I still fail to apply this library.

@mhtranbn
Please try setting KYDrawerController in the module.

Thanks. One more thing can this library blur the back view controller?