Quest

Changes / Comments
Initial upload for character sheets for board game Quest designed by William Hoyt. Quest is a board game based on MAR Barker's Empire of the Petal Throne. Bill was one of MAR Barker's original players in 1974 and helped get Empire of the Petal Throne published by TSR. Bill is planning to run Quest at Gary Con this year and this is part of the preparation for running the game virtually. It is a very basic sheet, I'm hoping to do some CSS clean up this week, but we wanted to get this up on Roll20 to play test the basics.
Roll20 Requests
Comments are very helpful for reviewing the code changes. Please answer the relevant questions below in your comment.

[X ] Does the pull request title have the sheet name(s)? Include each sheet name.
[ ] Is this a bug fix?
[ ] Does this add functional enhancements (new features or extending existing features) ?
[ ] Does this add or change functional aesthetics (such as layout or color scheme) ?
[ ] If changing or removing attributes, what steps have you taken, if any, to preserve player data ?
[ X] If this is a new sheet, did you follow Building Character Sheets standards ?

If you do not know English. Please leave a comment in your native language.

Character Sheet Info Roll20 Internal Use only.

Hey there @MichaelCalleia, could you remove the DS_Store file from the root folder and update your pull request? Thanks!

Doh! I believe this is resolved.
Warm regards,
Michael Calleia
917-753-7259
<EMAIL_ADDRESS><EMAIL_ADDRESS>calleia.com

On Feb 22, 2021, at 2:07 PM, Miguel Peres<EMAIL_ADDRESS>wrote:
Hey there @MichaelCalleia https://github.com/MichaelCalleia, could you remove the DS_Store file from the root folder and update your pull request? Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/Roll20/roll20-character-sheets/pull/8339#issuecomment-783603348, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANWGT7XP6WDOUTVPNMZCRDTAKTO7ANCNFSM4X7VWNEQ.

Going to use custom instead for this.

This was corrected a while back, any chance the pull request will be merged the week?
Warm regards,
Michael Calleia
917-753-7259
<EMAIL_ADDRESS><EMAIL_ADDRESS>calleia.com

On Feb 22, 2021, at 2:07 PM, Miguel Peres<EMAIL_ADDRESS>wrote:
Hey there @MichaelCalleia https://github.com/MichaelCalleia, could you remove the DS_Store file from the root folder and update your pull request? Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/Roll20/roll20-character-sheets/pull/8339#issuecomment-783603348, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANWGT7XP6WDOUTVPNMZCRDTAKTO7ANCNFSM4X7VWNEQ.


Remove my App from Dialer chooser

Hi,
some of my users report that when they try to make a call from the default phone app, it suggests to call from the default phone app or from my application.
Would you know how to prevent my application from appearing in these choices?

I am facing the same issue.
Do anyone found the solution?

I am facing the same issue.
Do anyone found the solution?


Docker-compose is not installed by the setup script.

A couple of people have hit the same problem when setting up on Ubuntu. It appears that docker-compose is not part of the docker.io package and needs to be installed separately.
The setup script should install docker-compose too.

I think it is best to get the to install it themselves as we ask them to do with docker. Docker compose comes bundled with the mac docker installer


[FLINK-27884] Move OutputFormatBase to flink-core to offer flush mechanism to all output formats

What is the purpose of the change
Move OutputFormatBase to flink-core to offer flush mechanism to all output formats
Brief change log
Move OutputFormatBase from connectors-cassandra module to flink-core to offer flush mechanism to all output formats.
Main change was to change (not yet public) send method signature so that it returns a CompletionStage and not a guava ListenableFuture. Implementers can chose their CompletionStage implementation as long as it implements CompletionStage#toCompletableFuture()
Verifying this change
This change is already covered by existing tests, such as OutputFormatBaseTest and CassandraConnectorITCase (for first implementation)
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes (not applicable / docs / JavaDocs / not documented)
If yes, how is the feature documented? javadocs

R: @zentol  as discussed in the other PR.

@flinkbot run azure

I don't understand the test_ci finegrained_resource_management failure it seems unrelated to this PR. Anyway I launched the corresponding command line locally removing the alibaba mentions and it passes.

@flinkbot run azure

Passes now, it was a flaky test.

I don't understand the test_ci finegrained_resource_management failure it seems unrelated to this PR. Anyway I launched the corresponding command line locally removing the alibaba mentions and it passes.

Passes now, it was a flaky test.

@zentol friendly ping

rebased on master

@flinkbot run azure

@zentol All discussions are resolved and CI passes. Is this PR ready for merging ?

@zentol thanks for your review and for adding the visibility I forgot. Build is green.


Proper docker support

See: #243 -- This superseeds #243 which should now be rejected / closed.
I think I have un-broken everything that I broke, it's compiling, working, and was rebased from v0.3.0 so should remain stable :-)
Change log is also fixed.

I might let the other @pagermon/collaborators look at this one - your branch seem to be wanting to pull lots more then what it should be.
you should be ok to use your other PR - all that would be needed is to resolve the CHANGELOG conflict

I'm not sure what GIT is doing with all those commits, tried to clean it up from the web interface but it's not having a bar of it. You also seem to have had some issues with the merge commit in the changelog, somethings gone very wrong there 😉
I'd like to clean that up first as recommiting the same changes will look messy as.
Project for this afternoon 😄

Yeah -- I am fixing it quickly.


Validation Loss Statistics: min=nan, med=nan, mean=nan, max=nan

Thank you for the code!
When I ran the code on mnist like this python train.py --dataset mnist, I got the output  Validation Loss Statistics: min=nan, med=nan, mean=nan, max=nan after training for several epochs.

Please help me, I have no idea what the matter is.

I am having the same issue. Any fixes for this?

The model collapse very quickly with exploded loss. I want to figure out why.

It seems nothing to do with initialization methods.

I found it! the adam parameter beta_2 should never be 0.01 :)

I found it! the adam parameter beta_2 should never be 0.01 :)

Still collapse after few epochs :(

Seems like collapse to one mode after 2/3,000 iterations, is here anybody can give a reason?

Seems like collapse to one mode after 2/3,000 iterations, is here anybody can give a reason?

The norm of h, i.e., f(x) gets bigger and bigger as training evolves, leading to the torch.exp(h) term approaches to Infinity. I looked into the model parameters and found the norm of them also increased a lot in training. One way to mitigate the issue is by using L1 regularization. It would also help if you use F.softplus to make the logistic prior calculation stable.


Update ToC style for docs

Add filter for docs page with where: "layout", "docs" to remove useless empty items.
Update style and colors for better legibility.

Thanks for your contribution.


Please include source from "spam404.com"

Hi,
Possible to include spam404 database as your sources of hosts?
https://raw.githubusercontent.com/Spam404/lists/master/adblock-list.txt
The db is available as filterlist in ublock origin.
Thinking if you can unified this source will be a good idea.
Thanks.

Thank you for this @qarqun
That file is a mess.  It would take awhile to write a parser to make this file usable.
I decline.
Closing.

Hi

StevenBlack : That file is a mess.

Calling a file a mess just because it is in a different format is offensive, the file is not a mess, it's a tidy file, just that in a different format - Adblock format, the list is also available in a pure domain format:
https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt,
main repo https://github.com/Spam404/lists

Please be free to take offense. I don't care.
That file is an obvious mess.
I wasn't referring to the list format. So we're clear.
The most basic attribute of a quality, properly curated list is it must be sorted.  Preferably annotated with comments.
I don't care how it's sorted.  By domain, by TLD, chronologically, categorically... It shouldn't be an obvious dumpster fire. Like this list for example.

I wasn't referring to the list format. So we're clear.

Okay, clear now, I didn't even notice that it was unsorted anyway
By the way, a story: someone has been trying here: https://github.com/AdguardTeam/AdguardFilters/issues/51906 to explain to AdGuard Filters guys they should try to sort their big list and remove duplicates :smiley:


Fixes two TypeError exceptions when up/down arrow pressed

It looks like issue #2035 is happening again in the latest v1.13 release. When using live search, if a user enters a query that has no matches and then presses the up or down arrows, two different TypeError exceptions can occur. This PR fixes both exceptions.
For reference, here are the steps to reproduce the issue, copied from the original issue:
Browser: All
OS environment: All
Steps to reproduce:

Open selctbox and type "xxx" to search
No results matched "xxx" will appear
Press DOWN key
Error is thrown

I can confirm that this issue exists, but this will require a different fix for v1.14.0, as it's been refactored.


Mailing service

Hi, do you have any plan to add mailing service? That can also handle sending bulk emails.
Thank you in advance🙏

Do you refer supporting some third-party mailing provider? If yes which one.
Or you want to support a custom bulk email provider?

Instead of integrating mailing provider one by one, I would recommend just integrating one of the existing go notification library (eg. https://github.com/nikoksr/notify ).


perf: middleware to short-circuit on 304

Description
I'm checking the work needed in the server to answer on 304 once it is warmed up. We return 304 when:

On a forced reload from the user
On HMR when there is a full reload for all code except for the modified modules
On second startup (both warm and cold), half or more of the modules could already be warmed up on large apps and they will be also hitting 304.

There isn't much work we do to reach the current condition, but we still do a few things.

We add CORS headers. If they are the same, we could skip this, but probably not worth the complexity to take into account the headers when computing the etag. See https://httpwg.org/specs/rfc7234.html#freshening.responses, the CORS headers will be refreshed on 304 if changed.
We have a few middlewares: proxy, open-in-editor, hmr-ping, public. The public one in particular has the most expensive check
Then in the transform middleware, we run a few regexes (CSS, JS, etc) and then need to run a getModuleByUrl that also adds up.

This PR implements a fast path for 304, with a new middleware after CORS that kicks in when there is an etag, and uses a etagToModuleMap.
I'm seeing a ~3-4% improvement when forcing a reload on my M1. I would say the extra map is ok here. It would be interesting to see the effect on slower machines too.

What is the purpose of this pull request?

[ ] Bug fix
[ ] New Feature
[ ] Documentation update
[x] Other

/ecosystem-ci run

I saw the same Nuxt fail in other PRs when running ecosystem-ci, I think it isn't related to this PR

I noticed that this breaks vite-plugin-static-copy. That plugin injects a middleware just before the serve public middleware. I will add something similar to https://github.com/vitejs/vite/pull/15586/files#diff-abb3345b6e3b2ec6d297c2ebc54cca85ae4487a31bac3cc9e78457f5114adb26R687-R697 on the plugin side as I guess the middlewares.stack is not part of the public API.

injects a middleware just before the serve public middleware

The names in middlewares.stack are part of the public API, but we have added and removed middlewares in minors in the past. In this case, IIUC, vite-plugin-static-copy will still work and only break when a static file is added with the same name as a source file (and that source file isn't deleted). I think it is an edge case worth solving in the plugin, but we could still move forward with this one in a minor.

I'm not sure if we should mark it as internal yet, because if not @sapphi-red won't be able to fix the edge case he talked about here 🤔
Let's merge it without marking as internal, but we can revisit this before releasing Vite 5.1.
I think the API will be useful for plugins in general.


When using lockable, failed_attempts is incremented by two after failed login attempt

When using devise-two-factor with devise lockable, there are somehow two calls being made to validate/valid_for_authentication? - the latter of which is doing the failed_attempts increment.
The first failed_attempts increment happens within this block, and the latter seems to be happening after the authenticate! method is finished executing.
if validate(resource) { !resource.otp_required_for_login ||
  resource.valid_otp?(params[scope]['otp_attempt']) }
  super
end

My devise initializer is configuring the warden strategy as follows:
config.warden do |manager|
  manager.default_strategies(:scope => :user).unshift :two_factor_authenticatable
end

My user model is set up with devise as follows:
devise :two_factor_authenticatable, :otp_secret_encryption_key => ENV['RSA_KEY_PW'], otp_secret_length: 64
devise :registerable, :recoverable, :invitable, :trackable, :timeoutable, :lockable

When using lockable devise increments the failed_attempts counter by one right in the beginning of the authentication even though the authentication has not yet failed nor succeed.
It happens in lib/devise/models/lockable.rb method valid_for_authentication?

To be more accurate it is the current_user and user_signed_in? helpers that are trying to authenticate the user from the request data. If none of those helpers are called during the authentication action then the failed_attempts counter is not increased.

@ShaneWilton could you look at this when you get a chance? This is a rather tricky bug.

@mwalsher, @mmustala, @ShaneWilton
guys, any update on that?

Two years elapsed, any progress?

Guessing no progress on this? Been hitting a wall and was wondering if it was me or devise or devise-two-factor

I have the same problem.
I found that both TwoFactorAuthenticatable#authenticate! and TwoFactorBackupable#authenticate! are run during the authentication workflow. Maybe I am doing something wrong.

another issue #127 still not fixed :(

I'm facing the same issue as well.  Our hacky fix was just to double the maximum_attempts to what we really want.  (ie. we want to lock after 10 failed attempts, so we set maximum_attempts to 20)

Any update on this issue? I've been doubling the maximum_attempts to hack my way in as @StevenJL.

We are encountering the same issue as everyone else


Client crashes when viewing dimlets in JEI

Minecraft v1.8.9
Issue: When you search for dimlets using JEI, and you hover your mouse over the symbol, your client crashes. Through in-game testing, it appears to affect most or all of the grey colored dimlet icons, screenshot of affected icons outlined below
Repeatability: This issue was repeated across multiple clients, one running Windows 8.1, another running Arch Linux
Screenshot:

Crash Report:
crash-2016-03-17_22.10.21-client.txt

I could not reproduce this but I did a blind fix for next version


Updating to 1.0.30001185 breaks application

I was trying to get an updated version of this package and after running the following command, I can no longer build the application:
npm update --depth 10 caniuse-lite

npm update --depth 10 caniuse-lite

This is now the recommended way to update caniuse-lite.
Try npx browserslist@latest --update-db

I can no longer build the application:

What exactly error do you have after this command?

I think this may be an issue with Vue CLI. The highest version I can go up to is 1.0.30001164. If I go higher than that version, then I start getting errors about postcss loaders etc. I'll dig into this more tomorrow and raise an issue with Vue CLI.

Very likely that this error came because you command updated all dependencies inside, not just caniuse-lite. Check the package-lock.json changes.

I confirmed that the issue is not caniuse-lite. I believe the issue is with a change in browserslist and how it processes queries.


[Carousel] If you put an image into a carousel-item, the whole carousel's height gets resized to the height of the image.

Description
When you add an  to the carousel-item it resizes the carousel's height to the image's height. If you edit the image's height it edits the carousel's. I found it impossible to set the height of the carousel, or the carousel-item with css.
Repro Steps
Insert an image into a carousel item.
Or check this Codepen
Screenshots

Would you prefer if you could override the height of the carousel with a plugin option?

Closed due inactivity. Feel free to reopen it, if it's still necessary.

same issue. I'm trying to place multiple images inside a single carousel-item and the whole carousel keeps resizing

Someone found the solution?

just give the carousel a fixed height with !important, seems like the easiest solution

Tks @lastnoob ! I will try it

I am putting images of pages in the carousel and want to view them on maximum size for the screen/device so setting a hard height does not work for me.  What would be ideal is to have a "fullSize" option that resizes the carousel to fit the image, making it as large as possible given the constraints of the screen, but does not crop.


Use PSBT instead of custom format in offer/accept

In retrospective, by far the hardest part of implementing DLCs spec was about calculating the fee rate of the funding transaction. (and this was even harder than implementing all the math stuff)
This PR is removing num_funding_inputs, funding_inputs, change_spk, payout_spk, feerate_per_vb from the protocol entirely, and replace it with partial_funding_psbt which convey the same information.
The advantage of it is that any wallet with basic support of PSBT can generate this partial_funding_psbt. So any wallet supporting PSBT can fund a DLC.
The feerate calculation is completely side stepped. Leaving it up to the end user to decide whether the resulting fund transaction has the appropriate feerate.
This allows both user to contibute what they want to the fee.
The CET/Refund transaction fee is half of the two partial funding psbt fee.
Example:
Sender has collateral of 40 000 sats and pay 4000 of fee. The receiver has collateral of 20 000 sats and pay 3000 of fee.
The funding transaction will have (4000 + 3000) / 2 = 3500  sats of fee. The funding output will be equals to 3500 + 40 000 + 20 000 = 63 500 sats.
So the cet/refunds will also have 3500 sats of fee.

if you don't want to use psbt for some reason, at least you should let feerate_per_vb go and replace it by contributed_fee in both offer and accept message.

Concept ACK
I think this will make things a lot easier. Like you note the fee calculations should no longer be an issue. This should also make it easier to fund from a something like a hardware wallet or any separate wallet (ie bitcoind).
One thing we will need to be mindful of in the spec is detailing how the implementation should be verifying the output values/fees.

NACK for me. Passing PSBT around is going to bloat messages. I think PSBTs are really nice and I don't think there is anything preventing using them outside of the protocol, but it doesn't make sense to me to send them over the wire, especially if we start using LN stack and have some tight packet constrains.
Also something I am not sure I fully understand. Even if you pass a PSBT, you still somehow need to decide how much fee to pay right? How do you do this computation? Don't you anyway need to compute the size of you inputs and potential change output to match a given fee-rate?
Similarly on the receiver side, don't you also need to check that the provided fee is proper?

if you don't want to use psbt for some reason, at least you should let feerate_per_vb go and replace it by contributed_fee in both offer and accept message. I can make another PR if you prefer than route.

In that case I guess you would still need to compute the size of the contributed data in order to determine whether it is an appropriate amount or not right? I'm sorry if I'm missing something here, but it looks to me like PSBT doesn't really solve the problem, unless what you are proposing is to just accept whatever fee the counter party has chosen without checking anything.
Finally I agree that fee computation is a big pain and source of error. But I think that is mainly because how to compute them is not specified at the moment. If we just add some pseudo code in the specs and some test vectors, I don't think it will be that difficult.

Why do you feel PSBT to be bloated? there is no superfluous information in it.
For example, the current protocol does not support funding from P2SH segwit wallets.
PSBT already specified all of it, so there is no need to re-implement the wheel.

In that case I guess you would still need to compute the size of the contributed data in order to determine whether it is an appropriate amount or not right? I'm sorry if I'm missing something here, but it looks to me like PSBT doesn't really solve the problem, unless what you are proposing is to just accept whatever fee the counter party has chosen without checking anything.

It solves the problem. The pain point is that currently both parties need to be in agreement down to the satoshi on how much is spent on fee. This sidestep the problem completely by not requiring it. If both party have small differences in how they calculate the estimated fee rate once the transaction is completely signed is not a problem anymore.
There is way to solve this problem without PSBT

if you don't want to use psbt for some reason, at least you should let feerate_per_vb go and replace it by contributed_fee in both offer and accept message. I can make another PR if you prefer than route.

But I don't see why reinventing the wheel. PSBT already specify how the information needed is formatted, and most wallet can generate it. There is no superfluous data.

So correct me if I'm wrong, but I feel you are saying hat PSBT solves two separate issues:

Formatting of data
Fee computation

But I don't see why reinventing the wheel. PSBT already specify how the information needed is formatted, and most wallet can generate it. There is no superfluous data.

That's 1. I agree with you on that point, but the issue is that I'm afraid that a single PSBT might not fit into the size limit of LN messages. Then we will need to start dividing it into smaller chunks, and I'm not sure we will end up in a better place than having custom messages.

It solves the problem. The pain point is that currently both parties need to be in agreement down to the satoshi on how much is spent on fee. This sidestep the problem completely by not requiring it. If both party have small differences in how they calculate the estimated fee rate once the transaction is completely signed is not a problem anymore.

There is way to solve this problem without PSBT

That's 2. And as you're saying, there is way to solve it without PSBT, and I would actually say that PSBT does not help for this. For compatibility, we could add a fund_output value given by the offerer that the receiver should accept if it's within a reasonable range of his own computation. But you still need to compute how much fee you need to pay and verify how much fee your counter party is paying.
In you change you put:

the fee rate of the partial_funding_psbt is insufficient

But how do I know it is insufficient? I need to first compute it right? And we need to agree to some extent on how should this be computed, so someone wanting to make this computation will have to go looking for the 220 lines you deleted somewhere else, and maybe end up with a computation that is way off what others have implemented. Personally I much prefer having something written down in the specs that I can follow to implement and some tests vector so that I'm sure that things are working as expected.

Those 220 lines were not specific enough to properly calculate the weight properly and were not even talking about how to handle integers divisions.

I agree with you on that. I don't think what is currently in the specs is enough, or even useful. And this is not what I was referring to when I talked about having pseudo code. Actually maybe the best would be to have a reference implementation so that there could be no room for confusion (and I again I think this would be useful regardless of whether we use PSBT or no). Then we could just say "The fee computation has to match this implementation (link) and pass this test vectors" and be done with it (and then maybe remove the 220 lines of explanation as I think in that case maybe code is better than words).
Anyway it's not like I will fight to death over that :) . If everybody is convinced that's the way to go I'll implement it, but I think it is trying to hide issues that will come and bite us later.

NACK regarding PSBT concept.  I believe that PSBT is useful for interoperability between wallets and signers that don't otherwise have a common protocol.  I don't think it makes sense to nest the PSBT format inside another format - it increases complexity and message sizes.  Therefore I would stick to the general format described in the DLC specs.
PSBT can still be used for local signing purposes.
I haven't looked at the fee calculation issues, so I'm not commenting about those.

That's 1. I agree with you on that point, but the issue is that I'm afraid that a single PSBT might not fit into the size limit of LN messages. Then we will need to start dividing it into smaller chunks, and I'm not sure we will end up in a better place than having custom messages.

You will have to do that anyway, since the size depends on the number of funding inputs, which is controlled by the peers.

But how do I know it is insufficient? I need to first compute it right?

"Somebody made this offer, he is paying for 10 sat/bytes, do you accept?".
The value "10 sat/bytes" is computed, but it does not need to be the exact same way across implementation, this is a UI concern, in practice if the fee rate is 10.01 sat/bytes instead of 9.997 sat/bytes, it should not cause the whole exchange to fail, as from user perspective there is no difference.
Whether a fee is sufficient is a subjective concern, it can't be computed, but it need to be ACK by the user.

You will have to do that anyway, since the size depends on the number of funding inputs, which is controlled by the peers what if the user has 10 inputs?

Of course, it's actually because we will likely need to have multiple messages that I think PSBT don't offer a big advantage.

The value "10 sat/bytes" is computed, but it does not need to be the exact same way across implementation, this is a UI concern, in practice if the fee rate is 10.01 sat/bytes instead of 9.997 sat/bytes, it should not cause the whole exchange to fail, as from user perspective there is no difference.

So first do we agree that the fee computation issue is dissociated from the use of PSBT?
Otherwise I agree that we could allow some error margin in the fee computation as I mentioned above. But that doesn't really change the fact that anybody wanting to implement the protocol will have to implement something to compute fees, and so I think that it has its place in the specifications, unless there is another specification for this that we could simply refer to (that would actually be the best).

So first do we agree that the fee computation issue is dissociated from the use of PSBT?

Yes. That's why I proposed another way to fix the fee issue in the event you don't want PSBT (https://github.com/discreetlogcontracts/dlcspecs/pull/82#issuecomment-689928845)

But that doesn't really change the fact that anybody wanting to implement the protocol will have to implement something to compute fees,

What I am trying to do is to make sure that people not calculating it the same way don't have frictions. Most Bitcoin libraries already do that out of the box, and I am quite sure none of them have same results.

NACK
I think this PR have the following issues :

free-rider behavior, "my feerate is lower than yours but my weight is higher thanks to aggregate my inputs"
protocol complexity, know you parse have to support both PSBT format AND still the format used to announce DLC non-transaction data
pinning issue, by adopting a low feerate it's easier to stuck the funding in the mempool, opening room for exploitation
future-compatibility, when we move off-chain, we will rely on channel funding transaction, and it's really likely it won't support PSBTs, so DLC wallet will have to understand TLV format
transaction malleability, not committing to the feerate (hashPrevouts, hashOutputs) and only using SIGHASH_SINGLE | SIGHASH_ANYONECANPAY means I can still your fee before broadcast

Multi-party distrusted transactions don't have the same requirement than normal transaction and thus sanitizing the feerate is a necessity, not only a matter of UX.

What I am trying to do is to make sure that people not calculating it the same way don't have frictions

The issue about calculating is a real one, but more a ecosystem-wise one and it would be better to have a BIP once for all, reusable by any protocol. Telling you to use weight, how to share common fields cost and a way to round up.

NACK-with-caveat
I think for many reasons people have listed and my opinion that PSBTs only really make any sense in manual, non-algorithmic contexts this isn't a good way to go, especially seeing as we're hoping to make DLCs entirely algorithmic without too many manual steps (especially during construction).
That said, I would like to re-iterate that if in the future we come across a use case where PSBT makes sense (say there's a lot of tech-savy OTC traders who want more freedom and are doing validation somewhat manually), then we can always introduce a new TLV type for an offer that just wraps a PSBT or alternatively a new contract flags byte could signal that you want to do the PSBT version of contract negotiation instead of the algorithmic one.
Either way I don't think the deleted lines in this PR should be deleted but I'm open to the idea that someday a use case is found for PSBTs and a new version/contract flag is deployed to add support for this.
That said, I definitely think this is out-of-scope for v0.1

if you don't want to use psbt for some reason, at least you should let feerate_per_vb go and replace it by contributed_fee in both offer and accept message. I can make another PR if you prefer than route.

In response to this comment, you should know that the plan for fee computation (which I am beginning to work on) is to be similar to the dual funding proposal where users give max witness size when communicating their funding inputs and later when inputs are added during signing messages, these max's are verified. Fee computation then becomes quite simple in a manner similar to what you are suggesting where rather than doing actual size computations one just adds up all the max sizes and uses that as the computed size to be used in fee calculation.

@nkohen awesome, I think this field would solve the main pain, closing this.


Add GPT visualizations

[x] rerun code to generate umap/tsne coordinates for GPT

[x] 2d
[ ] 3d

[x] recompute other info (e.g., attention)
[ ] add to attention-viz

[ ] should there be option to switch between gpt & bert?

if yes, should we pre-compute everything, or just load when user decides to switch between them?


Enable fieldmap jacobians by default only for PEPolar fieldmaps

With 9 months of experience behind us, IMO --ignore fmap-jacobian should always be used if the fieldmap is not PEPolar, but it is critical for at least some PEPolar cases.
I think we should disable it by default, which means we'll need to add a flag to reenable it. We could either do --fmap-jacobian, --force-fmap-jacobian or add a --force similar to --ignore, and permit fmap-jacobian as a value.
I think we should either go with --force fmap-jacobian or deprecate --ignore fmap-jacobian and use --[no-]fmap-jacobian.
If we go with --force, we could transition some other flags away from the --[no-]<whatever> form.

This might be a dumb question, but isn't the Jacobian weighting meant to deal with signal pileup in the BOLD image? Would the field map type affect whether or not there is signal pileup due to distortion?

Yes, that's the idea, which is why I enabled it by default. In practice, it doesn't seem to work for all types, possibly because the empirical warps and the theoretical warps aren't matched closely enough.

Sorry to chime in, but was a least-squares restoration option considered? It's my impression that least-squares restoration rather than jacobian modulation is the default in standard fsl for PEPolar/topup (based on https://web.mit.edu/fsl_v5.0.10/fsl/doc/wiki/topup(2f)ApplytopupFurtherInformation.html)

That one only theoretically works for PEPolar fieldmaps. I don't see an obvious way to extend it to situations where we do not have two directions to work with. As to using it for PEPolar, it's an additional complication (you need to pull through the raw fieldmap inputs instead of just the preprocessed fieldmap), and the Jacobian approach seems to work pretty well already.


problems encountered when running on the Win10 environment

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
bin E:\soft_ware\anaconda3\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so
False
E:\soft_ware\anaconda3\lib\site-packages\bitsandbytes\cuda_setup\main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/soft_ware/Git/etc/
config.site'), WindowsPath('E')}
warn(msg)
E:\soft_ware\anaconda3\lib\site-packages\bitsandbytes\cuda_setup\main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('E')}
warn(msg)
E:\soft_ware\anaconda3\lib\site-packages\bitsandbytes\cuda_setup\main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/Users/feeyo/AppDat
a/Local/Temp'), WindowsPath('C')}
warn(msg)
E:\soft_ware\anaconda3\lib\site-packages\bitsandbytes\cuda_setup\main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('E'), WindowsPath('/
soft_ware/Git/mingw64/include')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
E:\soft_ware\anaconda3\lib\site-packages\bitsandbytes\cuda_setup\main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib
64')}
warn(msg)
CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed?
E:\soft_ware\anaconda3\lib\site-packages\bitsandbytes\cuda_setup\main.py:149: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
warn(msg)
E:\soft_ware\anaconda3\lib\site-packages\bitsandbytes\cuda_setup\main.py:149: UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library...
warn(msg)
CUDA SETUP: Loading binary E:\soft_ware\anaconda3\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA library was not detected.
CUDA SETUP: Solution 1): Your paths are probably not up-to-date. You can update them via: sudo ldconfig.
CUDA SETUP: Solution 2): If you do not have sudo rights, you can do the following:
CUDA SETUP: Solution 2a): Find the cuda library via: find / -name libcuda.so 2>/dev/null
CUDA SETUP: Solution 2b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_2a
CUDA SETUP: Solution 2c): For a permanent solution add the export from 2b into your .bashrc file, located at ~/.bashrc
Traceback (most recent call last):
File "E:\soft_ware\anaconda3\lib\site-packages\transformers\utils\import_utils.py", line 1086, in get_module
return importlib.import_module("." + module_name, self.name)
File "E:\soft_ware\anaconda3\lib\importlib_init.py", line 126, in import_module
return _bootstrap.gcd_import(name[level:], package, level)
File "", line 1050, in gcd_import
File "", line 1027, in find_and_load
File "", line 1006, in find_and_load_unlocked
File "", line 688, in load_unlocked
File "", line 883, in exec_module
File "", line 241, in call_with_frames_removed
File "E:\soft_ware\anaconda3\lib\site-packages\transformers\training_args_seq2seq.py", line 21, in
from .training_args import TrainingArguments
File "E:\soft_ware\anaconda3\lib\site-packages\transformers\training_args.py", line 67, in
from accelerate.state import AcceleratorState, PartialState
File "E:\soft_ware\anaconda3\lib\site-packages\accelerate_init.py", line 3, in
from .accelerator import Accelerator
File "E:\soft_ware\anaconda3\lib\site-packages\accelerate\accelerator.py", line 35, in
from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
File "E:\soft_ware\anaconda3\lib\site-packages\accelerate\checkpointing.py", line 24, in
from .utils import (
File "E:\soft_ware\anaconda3\lib\site-packages\accelerate\utils_init.py", line 131, in
from .bnb import has_4bit_bnb_layers, load_and_quantize_model
File "E:\soft_ware\anaconda3\lib\site-packages\accelerate\utils\bnb.py", line 42, in
import bitsandbytes as bnb
File "E:\soft_ware\anaconda3\lib\site-packages\bitsandbytes_init.py", line 6, in
from . import cuda_setup, utils, research
File "E:\soft_ware\anaconda3\lib\site-packages\bitsandbytes\research_init.py", line 1, in
from . import nn
File "E:\soft_ware\anaconda3\lib\site-packages\bitsandbytes\research\nn_init.py", line 1, in
from .modules import LinearFP8Mixed, LinearFP8Global
File "E:\soft_ware\anaconda3\lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in
from bitsandbytes.optim import GlobalOptimManager
File "E:\soft_ware\anaconda3\lib\site-packages\bitsandbytes\optim_init.py", line 6, in
from bitsandbytes.cextension import COMPILED_WITH_CUDA
File "E:\soft_ware\anaconda3\lib\site-packages\bitsandbytes\cextension.py", line 20, in
raise RuntimeError('''
RuntimeError:
CUDA Setup failed despite GPU being available. Please run the following command to get more information:
    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "E:\llm\ChatGLM2-6B-main\ChatGLM-Efficient-Tuning-main\src\train_bash.py", line 1, in
from pet import get_train_args, run_sft, run_rm, run_ppo
File "E:\llm\ChatGLM2-6B-main\ChatGLM-Efficient-Tuning-main\src\pet_init_.py", line 1, in
from .core import get_train_args, get_infer_args, load_model_and_tokenizer
File "E:\llm\ChatGLM2-6B-main\ChatGLM-Efficient-Tuning-main\src\pet\core_init_.py", line 1, in
from .parse import get_train_args, get_infer_args
File "E:\llm\ChatGLM2-6B-main\ChatGLM-Efficient-Tuning-main\src\pet\core\parse.py", line 7, in
from transformers import HfArgumentParser, Seq2SeqTrainingArguments
File "", line 1075, in _handle_fromlist
File "E:\soft_ware\anaconda3\lib\site-packages\transformers\utils\import_utils.py", line 1076, in getattr
module = self._get_module(self._class_to_module[name])
File "E:\soft_ware\anaconda3\lib\site-packages\transformers\utils\import_utils.py", line 1088, in _get_module
raise RuntimeError(
RuntimeError: Failed to import transformers.training_args_seq2seq because of the following error (look up to see its traceback):
    CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

Bitsandbytes was not supported windows before, but my method can support windows.（yuhuang）
1  open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD
or WIN+R, CMD  。enter，cd /d J:\StableDiffusion\sdwebui
2  J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes
3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows
4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl
Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）


Lower some log levels from debug to  trace during plan execution

Which issue does this PR close?
Closes https://github.com/apache/arrow-datafusion/issues/6192
Rationale for this change
I was getting inundated with logging messages with debug . I wanted to see plans, not detailed tracing information about the execution
What changes are included in this PR?
Change several debug! calls to trace! calls during physical plan execution
Are these changes tested?
No -- I am not sure it makes sense to test logging changes
Are there any user-facing changes?
To see some messages now RUST_LOG=trace is required (previously RUST_LOG=debug showed them)

cc @NGA-TRAN  as I think you added many of these messages in #2231


Does this page applies for Online or Desktop - Power BI?

Hi,
I have never used Power Query Editor Online, this doc and the screenshots in it are not pointing to desktop or online.

Does this page applies to Desktop or Online?
How do we access power query online? Is it only for any paid services only?

Better start the page like...
"Power BI Desktop - Diagram view offers a visual way to prepare data in the Power Query editor - Desktop."
Instead of just...
"Diagram view offers a visual way to prepare data in the Power Query editor"
regards,
Singaravelu S
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 5cb61f83-9d14-2e3a-a2f3-6bd9b5a6f119
Version Independent ID: 28157cb9-851d-3127-cdc9-00cd5aa7582f
Content: Diagram view (Preview)
Content Source: powerquery-docs/diagram-view.md
Service: powerquery
GitHub Login: @ptyx507
Microsoft Alias: v-miesco

Thanks for your feedback.

This page applies to Power Query Online (and not Desktop).
You can access this feature (Diagram View) within Power BI dataflows or Power Apps dataflows. For example: With respect to Power BI, it is available within Power BI pro and Power BI premium. With respect to Power Apps, its available within the per app or per user plan.

We wanted the documentation to be generic because Power Query Online is available within several products such as Power BI, Power Apps and more. Hope this helps!


feat(parse): add parse pkg

:heavy_check_mark: Checklist:

[ ] Have only one commit (if not, squash them into one commit)
[ ] Tests for the changes have been added (for features / bug fixes / refactoring)

Codecov Report

Merging #259 (2fc69ed) into main (42baad2) will decrease coverage by 11.38%.
The diff coverage is 36.00%.

@@             Coverage Diff             @@
##             main     #259       +/-   ##
===========================================
- Coverage   74.57%   63.18%   -11.39%
===========================================
  Files          16       16
  Lines         582      584        +2
===========================================
- Hits          434      369       -65
- Misses        114      199       +85
+ Partials       34       16       -18

Impacted Files
Coverage Δ

internal/api/api.go
73.33% <ø> (ø)

internal/openapi3/build.go
6.89% <26.31%> (-47.35%)
:arrow_down:

internal/parse/parse.go
66.66% <66.66%> (ø)

internal/openapi3/example.go
89.47% <0.00%> (-10.53%)
:arrow_down:

internal/openapi3/openapi3.go
90.00% <0.00%> (-10.00%)
:arrow_down:

internal/faker/person.go
95.65% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4269af9...2fc69ed. Read the comment docs.


Animation not working with nested layout

Animation is not working when using the nesting of the layouts in the ShoppingKeyFrameAnimation

Hope to fix it

I have fixed the bug just did not get the time to change the UI and other stuff will update it soon 👍

Fixed the animation not working inside the card layout (nested layout)


Risk explanation is given as "increased" even with a low risk level

Avoid duplicates

[x] Bug is not mentioned in the FAQ
[x] Bug is specific for Android only, for general issues / questions that apply to iOS and Android please raise them in the documentation repository
[x] Bug is not already reported in another issue

Describe the bug
On the "Your Risk Status" screen, a green card with "Low Risk" is shown at the top. However, the last paragraph in the bottom "Risk of Infection" card reads (emphasis added):

Therefore, your risk of infection has been ranked as increased. Your risk of infection is calculated from the exposure logging data (duration and proximity) locally on your device. Your risk of infection cannot be seen by, or passed on to, anyone else. When you get home, please also avoid close contact with members of your family or household.

This seems to contradict the actual "low risk" rating and is thus confusing.
The Android app seems to display the risk_details_information_body_notice string, the wording of which seems to be specific to an increased risk, irrespective of the actual risk rating.
The iOS App stores this text under the more specific ExposureDetection_Explanation_Text_High key which only seems to be displayed when the risk rating is actually high.
Expected behaviour
When the top "Your Risk Status" section shows "Low Risk", the "Risk of Infection" section should not mention an "increased" risk, nor recommend avoiding contact with household members.
When the "Risk of Infection" section describes an "increased" risk or recommends avoiding contact with household members, the top "Your Risk Status" section should show "Increased Risk".
Steps to reproduce the issue
Given that you have no recorded risk contacts,

Open the app.
Click on "Low Risk".
Observe the contrast in verbiage between the topmost "Your Risk Status" card (with a green background) and the last paragraph in the bottommost "Risk of Infection" card (dark grey background).

Technical details

Mobile device: Nokia 6.1
Android version: 10
App version: 1.2.0

Possible Fix
Show different risk_details_information_body_notices depending on the actual risk level.
Additional context
n/a

This seems to be a regression in 1.2.0.
The previous version, 1.1.1, has a different text, which makes more sense to me:
https://github.com/corona-warn-app/cwa-app-android/blob/1.1.1/Corona-Warn-App/src/main/res/values/strings.xml shows
"Your risk of infection is calculated from the exposure logging data (duration and proximity) locally on your device. Your risk of infection cannot be seen by, or passed on to, anyone else."

Yes, I can confirm that this is a 1.2.0 regression.
This is a photo of the 1.1.1 version of the text:

And here is 1.2.0:

This problem seems to also affect iOS: https://twitter.com/gentlemanrajue/status/1293298429234499584
Maybe this issue should be moved to the documentation repo :)


Add naming options for devices in rack units display

Hello, Jeremy!
Are there any plans to add ability to select different device naming options on the rack layout display (on the Racks -> Rack Elevations -> ) page?
This is useful when exporting rack layout to a 3rd party like remote hands as they are more familiar not with the devices hostnames, but rather their models.
I propose a name in the rack unit to be a list of options to be chosen by user like following:

hostname (like now);
hostname + model;
just model.

Thanks!

If you hover over the device, you should see a callout with its device type and role. It wouldn't be feasible to squeeze both the hostname and the make/model into the U view unless we make the entire elevation much larger. We don't currently have a mechanism for recording individual user preferences.
Somewhat related, #494 was opened a while back to show the asset tag instead of the model for unnamed devices.

Ok, thanks, I've got the idea from #494.
Based on your answer I have one more proposal - if the model doesn't fit in the U description, will it be feasible to add a checkbox on a rack page like "show models" and it will pop-up the exact the same bubbles for each U as they're shown upon mouse-hover on the U, but for each unit? Also, this even can happen upon mouse-hover on the "Front" or "Rear" words on top of the rack.
Thanks!

The current logic is to show the device's name if it has one; otherwise, the device's role is displayed. We used to fall back to displaying the make and model, but in many cases it didn't fit well into the limited space for 1U devices. I don't see much benefit in implementing more complex logic since the make and model are readily available by hovering the mouse over a device.

Hello, Jeremy!
The only benefit in my proposal would be for guys, using "remote hands" service - where 3rd party company, unaware of your naming and models (sometimes,
firewall, router and even server looks for them the same) makes physical movements in your rack and the amount of info they have directly influences the result - if your infrastructure is ruined by them or they do something useful.
That's why I find this useful when you can "screen shot" your rack with different naming or visualization options to provide this "remote hands" more info.
I've found one example from the openDCIM - they have an option to display a device's image in the Rack Unit and a button to switch between name and image, maybe that will be suitable for implement in the netbox.
ON the screen shot, labels are disabled and only device pictures show. Picture error is a typical not well-prepared demo :-)


Changes: Make a dedicated Breaking Changes section

Coverage remained the same at 98.062% when pulling a0248837f9854b858bcc99d70c4d0e38e9fd9052 on changes-breaking-changes-section into a340295319d56893d3b5c3abdc3b9ecacac597f7 on releases/v0.14.0.

Coverage remained the same at 98.062% when pulling 36953bdc24366337f26bded8ce7c582b1f210e42 on changes-breaking-changes-section into a340295319d56893d3b5c3abdc3b9ecacac597f7 on releases/v0.14.0.


Valuation.map_sum_eq_of_lt

The valuation of a sum is equal to one of its summand if the valuation of the summand si greater than the valuation of all the other summands.
This is the moral combination of Valuation.map_add_eq_of_lt_left and Valuation.map_sum_lt.

Thanks!
bors merge


OOM error and memory usage question

I was trying to use vLLM on a finetined LLaMA 65B model. At first the model is a complete fp32 bin file, i.e., pytorch_model.bin (more than 200GB), and I found this will cause OOM error even the available memory was more than 1TB. This was be fixed after I split the big file info many small shards, e.g., 00001-00100.bin.  So, do we have to use the sliced version when loading super large models?
Another observation is that, I execute free -h from time to time when loading the model, and the memory usage keeps going up and down, although the general trend is up. Is it a normal phenomenon?

I aslo meet torch.cuda.OutOfMemoryError: CUDA out of memory , when i run this python :
from vllm import LLM, SamplingParams
import os

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="/home/ubuntu/wangjibo/models/facebook_opt-350m")
outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Error message like this:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 424.00 MiB (GPU 0; 23.69 GiB total capacity; 7.25
GiB already allocated; 405.31 MiB free; 7.30 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory
 try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and
PYTORCH_CUDA_ALLOC_CONF

Watch GPU memory usage: watch -n 2 nvidia-smi
Increments incrementally until OOM

If i miss some params or it's a vllm bug?


WebSocketPeer.connect_to_url() always returns 0 i.e, OK

Godot version
master - v4.2.dev.custom_build [4df80b0e6]
System information
Windows 11 Pro 22H2 - Vulkan API 1.3.250 - Forward+ - Using Vulkan Device #0: AMD - Radeon RX 570 Series
Issue description
WebSocketPeer.connect_to_url() always returns 0 i.e, OK regardless of.
Just try:
extends Node

var socket = WebSocketPeer.new()

func _ready():
	print(socket.connect_to_url("abc"))

in any script of any godot project. It will always return 0 i.e, OK.
Steps to reproduce

Run any project having a script that calls the function WebSocketPeer.connect_to_url() and check its return value.
It will print 0 in all cases(bad url, bad connection, non-existent WebSocket server). No errors or warnings are seen in debugger.

Minimal reproduction project
N/A.
Can be reproduced in any project with 2 lines:
var socket = WebSocketPeer.new()
print(socket.connect_to_url("abc"))

cc @Faless

FYI: abc is a valid URL. Because it's a valid hostname and WebSocketPeer defaults to ws:// when the scheme is missing.
Example invalid URLs:

ws:///
ws://abc.com:-42/

@timothyqiu Good point. I tried those 2 and they give 31 i.e, ERR_INVALID_PARAMETER. But that's just URL validation. Not sure but string validations usually throw(just implicitly log/print them in case of Godot) errors instead of returning as it is a pre-condition failure, not result of an function/action.
Maybe I am overthinking this.
Also, shouldn't there be some kind of connection failure errors? Getting OK as long as URL is valid might lead to hard to catch bugs in case URL isn't accessible or has a typo. We should add a note in docs at least.
Maybe a timeout after no response for some time makes sense?
Old way of websocketpeer.connection_established.connect(_connected) makes sense now.

The returned Error is for those who want to make better error report on the UI for user provided URL I think.
Apart from string validation, TLS validation is also done when calling this function since it doesn't take a long time.
Note connection_established is a signal of WebSocketClient, not WebSocketPeer. You can check for various conditions with get_ready_state() after each poll(). But detailed error code/message is still not available, like in 3.x. WebSocketClient is removed in 4.0, but you can use this demo as a reference.

WebSocketPeer methods are not blocking, as explained by @timothyqiu the URL abc is valid, so the method starts the resolution of it but doesn't wait for it to be resolved, nor it blocks until TCP connect_to_host succeed/fails.
For these reasons, the connect_to_host will most likely then not return OK (unless, the URL is invalid, the peer is already in use, etc).
As mentioned, you should check the value of get_ready_state() to check for state changes.

I am facing an issue where the get_ready_state() does not update when the network cable connected to the ip the websocket is connected to. I use poll() at the beginning in _process().

Affects me too.
Please stop not fixing it and fix it already.

Ok, but the method poll is broken and get_ready_state is getting stuck at 1 even after the socket disconnects


Fix thinInstanceCount setter to work with mesh clone

After cloning a mesh that has thin instances, setting thinInstanceCount on the clone currently doesn't work (unless it is set to 0 the setter will return without updating the internal instanceCount).
With this PR, if a mesh doesn't have the thin instance matrixData, then it checks to see if the mesh's source has it before defaulting the max instance count to 0.
PG Repro: https://playground.babylonjs.com/#217750#76
Forum: https://forum.babylonjs.com/t/cant-set-thininstancecount-for-clone/29674

Hmm, the format check is failing but the output doesn't say what the issue is, just that it failed... And I can't see what the formatting issue would be?

This is a breaking change, as now the cloned mesh will be displayed with the same thin instances than the source mesh.
But it does not seem unreasonable to me to expect this behaviour when cloning a mesh with thin instances. And if we don't want the thin instances in the cloned mesh, we can call makeGeometryUnique on the mesh.
Let's see what the Babylon.js team think about this.

This is a breaking change, as now the cloned mesh will be displayed with the same thin instances than the source mesh.

It won't be thou unless you explicitly set thinInstanceCount to the number of thin instances that you want displayed. Otherwise the behavior should be the same as it is now AFAIK.
Because the only change after the PR should be that E.G. when you set it to 3, you get the first 3 thin instances displayed as expected, if you set it to 2 you get just the first 2 as expected, etc. Currently nothing would happen at all (the setter just returns), which seems pretty unexpected IMO...


Moving policy with same name from one watch to other fails

Describe the bug
I updated my configuration to move one policy to a new watch.
Terraform apply seemed to destroy it, but when the api tries to generate it it fails with a 409 error:
module.watch-all-repos-xxx.xray_security_policy.critical_cvss: Destroying... [id=security-policy-critical-cvss]
module.watch-critical-all-repos-xxx.xray_security_policy.critical_cvss: Creating...
module.watch-all-repos-xxx.xray_security_policy.high_severity: Modifying... [id=security-policy-high-severity]
module.watch-all-repos-xxx.xray_security_policy.critical_cvss: Destruction complete after 0s
module.watch-all-repos-xxx.xray_security_policy.high_severity: Modifications complete after 0s [id=security-policy-high-severity]
module.watch-all-repos-xxx.xray_watch.all-repos: Modifying... [id=watch-all-repos-xxx]
module.watch-all-repos-xxx.xray_watch.all-repos: Modifications complete after 1s [id=watch-all-repos-xxx]
╷
│ Error:
│ 409 POST https://artifactory-staging.xxx.net/xray/api/v2/policies
│ {"error":"Policy already exists"}
│
│   with module.watch-critical-all-repos-xxx.xray_security_policy.critical_cvss,
│   on .terraform/modules/watch-critical-all-repos-xxx/ops/terraform/modules/xray-security/watch-critical/main.tf line 10, in resource "xray_security_policy" "critical_cvss":
│   10: resource "xray_security_policy" "critical_cvss" {
│
╵

The policy security-policy-critical-cvss used to exist in the watch
I'm using Terraform modules to save the configuration into the watch watch-all-repos-xxx. I was moving it to a new one named watch-critical-all-repos-xxx
My CI first ran terraform plan and then terraform apply. During plan it is stated that will be destroyed:
module.watch-all-repos-xxx.xray_security_policy.critical_cvss will be destroyed
resource "xray_security_policy" "critical_cvss" {
...
Plan: 2 to add, 4 to change, 1 to destroy.

But, as stated above it fails during apply.
I'm using version 1.11.1
Former configuration:
terraform {
  required_providers {
    xray = {
      source  = "registry.terraform.io/jfrog/xray"
      version = ">= 1.11.1, <= 1.12"
    }
  }
}
resource "xray_security_policy" "high_severity" {
  name        = "security-policy-high-severity"
  description = "Security policy to alert on High severity vulnerabilities"
  type        = "security"

  rule {
    name     = "rule-high-severity"
    priority = 1

    criteria {
      min_severity          = "High"
    }

    actions {
      webhooks                           = []
      mails                              = []
      notify_watch_recipients            = true

      block_download {
        unscanned = false
        active    = false
      }
    }
  }
}

resource "xray_security_policy" "critical_cvss" {
  name        = "security-policy-critical-cvss"
  description = "Security policy to flag artifacts with Critical CVSS score 10.0 vulnerabilities"
  type        = "security"

  rule {
    name     = "cvss-ten"
    priority = 1

    criteria {

      cvss_range {
        from = 10.0
        to   = 10.0
      }
    }

    actions {
      webhooks                           = []
      mails                              = []
      notify_watch_recipients            = true

      block_download {
        unscanned = false
        active    = false
      }
    }
  }
}

resource "xray_watch" "all-repos" {
  name        = "watch-critical-all-repos-lyft"
  description = "Watch policy violations for top critical vulnerabilities in all repositories"
  active      = true

  watch_resource {
    type = "all-repos"

    filter {
      type  = "regex"
      value = ".*"
    }
  }

  assigned_policy {
    name = xray_security_policy.high_severity.name
    type = "security"
  }

  assigned_policy {
    name = xray_security_policy.critical_cvss.name
    type = "security"
  }

  watch_recipients =<EMAIL_ADDRESS>}

New configuration:
terraform {
  required_providers {
    xray = {
      source  = "registry.terraform.io/jfrog/xray"
      version = ">= 1.11.1, <= 1.12"
    }
  }
}
resource "xray_security_policy" "high_severity" {
  name        = "security-policy-high-severity"
  description = "Security policy to alert on High severity vulnerabilities"
  type        = "security"

  rule {
    name     = "rule-high-severity"
    priority = 1

    criteria {
      min_severity          = "High"
    }

    actions {
      webhooks                           = []
      mails                              = []
      block_release_bundle_distribution  = false
      fail_build                         = false
      notify_watch_recipients            = false

      block_download {
        unscanned = false
        active    = false
      }
    }
  }
}

resource "xray_watch" "all-repos" {
  name        = "watch-critical-all-repos-lyft"
  description = "Watch policy violations for top critical vulnerabilities in all repositories"
  active      = true

  watch_resource {
    type = "all-repos"

    filter {
      type  = "regex"
      value = ".*"
    }
  }

  assigned_policy {
    name = xray_security_policy.high_severity.name
    type = "security"
  }

  watch_recipients =<EMAIL_ADDRESS>}

Artifactory version: 7.41.7
Xray version: 3.41.5
Terraform: 1.0.2
Requirements for and issue

[x] A fully functioning terraform snippet that can be copy&pasted (no outside files or ENV vars unless that's part of the issue)
[x] Your version of Artifactory and Xray (you can curl Artifactory version at $host/artifactory/api/system/version and Xray version at
$host/xray/api/v1/system/version
[x] Your version of terraform

Expected behavior
The policy from the original watch should be destroyed and a new one with the same name be created into the new watch.
Additional context
We use project atlantis to run Terraform as part of our pipeline.

@heryxpc If I read your message correctly, you are moving the policy resource from one module watch-all-repos-xxx to another watch-critical-all-repos-xxx?
Unless you use the Terraform moved block, TF won't know that this is a move operation and will try to do exactly what you described. Which is, delete the policy resource in watch-all-repos-xxx and create the same policy as new resource in watch-critical-all-repos-xxx. However, since TF doesn't know they are the same resource, it will try to perform both operation at the same time in parallel. This is, I think, the cause of the 409 error.


Mock screen width in decode tests

Running the tests locally with curses installed failed if terminal
wasn't 80 columns wide. Since the tests expect line breaks at 80,
we make sure that curses return that expected width.

Coverage remained the same at 96.464% when pulling 9d6e13501bf53ba08c0d0b8a083c343380564784 on simontegelid:mock_screen_width into 871581b57785fdbd79c118878f9b9c148984963c on eerimoq:master.


Commit dataclass refactor

Dawes - When My Time Comes [https://youtu.be/rQEj_9IhJSo]
Description

dataclass.py overrides loaders.py
Handles diff output states, state input concatenation, train/test split, per-episode data grouping, and sequential prediction

Type of change

[x] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] This change requires a documentation update

How Has This Been Tested?

[x] tests/test_dataclass.py
[x] tests/test_gboost.py
[x] tests/test_gboost_concat.py
[x] tests/test_gboost_diff.py
[x] notebooks/ddm_evaluation.ipynb

Reviewer Tasks
List here all the tasks the reviewer needs to review:

[ ] Test ddm_evaluation.ipynb

Test Configuration:

OS: Windows
Python Version: 3.7.7
Bonsai SDK Version: NA

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] Any dependent changes have been merged and published in downstream modules

Hitting an error when I try using concatenated steps
  concatenated_steps: 4
  concatenated_zero_padding: True

error:
~/miniconda3/envs/ddm2/lib/python3.7/site-packages/lightgbm/basic.py in _safe_call(ret)
     53     """
     54     if ret != 0:
---> 55         raise LightGBMError(decode_string(_LIB.LGBM_GetLastError()))
     56
     57

LightGBMError: Check failed: (best_split_info.left_count) > (0) at /Users/runner/work/1/s/python-package/compile/src/treelearner/serial_tree_learner.cpp, line 651 .

that is with lightgbm instead of xgboost.

Here's the config load for hyda:
initialize(config_path="../conf", job_name="model_validation")
cfg = compose(config_name="config", overrides=["data=cartpole_st_at", "model=lightgbm"])

Could you test if the sweeping still works? Having some issues getting it to run when having the concatenate states turned on:
(ddm2) ➜  datadrivenmodel git:(juan/dataclass_preprocessing) ✗ python ddm_trainer.py data=cartpole_st_at model=xgboost model.sweep.run=True
/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/hydra/plugins/config_source.py:190: UserWarning:
Missing @package directive data/cartpole_st_at.yaml in file:///Users/alizaidi-msft/Documents/bonsai/datadrivenmodel/conf.
See https://hydra.cc/docs/next/upgrades/0.11_to_1.0/adding_a_package_directive
  warnings.warn(message=msg, category=UserWarning)
/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/hydra/plugins/config_source.py:190: UserWarning:
Missing @package directive model/xgboost.yaml in file:///Users/alizaidi-msft/Documents/bonsai/datadrivenmodel/conf.
See https://hydra.cc/docs/next/upgrades/0.11_to_1.0/adding_a_package_directive
  warnings.warn(message=msg, category=UserWarning)
/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/hydra/plugins/config_source.py:190: UserWarning:
Missing @package directive simulator/cartpole_simparam.yaml in file:///Users/alizaidi-msft/Documents/bonsai/datadrivenmodel/conf.
See https://hydra.cc/docs/next/upgrades/0.11_to_1.0/adding_a_package_directive
  warnings.warn(message=msg, category=UserWarning)
[2021-06-03 10:54:02,133][datamodeler][INFO] - Configuration:
[2021-06-03 10:54:02,141][datamodeler][INFO] -
data:
  path: csv_data/cartpole_at_st.csv
  full_or_relative: relative
  inputs: state
  outputs: state
  augmented_cols:
  - config_length
  - config_masspole
  - action_command
  iteration_order: 1
  episode_col: episode
  iteration_col: iteration
  max_rows: -1
  scale_data: true
  diff_state: false
  concatenated_steps: 4
  concatenated_zero_padding: true
  test_perc: 0.25
model:
  name: xgboost
  build_params:
    model_type: xgboost
    fit_separate: false
    halt_model: false
    n_estimators: 100
    objective: reg:squarederror
    learning_rate: 0.3
    scale_data: true
    max_depth: 6
  saver:
    filename: models/boost/moab/xgboost_model
  sweep:
    run: true
    split_strategy: timeseries
    search_algorithm: bayesian
    num_trials: 25
    scoring_func: r2
    early_stopping: false
    results_csv_path: xgboost_gridsearch/search_results.csv
    params:
      estimator__max_depth:
      - 1
      - 3
      - 5
      - 10
      estimator__gamma:
      - 0
      - 0.5
      - 1
      - 5
      estimator__subsample:
      - 0.1
      - 0.5
      - 1
      estimator__eta:
      - 0.3
      - 0.1
      - 0.5
simulator:
  states:
  - state_x_position
  - state_x_velocity
  - state_angle_position
  - state_angle_velocity
  actions:
  - action_command
  configs:
  - config_length
  - config_masspole
  policy: bonsai
  logging: enable

[2021-06-03 10:54:02,158][root][INFO] - Using ['state_x_position', 'state_x_velocity', 'state_angle_position', 'state_angle_velocity', 'config_length', 'config_masspole', 'action_command'] as the features for modeling DDM
[2021-06-03 10:54:02,159][root][INFO] - Using ['state_x_position', 'state_x_velocity', 'state_angle_position', 'state_angle_velocity'] as the labels for modeling DDM
[2021-06-03 10:54:02,159][data_class][INFO] - Iteration order set to 1 so using outputs from next 1 row
[2021-06-03 10:54:02,169][data_class][INFO] - Next states are being added to same row with prefix next
[2021-06-03 10:54:02,169][data_class][INFO] - Feature columns are: ['next_state_x_position', 'next_state_x_velocity', 'next_state_angle_position', 'next_state_angle_velocity', 'next_config_length', 'next_config_masspole', 'next_action_command']
[2021-06-03 10:54:02,169][data_class][INFO] - Label columns are: ['state_x_position', 'state_x_velocity', 'state_angle_position', 'state_angle_velocity']
[2021-06-03 10:54:02,245][data_class][INFO] - trimmed dataframe across episodes. found (100) episodes.
[2021-06-03 10:54:02,245][data_class][INFO] - divided train & test set with (4165) and (735) iterations, respectively. Chosen split == 85.0%.
   >> Average episode length: (49.0). Average std dev: (0.0)
[2021-06-03 10:54:02,383][datamodeler][INFO] - Saving last 25.0% for test, using first 75.0% for training/sweeping
[2021-06-03 10:54:02,384][datamodeler][INFO] - Saving data to /Users/alizaidi-msft/Documents/bonsai/datadrivenmodel/outputs/2021-06-03/10-54-02/data
[2021-06-03 10:54:02,393][datamodeler][INFO] - Building model...
[2021-06-03 10:54:02,395][datamodeler][INFO] - Sweeping with parameters: {'estimator__max_depth': [1, 3, 5, 10], 'estimator__gamma': [0, 0.5, 1, 5], 'estimator__subsample': [0.1, 0.5, 1], 'estimator__eta': [0.3, 0.1, 0.5]}
2021/06/03 10:54:02 WARNING mlflow.utils.autologging_utils: You are using an unsupported version of sklearn. If you encounter errors during autologging, try upgrading / downgrading sklearn to a supported version, or try upgrading MLflow.
Traceback (most recent call last):
  File "/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/mlflow/sklearn/utils.py", line 728, in _all_estimators
    return all_estimators()
  File "/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/sklearn/utils/__init__.py", line 1145, in all_estimators
    module = import_module(modname)
  File "/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/sklearn/cluster/affinity_propagation_.py", line 7, in <module>
    from ..utils.deprecation import _raise_dep_warning_if_not_pytest
ImportError: cannot import name '_raise_dep_warning_if_not_pytest' from 'sklearn.utils.deprecation' (/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/sklearn/utils/deprecation.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
    return func()
  File "/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/hydra/_internal/utils.py", line 350, in <lambda>
    overrides=args.overrides,
  File "/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 112, in run
    configure_logging=with_log_configuration,
  File "/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/hydra/core/utils.py", line 128, in run_job
    ret.return_value = task_function(task_cfg)
  File "ddm_trainer.py", line 113, in main
    splitting_criteria=split_strategy,
  File "/Users/alizaidi-msft/Documents/bonsai/datadrivenmodel/base.py", line 734, in sweep
    mlflow.sklearn.autolog()
  File "/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/mlflow/utils/autologging_utils/__init__.py", line 468, in autolog
    return _autolog(*args, **kwargs)
  File "/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/mlflow/sklearn/__init__.py", line 943, in autolog
    _, estimators_to_patch = zip(*_all_estimators())
  File "/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/mlflow/sklearn/utils.py", line 730, in _all_estimators
    return _backported_all_estimators()
  File "/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/mlflow/sklearn/utils.py", line 772, in _backported_all_estimators
    from sklearn.utils.testing import ignore_warnings
  File "/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/sklearn/utils/testing.py", line 7, in <module>
    from ..utils.deprecation import _raise_dep_warning_if_not_pytest
ImportError: cannot import name '_raise_dep_warning_if_not_pytest' from 'sklearn.utils.deprecation' (/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/sklearn/utils/deprecation.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "ddm_trainer.py", line 126, in <module>
    main()
  File "/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/hydra/main.py", line 37, in decorated_main
    strict=strict,
  File "/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/hydra/_internal/utils.py", line 347, in _run_hydra
    lambda: hydra.run(
  File "/Users/alizaidi-msft/miniconda3/envs/ddm2/lib/python3.7/site-packages/hydra/_internal/utils.py", line 237, in run_and_report
    assert mdl is not None
AssertionError

I just pushed the changes to the repo to make "test_perc" used in dataclass, and avoiding having 2 splits made consecutively (one in dataclass and another in ddm_trainer).

The error I see with TuneGridSearch is the following:
E           TypeError: __init__() got an unexpected keyword argument 'loggers' c:\users\juanvergara\anaconda3\envs\ddm_env\lib\site-packages\tune_sklearn\tune_search.py:584: TypeError
The config used to run is:

DATA CONFIG (cartpole_st_at.yaml)

data:
path: csv_data/cartpole_at_st.csv
full_or_relative: relative
inputs: state
outputs: state
augmented_cols: ["config_length", "config_masspole", "action_command"]
iteration_order: 1
episode_col: episode
iteration_col: iteration
max_rows: -1
scale_data: True
diff_state: False
concatenated_steps: 4
concatenated_zero_padding: True
test_perc: 0.25

MODEL CONFIG:

model:
name: lightgbm
build_params:
model_type: lightgbm
fit_separate: False
halt_model: False
objective: reg:squarederror
scale_data: True
num_leaves: 10
n_estimators: 100
learning_rate: 0.3
max_depth: 6
saver:
filename: models/boost/lightgbm
sweep:
run: True
# can be any of bayesian, bohb, hyperopt, optuna, grid or random
search_algorithm: bayesian #grid
split_strategy: timeseries
num_trials: 50
scoring_func: r2
early_stopping: False
results_csv_path: lightgbm/search_results.csv
params:
# all parameters should be prefixed with estimator__
# see the list of available parameters:
# https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMModel.html
estimator__max_depth: [-1, 5] #[-1, 5, 10, 25]
estimator__learning_rate: [0.01, 0.05] #[0.01, 0.05, 0.1, 0.25]
estimator__num_leaves: [10] #[10, 31, 50]
estimator__n_estimators: [10, 100] #[10, 100, 250]
estimator__boosting_type: ["gbdt", "dart", "goss"]
estimator__reg_alpha: [0, 0.01] #[0, 0.01, 0.05, 0.1]
estimator__reg_lambda: [0, 0.01] #[0, 0.01, 0.05, 0.1]

On the side, I added sweeping tests @akzaidi  --  Had to comment the "torch" test (test_dataclass_sweeping > test_sweeping_torch), since I encountered the following error:

E       AttributeError: 'TuneSearchCV' object has no attribute 'best_estimator_'
torch_models.py:227: AttributeError

It also displayed the following 2 FAILS as well bellow that:

ERROR    ray.tune.syncer:syncer.py:63 Log sync requires rsync to be installed.
ERROR    ray.tune.trial_runner:trial_runner.py:567 Trial _Trainable_5e0e8c06: Error processing event.

Hitting this error when doing sweep.run=True for lightgbm on data.cartpole-updated.yaml. Will investigate:
[2021-07-15 17:51:45,973][datamodeler][INFO] - Configuration:
[2021-07-15 17:51:45,979][datamodeler][INFO] -
data:
  path: csv_data/cartpole-mixed-brain-rand-term.csv
  full_or_relative: relative
  inputs:
  - cart_position
  - cart_velocity
  - pole_angle
  - pole_angular_velocity
  outputs:
  - cart_position
  - cart_velocity
  - pole_angle
  - pole_angular_velocity
  augmented_cols:
  - pole_length
  - pole_mass
  - cart_mass
  - command
  iteration_order: -1
  episode_col: episode
  iteration_col: iteration
  max_rows: -1
  scale_data: true
  diff_state: false
  concatenated_steps: 2
  concatenated_zero_padding: true
  test_perc: 0.25
model:
  name: lightgbm
  build_params:
    model_type: lightgbm
    fit_separate: false
    halt_model: false
    objective: reg:squarederror
    scale_data: true
    num_leaves: 10
    n_estimators: 100
    learning_rate: 0.3
    max_depth: 6
  saver:
    filename: models/boost/lightgbm
  sweep:
    run: true
    search_algorithm: bayesian
    split_strategy: timeseries
    num_trials: 5
    scoring_func: r2
    early_stopping: false
    results_csv_path: lightgbm/search_results.csv
    params:
      estimator__max_depth:
      - -1
      - 5
      - 10
      - 25
      estimator__learning_rate:
      - 0.01
      - 0.05
      - 0.1
      - 0.25
      estimator__num_leaves:
      - 10
      - 31
      - 50
      estimator__n_estimators:
      - 10
      - 100
      - 250
      estimator__boosting_type:
      - gbdt
      - dart
      - goss
      estimator__reg_alpha:
      - 0
      - 0.01
      - 0.05
      - 0.1
      estimator__reg_lambda:
      - 0
      - 0.01
      - 0.05
      - 0.1
simulator:
  states:
  - cart_position
  - cart_velocity
  - pole_angle
  - pole_angular_velocity
  actions:
  - command
  configs:
  - pole_length
  - pole_mass
  - cart_mass
  episode_inits:
    pole_length: 0.4
    pole_mass: 0.055
    cart_mass: 0.31
  policy: bonsai
  logging: enable
  workspace_setup: true

[2021-07-15 17:51:46,946][root][INFO] - Using ['cart_position', 'cart_velocity', 'pole_angle', 'pole_angular_velocity', 'pole_length', 'pole_mass', 'cart_mass', 'command'] as the features for modeling DDM
[2021-07-15 17:51:46,946][root][INFO] - Using ['cart_position', 'cart_velocity', 'pole_angle', 'pole_angular_velocity'] as the labels for modeling DDM
[2021-07-15 17:51:46,946][data_class][INFO] - Iteration order set to -1 so using inputs from previous -1 row
[2021-07-15 17:51:46,986][data_class][INFO] - Previous states are being added to same row with prefix prev
[2021-07-15 17:51:46,987][data_class][INFO] - Feature columns are: ['prev_cart_position', 'prev_cart_velocity', 'prev_pole_angle', 'prev_pole_angular_velocity', 'prev_pole_length', 'prev_pole_mass', 'prev_cart_mass', 'prev_command']
[2021-07-15 17:51:46,987][data_class][INFO] - Label columns are: ['cart_position', 'cart_velocity', 'pole_angle', 'pole_angular_velocity']
[2021-07-15 17:51:47,256][data_class][INFO] - trimmed dataframe across episodes. found (500) episodes.
[2021-07-15 17:51:50,400][data_class][INFO] - divided train & test set with (42978) and (14339) iterations, respectively. Chosen split == 75.0%.
   >> Average episode length: (114.634). Average std dev: (97.02504853902418)
[2021-07-15 17:51:51,058][datamodeler][INFO] - From the full dataset, 25.0% will be used for test, while 75.0% for training/sweeping
[2021-07-15 17:51:51,268][datamodeler][INFO] - Saving data to /home/alizaidi/bonsai/datadrivenmodel/outputs/2021-07-15/17-51-45/data
[2021-07-15 17:51:51,283][datamodeler][INFO] - Building model...
[2021-07-15 17:51:51,287][datamodeler][INFO] - Sweeping with parameters: {'estimator__max_depth': [-1, 5, 10, 25], 'estimator__learning_rate': [0.01, 0.05, 0.1, 0.25], 'estimator__num_leaves': [10, 31, 50], 'estimator__n_estimators': [10, 100, 250], 'estimator__boosting_type': ['gbdt', 'dart', 'goss'], 'estimator__reg_alpha': [0, 0.01, 0.05, 0.1], 'estimator__reg_lambda': [0, 0.01, 0.05, 0.1]}
[2021-07-15 17:51:53,946][tune_sklearn.tune_basesearch][INFO] - TIP: Hiding process output by default. To show process output, set verbose=2.
[2021-07-15 17:51:54,671][ray.tune.trial_runner][WARNING] - Trial Runner checkpointing failed: can't pickle dict_values objects
(pid=2493) [LightGBM] [Warning] Accuracy may be bad since you didn't explicitly set num_leaves OR 2^max_depth > num_leaves. (num_leaves=31).
[2021-07-15 17:51:55,494][ray.tune.trial_runner][ERROR] - Trial _Trainable_57931d94: Error processing event.
Traceback (most recent call last):
  File "ddm_trainer.py", line 109, in main
    splitting_criteria=split_strategy,
  File "/home/alizaidi/bonsai/datadrivenmodel/base.py", line 788, in sweep
    search.fit(X, y)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/tune_sklearn/tune_basesearch.py", line 664, in fit
    result = self._fit(X, y, groups, **fit_params)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/tune_sklearn/tune_basesearch.py", line 565, in _fit
    analysis = self._tune_run(config, resources_per_trial)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/tune_sklearn/tune_search.py", line 715, in _tune_run
    analysis = tune.run(trainable, **run_args)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/ray/tune/tune.py", line 421, in run
    runner.step()
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/ray/tune/trial_runner.py", line 402, in step
    self._process_events(timeout=timeout)  # blocking
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/ray/tune/trial_runner.py", line 560, in _process_events
    self._process_trial(trial)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/ray/tune/trial_runner.py", line 586, in _process_trial
    results = self.trial_executor.fetch_result(trial)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/ray/tune/ray_trial_executor.py", line 609, in fetch_result
    result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/ray/worker.py", line 1456, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(LightGBMError): ray::_Trainable.train_buffered() (pid=2485, ip=<IP_ADDRESS>)
  File "python/ray/_raylet.pyx", line 480, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/ray/tune/trainable.py", line 167, in train_buffered
    result = self.train()
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/ray/tune/trainable.py", line 226, in train
    result = self.step()
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/tune_sklearn/_trainable.py", line 106, in step
    return self._train()
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/tune_sklearn/_trainable.py", line 246, in _train
    error_score="raise")
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/sklearn/utils/validation.py", line 72, in inner_f
    return f(**kwargs)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 248, in cross_validate
    for train, test in cv.split(X, y, groups))
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/joblib/parallel.py", line 1048, in __call__
    if self.dispatch_one_batch(iterator):
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/joblib/parallel.py", line 866, in dispatch_one_batch
    self._dispatch(tasks)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/joblib/parallel.py", line 784, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 208, in apply_async
    result = ImmediateResult(func)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 572, in __init__
    self.results = batch()
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/joblib/parallel.py", line 263, in __call__
    for func, args, kwargs in self.items]
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/joblib/parallel.py", line 263, in <listcomp>
    for func, args, kwargs in self.items]
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 531, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/sklearn/multioutput.py", line 176, in fit
    for i in range(y.shape[1]))
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/joblib/parallel.py", line 1048, in __call__
    if self.dispatch_one_batch(iterator):
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/joblib/parallel.py", line 866, in dispatch_one_batch
    self._dispatch(tasks)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/joblib/parallel.py", line 784, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 208, in apply_async
    result = ImmediateResult(func)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 572, in __init__
    self.results = batch()
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/joblib/parallel.py", line 263, in __call__
    for func, args, kwargs in self.items]
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/joblib/parallel.py", line 263, in <listcomp>
    for func, args, kwargs in self.items]
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/sklearn/multioutput.py", line 41, in _fit_estimator
    estimator.fit(X, y, **fit_params)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/lightgbm/sklearn.py", line 779, in fit
    callbacks=callbacks, init_model=init_model)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/lightgbm/sklearn.py", line 617, in fit
    callbacks=callbacks, init_model=init_model)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/lightgbm/engine.py", line 252, in train
    booster.update(fobj=fobj)
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/lightgbm/basic.py", line 2460, in update
    ctypes.byref(is_finished)))
  File "/anaconda/envs/ddm/lib/python3.7/site-packages/lightgbm/basic.py", line 55, in _safe_call
    raise LightGBMError(decode_string(_LIB.LGBM_GetLastError()))
lightgbm.basic.LightGBMError: Check failed: (best_split_info.left_count) > (0) at /__w/1/s/python-package/compile/src/treelearner/serial_tree_learner.cpp, line 651 .

Works just fine with xgboost though (even with sweeping):
[2021-07-15 17:53:10,286][datamodeler][INFO] - Configuration:
[2021-07-15 17:53:10,291][datamodeler][INFO] -
data:
  path: csv_data/cartpole-mixed-brain-rand-term.csv
  full_or_relative: relative
  inputs:
  - cart_position
  - cart_velocity
  - pole_angle
  - pole_angular_velocity
  outputs:
  - cart_position
  - cart_velocity
  - pole_angle
  - pole_angular_velocity
  augmented_cols:
  - pole_length
  - pole_mass
  - cart_mass
  - command
  iteration_order: -1
  episode_col: episode
  iteration_col: iteration
  max_rows: -1
  scale_data: true
  diff_state: false
  concatenated_steps: 2
  concatenated_zero_padding: true
  test_perc: 0.25
model:
  name: xgboost
  build_params:
    model_type: xgboost
    fit_separate: false
    halt_model: false
    n_estimators: 100
    objective: reg:squarederror
    learning_rate: 0.3
    scale_data: true
    max_depth: 6
  saver:
    filename: models/xgboost_models/mixed-cartpole
  sweep:
    run: true
    split_strategy: timeseries
    search_algorithm: bayesian
    num_trials: 2
    scoring_func: r2
    early_stopping: false
    results_csv_path: xgboost_gridsearch/search_results.csv
    params:
      estimator__max_depth:
      - 1
      - 3
      - 5
      - 10
      estimator__gamma:
      - 0
      - 0.5
      - 1
      - 5
      estimator__subsample:
      - 0.1
      - 0.5
      - 1
      estimator__eta:
      - 0.3
      - 0.1
      - 0.5
simulator:
  states:
  - cart_position
  - cart_velocity
  - pole_angle
  - pole_angular_velocity
  actions:
  - command
  configs:
  - pole_length
  - pole_mass
  - cart_mass
  episode_inits:
    pole_length: 0.4
    pole_mass: 0.055
    cart_mass: 0.31
  policy: bonsai
  logging: enable
  workspace_setup: true

[2021-07-15 17:53:11,221][root][INFO] - Using ['cart_position', 'cart_velocity', 'pole_angle', 'pole_angular_velocity', 'pole_length', 'pole_mass', 'cart_mass', 'command'] as the features for modeling DDM
[2021-07-15 17:53:11,221][root][INFO] - Using ['cart_position', 'cart_velocity', 'pole_angle', 'pole_angular_velocity'] as the labels for modeling DDM
[2021-07-15 17:53:11,221][data_class][INFO] - Iteration order set to -1 so using inputs from previous -1 row
[2021-07-15 17:53:11,267][data_class][INFO] - Previous states are being added to same row with prefix prev
[2021-07-15 17:53:11,268][data_class][INFO] - Feature columns are: ['prev_cart_position', 'prev_cart_velocity', 'prev_pole_angle', 'prev_pole_angular_velocity', 'prev_pole_length', 'prev_pole_mass', 'prev_cart_mass', 'prev_command']
[2021-07-15 17:53:11,268][data_class][INFO] - Label columns are: ['cart_position', 'cart_velocity', 'pole_angle', 'pole_angular_velocity']
[2021-07-15 17:53:11,527][data_class][INFO] - trimmed dataframe across episodes. found (500) episodes.
[2021-07-15 17:53:14,595][data_class][INFO] - divided train & test set with (42978) and (14339) iterations, respectively. Chosen split == 75.0%.
   >> Average episode length: (114.634). Average std dev: (97.02504853902418)
[2021-07-15 17:53:15,230][datamodeler][INFO] - From the full dataset, 25.0% will be used for test, while 75.0% for training/sweeping
[2021-07-15 17:53:15,441][datamodeler][INFO] - Saving data to /home/alizaidi/bonsai/datadrivenmodel/outputs/2021-07-15/17-53-10/data
[2021-07-15 17:53:15,455][datamodeler][INFO] - Building model...
[2021-07-15 17:53:15,459][datamodeler][INFO] - Sweeping with parameters: {'estimator__max_depth': [1, 3, 5, 10], 'estimator__gamma': [0, 0.5, 1, 5], 'estimator__subsample': [0.1, 0.5, 1], 'estimator__eta': [0.3, 0.1, 0.5]}
[2021-07-15 17:53:18,200][tune_sklearn.tune_basesearch][INFO] - TIP: Hiding process output by default. To show process output, set verbose=2.
[2021-07-15 17:53:18,948][ray.tune.trial_runner][WARNING] - Trial Runner checkpointing failed: can't pickle dict_values objects
(pid=3373) /anaconda/envs/ddm/lib/python3.7/site-packages/xgboost/data.py:106: UserWarning: Use subset (sliced data) of np.ndarray is not recommended because it will generate extra copies and increase memory consumption
(pid=3373)   "because it will generate extra copies and increase " +
(pid=3371) /anaconda/envs/ddm/lib/python3.7/site-packages/xgboost/data.py:106: UserWarning: Use subset (sliced data) of np.ndarray is not recommended because it will generate extra copies and increase memory consumption
(pid=3371)   "because it will generate extra copies and increase " +
(pid=3373) /anaconda/envs/ddm/lib/python3.7/site-packages/xgboost/data.py:106: UserWarning: Use subset (sliced data) of np.ndarray is not recommended because it will generate extra copies and increase memory consumption
(pid=3373)   "because it will generate extra copies and increase " +
(pid=3373) /anaconda/envs/ddm/lib/python3.7/site-packages/xgboost/data.py:106: UserWarning: Use subset (sliced data) of np.ndarray is not recommended because it will generate extra copies and increase memory consumption
(pid=3373)   "because it will generate extra copies and increase " +
(pid=3371) /anaconda/envs/ddm/lib/python3.7/site-packages/xgboost/data.py:106: UserWarning: Use subset (sliced data) of np.ndarray is not recommended because it will generate extra copies and increase memory consumption
(pid=3371)   "because it will generate extra copies and increase " +
(pid=3373) /anaconda/envs/ddm/lib/python3.7/site-packages/xgboost/data.py:106: UserWarning: Use subset (sliced data) of np.ndarray is not recommended because it will generate extra copies and increase memory consumption
(pid=3373)   "because it will generate extra copies and increase " +
(pid=3373) /anaconda/envs/ddm/lib/python3.7/site-packages/xgboost/data.py:106: UserWarning: Use subset (sliced data) of np.ndarray is not recommended because it will generate extra copies and increase memory consumption
(pid=3373)   "because it will generate extra copies and increase " +
(pid=3371) /anaconda/envs/ddm/lib/python3.7/site-packages/xgboost/data.py:106: UserWarning: Use subset (sliced data) of np.ndarray is not recommended because it will generate extra copies and increase memory consumption
(pid=3371)   "because it will generate extra copies and increase " +
[2021-07-15 17:54:39,755][ray.tune.trial_runner][WARNING] - Trial Runner checkpointing failed: can't pickle dict_values objects
(pid=3371) /anaconda/envs/ddm/lib/python3.7/site-packages/xgboost/data.py:106: UserWarning: Use subset (sliced data) of np.ndarray is not recommended because it will generate extra copies and increase memory consumption
(pid=3371)   "because it will generate extra copies and increase " +
(pid=3371) /anaconda/envs/ddm/lib/python3.7/site-packages/xgboost/data.py:106: UserWarning: Use subset (sliced data) of np.ndarray is not recommended because it will generate extra copies and increase memory consumption
(pid=3371)   "because it will generate extra copies and increase " +
[2021-07-15 17:55:41,892][ray.tune.trial_runner][WARNING] - Trial Runner checkpointing failed: can't pickle dict_values objects
[2021-07-15 17:55:41,894][ray.tune.tune][WARNING] - Trial Runner checkpointing failed: can't pickle dict_values objects
2021/07/15 17:55:41 WARNING mlflow.sklearn.utils: Truncated the value of the key `estimator`. Truncated value: `XGBRegressor(base_score=None, booster=None, colsample_bylevel=None,
             colsample_bynode=None, colsample_bytree=None, eta=0.5, gamma=0.5,
             gpu_id=None, importance_type='gain', interaction_constraints=None,
             learnin...`
/anaconda/envs/ddm/lib/python3.7/site-packages/xgboost/data.py:106: UserWarning: Use subset (sliced data) of np.ndarray is not recommended because it will generate extra copies and increase memory consumption
  "because it will generate extra copies and increase " +
[2021-07-15 17:56:05,535][base][INFO] - Saving sweeping results to xgboost_gridsearch/search_results.csv
[2021-07-15 17:56:05,538][base][INFO] - Best hyperparams: {'estimator__max_depth': 3, 'estimator__gamma': 0.5, 'estimator__subsample': 1.0, 'estimator__eta': 0.5}
[2021-07-15 17:56:05,538][base][INFO] - Best score: 0.9806443298419201
[2021-07-15 17:56:05,683][datamodeler][INFO] - Sweep results:                                               params  split0_test_score  ...  param_estimator__subsample  param_estimator__eta
0  {'estimator__max_depth': 1, 'estimator__gamma'...           0.966747  ...                           1                   0.1
1  {'estimator__max_depth': 3, 'estimator__gamma'...           0.977150  ...                           1                   0.5

[2 rows x 15 columns]
[2021-07-15 17:56:05,683][datamodeler][INFO] - Saving model to /home/alizaidi/bonsai/datadrivenmodel/models/xgboost_models/mixed-cartpole.pkl
[2021-07-15 17:56:05,683][gboost_models][INFO] - Scale transformations used, saving to /home/alizaidi/bonsai/datadrivenmodel/models/xgboost_models/mixed-cartpole.pkl