Implement LongLoRA trick for efficient tuning of long-context models

Feature request
The authors of LongLoRA explore a trick you can toggle on during training and toggle off during inference. The key takeaways are:

LoRA perplexity deteriorates as context length increases, LongLoRA solves this and has the same perplexity as FP16 on longer contexts
Able to train with longer contexts much faster than LoRA while keeping the same VRAM usage as LoRA

Paper: https://arxiv.org/pdf/2309.12307.pdf
Code: https://github.com/dvlab-research/LongLoRA
Trick is implemented here: https://github.com/dvlab-research/LongLoRA/blob/4cbb61b93ac8c7ea4db7ffc24a824f3fb153554d/llama_attn_replace.py
Motivation
The key motivation is to enable a LoRA method that scales better for fine-tuning models with longer context sizes. We know this to be expensive both in compute hours and memory requirements, but LongLoRA seems to optimize for this. Using LongLoRA, we are able to train models faster and keep the perplexity of FP16 on longer context sizes.
Your contribution
The trick is implemented in the function below. It is a 2-step process.
Shift short attention:
def shift(qkv, bsz, q_len, group_size, num_heads, head_dim):
    qkv[:, num_heads // 2:] = qkv[:, num_heads // 2:].roll(-group_size // 2, dims=2)
    qkv = qkv.transpose(1, 2).reshape(bsz * (q_len // group_size), group_size, num_heads, head_dim).transpose(1, 2)
    return qkv

Unpacking after attention computations:
output[:, :, self.num_heads//2:] = output[:, :, self.num_heads//2:].roll(group_size//2, dims=1)

This looks very promising, especially if it really just requires a couple of lines of extra code. Thanks for bringing this to our attention. Do you want to work on adding this?

@BenjaminBossan I am currently occupied with a lot of other work, and I do not know where to start with implementing this in PEFT, so I would hope one of the great HF MLEs can pick this up🙏

Haha, let's see if flattery does the trick for you ;-)
I was mainly asking because it was not clear to me from your post if you wanted to add it or just suggested it to be added, not to push you.

Haha, maybe it will :)
I mainly posted here because I saw it could have a large benefit for all users looking to train LLMs with QLoRA and long contexts

I'd like to add this feature if one of the great HF MLEs hasn't picked it up yet ;)

We haven't started on it yet, so feel free to take a shot. Don't hesitate to ask questions or to create a draft PR early, so that we can give quick feedback.

I read some code and have a few thoughts:

Should there be a boolean argument to LoraConfig() in peft/tuners/lora/config.py to specify if LongLora is to be used
The authors of LongLoRA implemented this for Llama by updating the forward pass in transformers.models.llama.modeling_llama.LlamaAttention then passing that model to LoraConfig. It almost feels like a transformers project rather than peft

So does supporting this feature require custom LongLoRA attention implementations for each model?

2. The authors of LongLoRA implemented this for Llama by updating the forward pass in transformers.models.llama.modeling_llama.LlamaAttention then passing that model to LoraConfig. It almost feels like a transformers project rather than peft
So does supporting this feature require custom LongLoRA attention implementations for each model?

Interesting. In theory, we can do that in PEFT also, whether that's a good idea or not depends on what exactly is happening in the modified forward method.
Looking at the code here, it looks quite difficult to implement this in a generic way. In particular, these new methods (actually, functions) don't call super().forward, i.e. they need to replicate exactly what happens inside the method they replace. Perhaps it would be possible to extract the generic parts and put them into a solution that is not model-specific, but that would require some deep digging to figure out.

@BenjaminBossan So in other words, it would be difficult to implement in PEFT because you have to hijack the attention module and add the appropriate calls in the right places? Another solution is for other training libraries to implement custom forward methods and implement it in those, e.g. as they do in axolotl for llama or mistral models. The problem is that this causes a lot of overhead when trying to support this feature.

So in other words, it would be difficult to implement in PEFT because you have to hijack the attention module and add the appropriate calls in the right places?

I think the answer is "it depends". We could do the same thing as in that repo, i.e. provide highly model-specific solutions and then only enable LongLoRA on those models. But I would argue that there would only be little benefit of using PEFT if it just does the exact same, narrow thing.
It would be much better to have a more generic implementation, but whether that works depends on the changes that were introduced. I cannot comment on that, because I haven't looked into the changes in detail. But if they could be generalized and abstracted, they would be great additions to PEFT.
To give a toy example, if the changes amount to adding 1 to to the output of the attention layer, that would be possible to add to PEFT in a generalized fashion. If the changes require 10 modifications in different lines in the forward method, that could not be generalized. Does that make sense?

Another solution is for other training libraries to implement custom forward methods and implement it in those

Could you elaborate on how that is different? Thanks.

Yes, I do agree. It looks like you need several modifications that are neither at the start nor the end of the forward. That makes it quite hard to generalize.

Could you elaborate on how that is different? Thanks.

Training libraries like axolotl implement custom forward functions for some models in order to implement sample packing and enable features like flash attention. They also build on top of PEFT and other Huggingface libraries.

the task that motivates this paper is to extend the context length for transformer language models. Does PEFT support extending the context length (I couldn't find it)? The code we're discussing is only Shift Short Attention, so just implementing Shift Short Attention would allow easy efficient long context fine-tuning but not efficient context extension. I think the Shift Short Attention implementation is the bigger fish to fry but thought this was worth mentioning to appreciate the scope of this feature

Training libraries like axolotl implement custom forward functions for some models in order to implement sample packing and enable features like flash attention. They also build on top of PEFT and other Huggingface libraries.

Okay, so your suggestion is to offload the task of implementing the custom forward methods to the training libs? Yes, there is a bit of a blurry line if these types of changes are in the scope of PEFT or not. I'd say yes, as it's not strictly related to specific training methods, but it could be done either way.

Does PEFT support extending the context length (I couldn't find it)?

No, the idea would be for PEFT to make the modification to the layers, the training itself on longer contexts would be up to the user.

dead?

I'd like to add this feature if one of the great HF MLEs hasn't picked it up yet ;)

You still working on that? I wanna help

如果一个伟大的 HF MLE 还没有拿起它，我想添加这个功能;)

你还在努力吗？我想帮忙

same question

can we implement this on longformer model


Reactify User Profile Page

Open the settings on Beta by clicking your avatar. Then click Show Me My Public Profile. The avatar image and profile image then expand to a 400x400 size.
Edit @quincylarson: We're making this the issue around-which we'll reactify the user profile and settings pages.

Navbar css has changed a lot on the react side, but these changes have not trickled into the static user navbar.
It might be a better option to convert the user page into a react page. I believe this is the last static page that exist in the codebase. (
We would have to take care to handle the case when the user is looking at their own page or an anon user. We definitely don't want to start rendering pages with sensitive user information server side. This is due to our efforts to eventually move to a completely static generated site.

Yes, I have been wondering for some time now that we should be doing this.! If no else beats me to this I would love to get my hands a bit dirty on this one.
Of course after I wind up other pressing PRs..!

Here's the main view that currently powers the profiles: https://github.com/freeCodeCamp/freeCodeCamp/blob/staging/server/views/account/show.jade
The settings page is already reactified. This can serve as a model for what the reactified profile might look like structurally. Here are the components: https://github.com/freeCodeCamp/freeCodeCamp/tree/staging/common/app/routes/settings

@raisedadead have you had a chance to look at this? Would you or @dhcodes be interested in tackling this?

Can I just check if @raisedadead or @dhcodes are working on this?

@zk433 please go ahead... if you would be interested in working on this. happy coding!

agreed!

Ok, me, @chiefoleka and @Exclusiveyash are working on it.

@zk433 @chiefoleka @Exclusiveyash How is this going so far? Let me know if there's anything blocking your path here. We are all extremely excited about a reactified, updated user profiles 😄

@QuincyLarson,   @Exclusiveyash is working on the HTML and CSS now, I and @chiefoleka will attempt to reactify it. I was busy over the past few days but I'll get to it this week. If we struggle, you'll be the first to know :)

@zk433 Thanks for keeping me posted on this. I'm happy to help as necessary. My advice is demo early and often to get feedback :)

Can I try this?

I was just eyeballing this also, for Hacktoberfest... It looks like this is actually in progress by @zk433 et al... If that's the case, you might want to remove the label on this particular issue, or you may just get more and more inquiries as it gets closer to October :)

Hi, yes, we are still working on it. If we get stuck, I'll ask for your help @codejockie  and @Velveeta, so stand by! :)

@zk433 Awesome - thank you for keeping us updated on this.

@zk433 How is it going with this? I'd like to take part working on this as well if help is still needed?

@martindavid Hi! Please see issue #15942 for more details. I'll check with the rest of the team if they need any help, I'm currently working on the user page wireframe.

Is this still available?

@theoutlander It's almost done - @jonathan-grah is finalizing it.

Thanks. I think I’m missing something. Shouldn’t these issues be assigned? I’ve had a problem picking an issue on and off for the past week as everything I tried to claim was being worked on.
Sent from my iPhone
On Dec 22, 2017, at 7:10 PM, Quincy Larson <notifications<EMAIL_ADDRESS>wrote:
@theoutlanderhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftheoutlander&data=02|01||dd8a4057405b4d349cf008d549b2b266|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|636495954196121446&sdata=Td5%2BjvhSpgBwRxbFydjLmKNnijvL2pBLv9YGDKRBhoc%3D&reserved=0 It's almost done - @jonathan-grahhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjonathan-grah&data=02|01||dd8a4057405b4d349cf008d549b2b266|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|636495954196121446&sdata=CL%2BJUK9KNVtj10dlpgOJkrloGdQNd79PyaDrNAV0slo%3D&reserved=0 is finalizing it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FfreeCodeCamp%2FfreeCodeCamp%2Fissues%2F15729%23issuecomment-353703488&data=02|01||dd8a4057405b4d349cf008d549b2b266|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|636495954196121446&sdata=qWjKH7LgvjgGU7hxFtMr1pA2jaUBv4faI%2BYJzsmJzOQ%3D&reserved=0, or mute the threadhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAtuHNfN7AD19Xx6n0x6f6cMQR0aDEVPks5tDG8agaJpZM4OtOKI&data=02|01||dd8a4057405b4d349cf008d549b2b266|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|636495954196121446&sdata=vFFauYa3FguQvOS0yhc%2BoCFK30PkcXqWeDWp4hq1OI0%3D&reserved=0.

@theoutlander Yes - it should have been assigned. We've assigned this one to @jonathan-grah.

Just going through old issues, was this fixed with #16743? https://github.com/freeCodeCamp/freeCodeCamp/pull/16743

Yup. Thanks @akkatracker


[Event Request] Table 36 "Sales Header" - OnBeforeSalesLinesExist

Hello,
We request that the OnBeforeSalesLinesExist() event is added to the SalesLinesExist() procedure in table 36 Sales Header as below.
    procedure SalesLinesExist(): Boolean
+   var
+      IsHandled: Boolean;
    begin
+       IsHandled := false;
+       OnBeforeSalesLinesExist(Rec, xRec, IsHandled);
+       if IsHandled then
+           exit;

        SalesLine.Reset();
        SalesLine.SetRange("Document Type", "Document Type");
        SalesLine.SetRange("Document No.", "No.");
        exit(not SalesLine.IsEmpty);
    end;

    [IntegrationEvent(false, false)]
    local procedure OnBeforeSalesLinesExist(var SalesHeader: Record "Sales Header"; xSalesHeader: Record "Sales Header"; var IsHandled: Boolean)
    begin
    end;

Thanks!

I'm curious what the use-case for this is? And must it always return false, as your added code implicitly will?

Well, it will not always return false, right? Only if IsHandled is set to true. As for the use case; There are situations in which certain (non standard) fields on the Sales Header determine the fact that the Sales Lines should to be changed, for example in RecreateSalesLines() defined in object Sales Header.

For your mentioned use case you should subscribe to the OnBeforeRecreateSalesLinesHandler event in the Sales Header and set the IsHandled to true. I would consider this a code smell if I call SalesHeader.SalesLinesExist() and it would return false even though there are sales lines.

We agree with the above comment.
This is a very straightforward and atomic procedure, that is probably used by many and over-riding it is not a good idea.
Please consider using events that exist in places where this is called but produces the undesirable (for you) result instead of this event.


Disabled shallow clone

Pull Request Test Coverage Report for Build 402723210

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 11.731%

Totals

Change from base Build 402715420:
0.0%

Covered Lines:
224

Relevant Lines:
1831

💛  - Coveralls


Problem with link

Link which is mentioned on the main page doesn't work:
http://codeigniter.com/wiki/imap_pop_class/

sorry - mistake


Relax const-ness in tds_vec_sort signature for Lua 5.2 compatibility

@andresy @soumith - could you merge this PR? It is very simple and #25 is quite annoying for tds users with Lua 5.2. Let us know if we need to check something else before.

done. if you need any other PRs merged, just tag me. sorry for delay Jean.

np - thanks Soumith!


Fix #1070, patch PSP include directory reference

Describe the contribution
Copy of #1072 but main target (Caelum development)
Testing performed
Nothing new, #1072 tested the fix
Expected behavior changes
PSP modules should be able to build again
System(s) tested on
NA, leveraging previous tests
Additional context
Maybe add a PSP module test into the matrix to catch issues like this?
Third party code
None
Contributor Info - All information REQUIRED for consideration of pull request
Jacob Hageman - NASA/GSFC

Or do we want to update w/ --ours and then can use the same commit?

Or do we want to update w/ --ours and then can use the same commit?

Disregard... just needed to merge -s ours with v6.8.0-rc2 on main and can use #1137

Disregard... just needed to merge -s ours with v6.8.0-rc2 on main and can use #1137


KAFKA-7997: Use automatic RPC generation in SaslAuthenticate

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

retest this please

retest this please


3.0 rc1 preparation

This PR prepares the formal next release of pf 3.0-rc1.

Didn't look at the dist stuff (since it should be auto-generated), but the other stuff looks good. +1.

Thx!


DI failing to find Mikro-Orm repositories after upgrading to NestJs v8

Describe the bug
Dependency injection errors are thrown as a result of Nest not being able to find an entity's repository. Repositories are all imported in a single module (database.module.ts) using the forFeature() function.
Stack trace
 at Injector.lookupComponentInParentModules (/home/reganvanheerden/WebstormProjects/EntrostatProjects/askgogo-git-issue/backend/node_modules/@nestjs/core/injector/injector.js:193:19)
    at Injector.resolveComponentInstance (/home/reganvanheerden/WebstormProjects/EntrostatProjects/askgogo-git-issue/backend/node_modules/@nestjs/core/injector/injector.js:149:33)
    at resolveParam (/home/reganvanheerden/WebstormProjects/EntrostatProjects/askgogo-git-issue/backend/node_modules/@nestjs/core/injector/injector.js:103:38)
    at async Promise.all (index 0)
    at Injector.resolveConstructorParams (/home/reganvanheerden/WebstormProjects/EntrostatProjects/askgogo-git-issue/backend/node_modules/@nestjs/core/injector/injector.js:118:27)
    at Injector.loadInstance (/home/reganvanheerden/WebstormProjects/EntrostatProjects/askgogo-git-issue/backend/node_modules/@nestjs/core/injector/injector.js:47:9)
    at Injector.loadProvider (/home/reganvanheerden/WebstormProjects/EntrostatProjects/askgogo-git-issue/backend/node_modules/@nestjs/core/injector/injector.js:69:9)
    at async Promise.all (index 3)
    at InstanceLoader.createInstancesOfProviders (/home/reganvanheerden/WebstormProjects/EntrostatProjects/askgogo-git-issue/backend/node_modules/@nestjs/core/injector/instance-loader.js:44:9)
    at /home/reganvanheerden/WebstormProjects/EntrostatProjects/askgogo-git-issue/backend/node_modules/@nestjs/core/injector/instance-loader.js:29:13

To Reproduce
Steps to reproduce the behavior:

Open repo : git@github.com:Regan-entrostat/askgogo_issue_1.git
Run 'npm install && nest start' in the backend folder

Expected behavior

Repository is supposed to be injected into the service.

Additional context
can get rid of this particular error by putting MikroOrmModule.forFeature([Profanity]) in the imports section of profanity.module.ts file; except this is not a good solution as the errors continue onto the next repository until I come across a repository that is used across multiple services all provided in different modules.
Versions
"@mikro-orm/cli": "^4.5.7",
"@mikro-orm/core": "^4.5.7",
"@mikro-orm/nestjs": "^4.3.0",
"@mikro-orm/postgresql": "^4.5.7",
"@mikro-orm/reflection": "^4.5.7",
"@mikro-orm/sql-highlighter": "^1.0.1",
"@nestjs/common": "^8.0.6",
"@nestjs/core": "^8.0.6",
"@nestjs/jwt": "^8.0.0",
"@nestjs/passport": "^8.0.1",
"@nestjs/platform-express": "^8.0.6",
"@nestjs/throttler": "^2.0.0",

Node version: 14.17.5
Platform: Linux

support for nest v8 was added in v4.3, you are using v4.2
so first try to upgrade to latest, it should be already fixed there

This helped resolve quite a few of the DI failures though unfortunately not all of them. I am happy though to close this issue and do another if I don't get the rest resolved. Thanks very much !

So I am still running into the same issue as I have resented except but with different repositories. I have update the mikro-orm/nest package as suggested. The issue has been replicated and can be found in the link in the above issue. I have also adjusted the issue's stack trace and versions to reflect the changes.

This is actually not really about MikroORM, the issue here is that your custom repositories for some entities are not even registered, like the one for AdminUser, because the @Repository decorator does not get executed before the ORM initialization.
I will probably remove that decorator in v5, it only adds weird issues like this. It can work only if you ensure the file will get required soon enough.
So to resolve this, drop your @Repository decorators from all custom repositories, and provide them via @Entity({ customRepository: () => AdminUserRepository }) etc.


Fix anchors in links to absent pages.

See https://github.com/gollum/gollum/issues/1096

A new edge case I just thought of: an anchored link to a page named Potato#1.md. It will first look for the page Potato#1.md#anchor, and when that fails look for Potato and treat #1.md#anchor as the anchor.
Can/should we fix this by looking for the last rather than the first instance of # and treat that as the anchor? That would break anchored links in which the anchors itself contains the # character, e.g. Potato#achor#doubleanchor, but I'm not sure if that's valid HTML anyway.

Anchors containing # seem to be illegal, see e.g. here

This should fix things. Have also added a couple of tests to deal with further edge cases.


Handle re-declaration of a global variable. Fix #47

Looks great, thanks!
@bors-servo: r+

:pushpin: Commit 8b41159 has been approved by emilio

:zap: Test exempted - status


Add explicit version to .ocamlformat

I believe this is best practice although the error message isn't necessarily great it's still better than reformatting everything by mistake if you run the wrong version of ocamlformat.
It will also be helpful to benefit from https://github.com/ocaml-ci/ocaml-ci/pull/49

We should probably enable ocaml-ci here @avsm !

Upgrading ocamlformat result in a fairly big reformatting of the code base so I'd rather wait for a more stable release, especially given we haven't hit any major bug so far with 0.9!


EachValidator does not respect skipOnEmpty for individual values

As I can see - the underlying problem is in \yii\validators\Validator itself - because skipOnEmpty check is done in validateAttributes().
And \yii\validators\EachValidator directly calls validateAttribute() - so check for skipOnEmpty is bypassed.

Well, for me it seems to be expected. In my projects I always use filter before each to normalize attribute before using each over it.
I'd like to see @klimov-paul's opinon

If it is by design - at least a note in phpdoc should be added, because as of now it is confusing:
...in the same format as at [[yii\base\Model::rules()]]...

Resolved by commit d22a3a0a07befc274cac49a96dfcea3372953138


Issue warning on identical derivative filenames.

There is no prohibition against identical filenames in different derived datasets, although users should be aware of the potential ambiguity this can create and use the sidecar JSON files to detail the specifics of individual files.

Derivatives are validated independently. I don't think there's continued interest in changing this.


Move command to standard cmd folder used in Go

@lorin PTAL. You need to update the Wiki too. 😄

@mlafeldt If you merge in the latest version of master, it should resolve the failing Travis build.

Rebased on top of master


[Gettr] Add GettrStreamingIE

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

[x] At least skimmed through contributing guidelines including yt-dlp coding conventions
[x] Searched the bugtracker for similar pull requests
[x] Checked the code with flake8

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[x] I am the original author of this code and I am willing to release it under Unlicense
[ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[ ] Bug fix
[ ] Improvement
[x] New extractor
[ ] New feature

Description of your pull request and other information
Add support for gettr /streaming/ url
(closes #2654)

Could you also have a look at #2557?


Fix issue #322 Pasting a string containing a comma set the caret position at the wrong position

Coverage increased (+0.03%) to 49.485% when pulling e3b2ac2b885e1730aa299b563ef275e7a474f4ef on AlexandreBonneau:issue_322 into 5e401f24a6ae5d97e439f2912d9d43953608f00e on BobKnothe:next.


Revert arrows reverting the wrong chunk

Testing #186213

Thanks for reporting! Please share the input files the next time ;)
Original:
a
b
c
d
e
f
g

Modified:
a
c
d
e
f


2 mouse clicks instead of one

Hello,
When i'm lazy i sometimes prefer using the mouse over vim movements ( :-1: )... like clicking on files in nerdtree  or to select some text.
In pynvim, it looks like doing one mouse click automatically does 2 mouse clicks, since i only need to do one click instead of two to open a file or directory in nerdtree (which is pretty annoying if i misclicked), and it takes only one click to expand region in visual mode (which is annoying if i want to select only 2 words for example).
As far as i can tell, this is not the default behavior in neovim, i tried this in neovim-qt and it also seems to be ok.
Is there anyway to configure this in pynvim ?
Cheers,
Mathiasb17

Adding return True to the end of _gtk_button_press improves it somewhat but not sure it is still completely correct.

I have the same behavior running python-gui in a virtual box

I've updated the repo description:

Proof-of-concept Neovim GUI. Not maintained, but patches are welcome.

This GUI is mostly a demo. Help in maintaining it would be welcome, but currently there's no one actively maintaining it.

just made some investigation to the problem:
# simple.py
# taken from https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html
import gi
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk

def log_event(widget, event):
    print("event", widget, event.type)

win = Gtk.Window()
win.connect("delete-event", Gtk.main_quit)
win.connect("button-press-event", log_event)
win.connect("button-release-event", log_event)
win.show_all()
Gtk.main()

results in a duplicate event printed:
# python simple.py
('event', <Gtk.Window object at 0x7ffa3643b370 (GtkWindow at 0x2a74240)>, <enum GDK_BUTTON_PRESS of type Gdk.EventType>)
('event', <Gtk.Window object at 0x7ffa3643b370 (GtkWindow at 0x2a74240)>, <enum GDK_BUTTON_PRESS of type Gdk.EventType>)
('event', <Gtk.Window object at 0x7ffa3643b370 (GtkWindow at 0x2a74240)>, <enum GDK_BUTTON_RELEASE of type Gdk.EventType>)
('event', <Gtk.Window object at 0x7ffa3643b370 (GtkWindow at 0x2a74240)>, <enum GDK_BUTTON_RELEASE of type Gdk.EventType>)

seems to be an issue with pygtk...


Amazon Hub Support

Proposed change

Handle Amazon Hub emails
Parse combo from subject line and populate as sensor attribute

Type of change

[ ] Dependency upgrade
[ ] Bugfix (non-breaking change which fixes an issue)
[x] New feature (which adds functionality)
[ ] Breaking change (fix/feature causing existing functionality to break)
[ ] Code quality improvements to existing code or addition of tests

Additional information

This PR fixes or closes issue: fixes #206

Still need to test this locally.

Local testing, no errors.


Added Java Maze Generator

Community Contribution
Link to live project, video, or image:

Sharing

[x] I would be happy to have my project shared on The Coding Train's social media!

Instagram:
Twitter:

:)


"OSError: Fail to write. Disk may be full" when create model for trainning

Hi,
I'm using visdrone dataset for training a person/pedestrian detection model. Visdrone2018 train dataset has around 25k images and I'm using 14k for trainning. This is my structure for json file:

Each line is an annotation for an image which contains many bounding boxes for 1 label. It's exactly like the structure of data requirements from docs.
The problem is when i loaded data to sframe variable and init a model with tc.object_detector.create. During state "Using GPU to create model", it keeps failing beucause OSError.
This is my code for init a model:
  print ('-----------------------------------PREPARATION DATA-----------------------------------')
  #path = "/storageStudents/Datasets/VisDrone/VisDrone2018-VID-train/VisDrone2018-VID-train/annotations/*.txt"
  path = os.path.join(anno,"*.txt")
  files = glob.glob(path)

  data=[]
  print ('---------------------------------------------------------------------------------------------')
  print ('---------------------------------------------------------------------------------------------')
  print ('---------------------------------------------------------------------------------------------')

  train_imgs_dir = os.path.join(os.getcwd(),"image_train")

  raw_sf_train = tc.image_analysis.load_images(train_imgs_dir, recursive=False, random_order=False)

  # Split file names so that we can determine what kind of image each row is
  info_train = raw_sf_train['path'].apply(lambda path: os.path.basename(path).split('.')[:2])

  # Rename columns to 'name' and 'type'
  info_train = info_train.unpack().rename({'X.0': 'name', 'X.1': 'type'})

  # Add to our main SFrame
  raw_sf_train = raw_sf_train.add_columns(info_train)

  # Extract label (e.g. 'bike') from name (e.g. 'bike_003')
  raw_sf_train['label'] = 'person'

  # Original path no longer needed
  del raw_sf_train['path']
  del raw_sf_train['type']

  print ('------------------------LOAD DATA ANNOTATION TRAIN--------------------------')
  # Load the data
  train_data = SFrame.read_json(os.path.join(os.getcwd(),'data_train_visdrone.json'), orient='lines')
  train_annotations = SFrame(train_data)

  sf_train = raw_sf_train.join(train_annotations, on='name', how='right')

  sf_train['annotations'] = sf_train['annotations'].fillna([])

  sf_train['label'] = sf_train['label.1']
  sf_train = sf_train.remove_column('label.1')

  print ('------------------------PREPARE DATA COMPLETED--------------------------')

  # Create a model
  model = tc.object_detector.create(sf_train, feature='image', annotations='annotations', batch_size=16)

  file_name = 'ObjectDetectionVisdrone.model'

  # Save the model for later use in Turi Create
  model.save(file_name)

And i got this error:

Is there some setting that needs to be enabled for these memory issues?

I still have a large memory on disk space, so there is no sense about hard disk memory

Any solution ?

What do mean by "Each line is an annotation for an image which contains many bounding boxes for 1 label?" Do you mean that each image has many bounding boxes? Typically each annotation has exactly one bounding box and exactly one label.
The error is occurring in a part of our code where we convert the SFrame from having one row per image to having one row per image-annotation pair. As a sanity check, could you see what happens if you change your annotation column to have just one annotation per row (stored as a single dictionary, instead of a list of dictionaries)?

I'm going to close this issue due to inactivity. Free feel to reopen with the requested information.


update oauth2-mondo repo references to oauth-mondo

When I first added the library to my project the composer require command was failing as the package name was wrong. I have updated that in this PR and also updated the two links to the contribute and license files.

Whoops - good spot! 👍

I've actually fixed Composer so that the package has the correct name of oauth2-mondo - the naming was a mistake and should've started oauth2- as per the recommendations here:
http://oauth2-client.thephpleague.com/providers/implementing/


Add venue

Add an option for the user to input the name of the venue the concert was at.

I'll work on this one!


breadcrumbs should be coherent across pages

Description
Here are some rules that should enable this:

current page not in breadcrumbs
tabs not in breadcrumbs

Closing this and we'll maybe rework this once design is complete


How we change the server id of Arnagodb 3.0 server?

Version: Arangodb 3.0
OS: Ubuntu 14.04 (AWS EC2 instance)
I am working on replication feature of Arangodb. So first I have setup and start a new server on AWS EC2 instance. It is working fine.
Then for replication feature, I have created an AMI (image) of the previous instance and launch it. Now both server is working properly.
When I want to start replication between two servers, I am getting the following issue as below.
JavaScript exception in file<EMAIL_ADDRESS>at 181,7: ArangoError 1405: got same server id<PHONE_NUMBER>61541) from endpoint 'tcp://******:8529' as the local applier server's id
!      throw err;
!      ^
stacktrace: ArangoError: got same server id<PHONE_NUMBER>61541) from endpoint 'tcp://******:8529' as the local applier server's id
    at Object.exports.checkRequestResult (/usr/local/share/arangodb3/js/client/modules/@arangodb/arangosh.js:98:21)
    at waitForResult (/usr/local/share/arangodb3/js/client/modules/@arangodb/replication.js:178:14)
    at Object.sync (/usr/local/share/arangodb3/js/client/modules/@arangodb/replication.js:228:10)
    at <shell command>:1:34

I have found the answer for same on stackoverflow. So this issue due to same AMI. So I want to edit server-id.
But I want to know, where I find this file to edit server id?

I have found the SERVER file at /usr/local/var/lib/arangodb3.


⚠ Spryr has degraded performance

In a7ba6dd, Spryr (https://spryr.com) experienced degraded performance:

HTTP code: 200
Response time: 3613 ms

Resolved: Spryr performance has improved in 9b63d21 after 7 minutes.


Public share improvement?

First thing first: awesome project!
Pleasantries aside, let's cut to the chase: code installed on Apache 2.4.18 with proper mods & settings (as explained in the wiki itself), public mode on false.
Well...
Expected behavior by me: shared links are public, litteraly, one doesn't need login to reach them properly (preferred choice), or else appears a login request
Actual behavior: if one isn't logged in, shared links lead to a white page with Droppy log showing error [401] (authentication required)
I guess actual behavior is the planned one, but...do you plan to improve the feature, by any chance?
Thanks

That sounds like a bug, I'll try to replicate.

Fixed in v6.6.8. I'm sorry I didn't notice it earlier those links they were broken.

Just tried: now it's working flawlessly, of course!
Very kind of you 👍

Sorry, still me...I spoke too soon...after further tests, I found out another strange behavior: in both public and private mode, directory share link (without "triggered download" flag) leads to a file (which is a proper zip file of the direcotry, in fact) named like the last five random letters of the link itself, instead of the directory listing, as I guess one would expected.
Execution log reports a standard "GET /$/<last 5 letters> [200]".
If required, I'll open another issue.
Thanks!

Ah, there was a authentication check on directory downloads that I missed, fixed now and will be released as v6.6.9 in a few minutes.
I also changed the trigger download functionality on directory downloads. The links will now always result in the correct filename and when "Is DL" is not checked, the download will be sent "inline", which leaves it to the browser to decide what to do with the zip (They'll most likely download it anyways unless they can display the zip "inline").

Bingo! Very appreciated your responsiveness 👍


upgrade-package

General Checklist:

[ ] Tests added for this feature/bug

[ ] All tests are passed

[ ] No lint error

Review

[ ] Has been reviewed

[ ] Conform by the [merge request performance guides]

[ ] Conform by the [style guides]

[ ] Squashed related commits together

Tasks Checklist

[ ] Todo something 1
[ ] Todo something 2
[ ] Todo something 3

What are the relevant issue numbers?
Closes #

Pull Request Test Coverage Report for Build 190

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 69.427%

Totals

Change from base Build 188:
0.0%

Covered Lines:
213

Relevant Lines:
304

💛  - Coveralls

Pull Request Test Coverage Report for Build 190

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 69.427%

Totals

Change from base Build 188:
0.0%

Covered Lines:
213

Relevant Lines:
304

💛  - Coveralls


Custom float delimeter or parse comma as dot.

Hello,
i tried plugin i have some problem with parsing float i discovered that its not compatible with float with comma deliver like:
0,01 but it need 0.01 It would be nice to handle that, for these who are not familiar with Linux scripting, sed etc.. Comma delimeter is used for lots of locales.

Thanks for reporting this.
Should be fixed in v1.2
Please report back if it's working as expected.


Missing stdbool.h on x86_64-pc-windows-gnu

Checklist

[X] I've looked through the issues and pull requests for similar reports

Describe your issue
When cross compiling wgpu-native for x86_64-pc-windows-gnu, compilation fails with
ffi/webgpu-headers/webgpu.h:53:10: fatal error: 'stdbool.h' file not found
What target(s) are you cross-compiling for?
x86_64-pc-windows-gnu
Which operating system is the host (e.g computer cross is on) running?

[ ] macOS
[ ] Windows
[X] Linux / BSD
[ ] other OS (specify in description)

What architecture is the host?

[X] x86_64 / AMD64
[ ] arm32
[ ] arm64 (including Mac M1)

What container engine is cross using?

[X] docker
[ ] podman
[ ] other container engine (specify in description)

cross version
cross 0.2.5 (892fdfa 2023-05-02)
Example
git clone https://github.com/gfx-rs/wgpu-native
cd wgpu-native
cross build --target x86_64-pc-windows-gnu

exitied with code: 101
   Compiling wgpu-native v0.0.0 (/home/beyley/Projects/PROGRAMMING/Zig/ztyping/libs/wgpu-native)
   Compiling wgpu-hal v0.16.0 (https://github.com/gfx-rs/wgpu?rev=011a4e26d04f388ef40e3baee3f19a255b9b5148#011a4e26)
error: failed to run custom build command for `wgpu-native v0.0.0 (/home/beyley/Projects/PROGRAMMING/Zig/ztyping/libs/wgpu-native)`

Caused by:
  process didn't exit successfully: `/target/debug/build/wgpu-native-0dd799e855b15816/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=ffi/webgpu-headers/webgpu.h
  cargo:rerun-if-changed=ffi/wgpu.h
  cargo:rerun-if-env-changed=TARGET
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64-pc-windows-gnu
  cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_x86_64_pc_windows_gnu

  --- stderr
  ffi/webgpu-headers/webgpu.h:53:10: fatal error: 'stdbool.h' file not found
  thread 'main' panicked at 'Unable to generate bindings: ClangDiagnostic("ffi/webgpu-headers/webgpu.h:53:10: fatal error: 'stdbool.h' file not found\n")', build.rs:65:39

Additional information / notes
No response

You need to install clang, see https://github.com/cross-rs/cross/wiki/FAQ#using-clang--bindgen
in this case, since you're using cross 0.2.5 which doesn't use ubuntu 20.04 it'd be
# Cross.toml
[build]
pre-build = ["apt-get update && apt-get install --assume-yes --no-install-recommends libclang-3.9-dev clang-3.9"]

does that work?

You need to install clang, see https://github.com/cross-rs/cross/wiki/FAQ#using-clang--bindgen
in this case, since you're using cross 0.2.5 which doesn't use ubuntu 20.04 it'd be
# Cross.toml
[build]
pre-build = ["apt-get update && apt-get install --assume-yes --no-install-recommends libclang-3.9-dev clang-3.9"]

does that work?

Adding that to a Cross.toml file in my wgpu-native submodule then causes this output (i call cross from inside a zig build script)
┌─[beyley@arch] - [~/Projects/PROGRAMMING/Zig/ztyping] - [2023-05-04 02:24:50]
└─[0] <git:(master 7c48b52✱✈) > zig build -Dtarget=x86_64-windows
exitied with code: 1
unknown flag: --progress
See 'docker --help'.

Usage:  docker [OPTIONS] COMMAND

A self-sufficient runtime for containers

Common Commands:
  run         Create and run a new container from an image
  exec        Execute a command in a running container
  ps          List containers
  build       Build an image from a Dockerfile
  pull        Download an image from a registry
  push        Upload an image to a registry
  images      List images
  login       Log in to a registry
  logout      Log out from a registry
  search      Search Docker Hub for images
  version     Show the Docker version information
  info        Display system-wide information

Management Commands:
  builder     Manage builds
  container   Manage containers
  context     Manage contexts
  image       Manage images
  manifest    Manage Docker image manifests and manifest lists
  network     Manage networks
  plugin      Manage plugins
  system      Manage Docker
  trust       Manage trust on Docker images
  volume      Manage volumes

Swarm Commands:
  swarm       Manage Swarm

Commands:
  attach      Attach local standard input, output, and error streams to a running container
  commit      Create a new image from a container's changes
  cp          Copy files/folders between a container and the local filesystem
  create      Create a new container
  diff        Inspect changes to files or directories on a container's filesystem
  events      Get real time events from the server
  export      Export a container's filesystem as a tar archive
  history     Show the history of an image
  import      Import the contents from a tarball to create a filesystem image
  inspect     Return low-level information on Docker objects
  kill        Kill one or more running containers
  load        Load an image from a tar archive or STDIN
  logs        Fetch the logs of a container
  pause       Pause all processes within one or more containers
  port        List port mappings or a specific mapping for the container
  rename      Rename a container
  restart     Restart one or more containers
  rm          Remove one or more containers
  rmi         Remove one or more images
  save        Save one or more images to a tar archive (streamed to STDOUT by default)
  start       Start one or more stopped containers
  stats       Display a live stream of container(s) resource usage statistics
  stop        Stop one or more running containers
  tag         Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
  top         Display the running processes of a container
  unpause     Unpause all processes within one or more containers
  update      Update configuration of one or more containers
  wait        Block until one or more containers stop, then print their exit codes

Global Options:
      --config string      Location of client config files (default
                           "/home/beyley/.docker")
  -c, --context string     Name of the context to use to connect to the
                           daemon (overrides DOCKER_HOST env var and
                           default context set with "docker context use")
  -D, --debug              Enable debug mode
  -H, --host list          Daemon socket(s) to connect to
  -l, --log-level string   Set the logging level ("debug", "info",
                           "warn", "error", "fatal") (default "info")
      --tls                Use TLS; implied by --tlsverify
      --tlscacert string   Trust certs signed only by this CA (default
                           "/home/beyley/.docker/ca.pem")
      --tlscert string     Path to TLS certificate file (default
                           "/home/beyley/.docker/cert.pem")
      --tlskey string      Path to TLS key file (default
                           "/home/beyley/.docker/key.pem")
      --tlsverify          Use TLS and verify the remote
  -v, --version            Print version information and quit

Run 'docker COMMAND --help' for more information on a command.

For more help on how to use Docker, head to https://docs.docker.com/go/guides/

Error:
   0: could not run container
   1: when building custom image
   2: when pre-building
   3: `docker buildx build --progress auto --label 'org.cross-rs.for-cross-target=x86_64-pc-windows-gnu' --label 'org.cross-rs.runs-with=x86_64-unknown-linux-gnu' --label 'org.cross-rs.workspace_root=/home/beyley/Projects/PROGRAMMING/Zig/ztyping/libs/wgpu-native' --tag localhost/cross-rs/cross-custom-wgpu-native:x86_64-pc-windows-gnu-38c49-pre-build --build-arg 'CROSS_CMD=apt-get update && apt-get install --assume-yes --no-install-recommends libclang-3.9-dev clang-3.9' --file /home/beyley/Projects/PROGRAMMING/Zig/ztyping/libs/wgpu-native/target/x86_64-pc-windows-gnu/Dockerfile.x86_64-pc-windows-gnu-custom --output 'type=docker' /home/beyley/Projects/PROGRAMMING/Zig/ztyping/libs/wgpu-native` failed with exit status: 125

There should be more text underneath, suggesting that buildkit needs to be installed or disabled. See #1055

I had to update the libclang to the second set of packages on the FAQ, but after that, its all worked and my script is finding the static lib! all i have to do now is fix the linker errors when i link against the static lib (unrelated to cross, i just need to add a few more libs to my linker args)

awesome! fyi, we have integration with cargo-zigbuild via https://github.com/cross-rs/cross/wiki/Configuration#buildzig don't know if it could be useful for you but just so you know :)


framework: add event callback for controller button key presses

GearVRf-DCO-1.0-Signed-off-by: Mihail Marinov<EMAIL_ADDRESS>

Will add the rest of the buttons later

Planning to merge on Friday


Refactor(navigation): Move navigation functions to library for use in…

… other test subsets (RHOSR/RHOAD/...)

failed tests are not related to this PR


Memory Leak in webview::bind (sync)

THIS IS ONLY FOR THE SYNC VERSION OF BIND
I'm receiving a C string that I need to free from a library, but I want to send it to JS. So I have 2 possible situations:

I can't free the string before returning, that would segfault
If I don't free it's a memory leak, because std::string won't free memory initialized elsewhere

What OS are you using (uname -a, or Windows version)?
Windows 10 21H1 build 19043.1889
What programming language are you using (C/C++/Go/Rust)?
>cl
Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30133 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

What did you expect to see and what you saw instead?
I "tested" (I ran the program and looked at task manager) to confirm if the memory leaks or not, and it does.
I tested with the following code
#include "webview.h"
#include <stdlib.h>

int main(int argc, char** argv) {
    auto webview = new webview::webview(true);

    webview->set_title("Webview Example");
    webview->set_size(480, 320, WEBVIEW_HINT_NONE);
    webview->bind("testMemoryLeak", [](const std::string &req) {
        char *bytes = (char*)malloc(1 << 25);
        bytes[0] = '"';
        bytes[(1 << 25) - 2] = '"';
        bytes[(1 << 25) - 1] = 0;

        for (auto i = 1; i < (1 << 25) - 2; i++) bytes[i] = ' ';

        return std::string(bytes);
    });
    webview->bind("testMemorySecure", [](const std::string &req) {
        std::string bytes = std::string(1 << 25, ' ');
        bytes[0] = '"';
        bytes[(1 << 25) - 2] = '"';
        bytes[(1 << 25) - 1] = 0;

        return bytes;
    });
    webview->navigate(
        "data:text/html, "
        "<button onclick=\"testMemoryLeak()\">"
        "  Test Leaky"
        "</button>"
        "<button onclick=\"testMemorySecure()\">"
        "  Test Secure"
        "</button>"
    );
    webview->run();

    webview->terminate();
    delete webview;
}

and compiled with cl /W4 /wd4100 /I "." /I ".\script\microsoft.web.webview<IP_ADDRESS>3.44\build\native\include" ".\dll\x64\webview.lib" ".\dll\x64\WebView2Loader.lib" "webview.cc" /std:c++17 /EHsc /OUT:webview.exe
There are two buttons, both will build a string with (1 << 25) characters, one builds with std::string, the other with malloc. The one that builds with std::string frees the memory (1st image), the other does not (2nd image)

The current workaround is to use the async version, so you can call result and then free. It would be nice to warn whoever is using the method of this, or provide a way to solve this problem (I think rewriting as async is good enough)

Sorry, I am not sure I fully understand the problem you are describing.
Can you please carefully explain why you do not or cannot free bytes before returning from the function, e.g. like this?
webview->bind("testMemoryLeak", [](const std::string &req) {
    char *bytes = (char*)malloc(1 << 25);
    bytes[0] = '"';
    bytes[(1 << 25) - 2] = '"';
    bytes[(1 << 25) - 1] = 0;

    for (auto i = 1; i < (1 << 25) - 2; i++) bytes[i] = ' ';

    std::string result(bytes);
    free(bytes);
    return result;
});

To have to copy seems kind of a waste, but definitely a solution (only possible in C++ though).
Giving the full context, I needed to write UI for a Zig program, so I wrote bindings to webview in it. I saw that bind sync was only exposed to C++, but seemed like a nice feature to have as well, so I made some edits and exposed it to the C api (this is why I didn't write about copying the contents to a std::string). Being Zig a manually managed memory language, I have to free after what I allocate, thus leading to the problem described.
I thought C++ was a better choice for explaining the problem, as the API used is only exposed there. I don't think it's a problem for those paying attention to what they are doing with the memory, but for a less well-advised user it's easy to return a char* because it will be implicitly cast to as std::string, so It would be nice to, at least, warn the user in the docs about this situation
TL;DR: I exposed bind sync to the C api, used it in another language, but I can't return and free the memory in the same function.

I am going to close this for now.
I think I see your point but it is hard for me to see it as an issue because the library does not expose this in the C API officially.
It is not an issue in the C++ world as the callback function returns std::string which becomes more obvious if you write it as [](const std::string &req) -> std::string {}.
If you want to make a C style callback function return char * then you need to decide who owns the string, let the owner free the string, and call the matching allocator/deallocator.
Instead of dealing with all this, maybe you could just wrap the asynchronous bind in a synchronous way in Zig.


Fixed empty array on $or

Fixed Error: Failed running search for results (results) at result: MongoError: $and/$or/$nor must be a nonempty array

Fails

:no_entry_sign:

The changelog has not been updated. Please update the changelog.

:no_entry_sign:

The version was not updated. Please update the version.

Warnings

:warning:

The README has not been updated. Please update the README.

Messages

:book:

We were able to automatically fix some formatting issues in this PR for you!

  Generated by :no_entry_sign: dangerJS


Rethink notification system

Maybe do that thing that Intellij does where you pop up a little bubble at the bottom of the page? The notification pane is annoying
(Also needs to be changed in smaller windows like branchcheckout and merge windows)

Moved it to the bottom, less annoying there, leaving for later

Added via e6a57b831b29d0903213f46a52b06988277f4333


Capybara replaced with ultrafeedback_binarized

What does this PR do?
Fixes #2176
Before submitting

[x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you read the contributor guideline,
Pull Request section?
[x] Was this discussed/approved via a GitHub issue? Please add a link
to it if that's the case.
[ ] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines.
[ ] Did you write any new necessary tests?

Who can review?
@qgallouedec
Did I miss any?

Thanks a lot @August-murr!


chore(deps): update onsi

This PR contains the following updates:

Package
Type
Update
Change

github.com/onsi/ginkgo/v2
require
patch
v2.20.1 -> v2.20.2

github.com/onsi/gomega
require
patch
v1.34.1 -> v1.34.2

Release Notes

onsi/ginkgo (github.com/onsi/ginkgo/v2)
v2.20.2
Compare Source
2.20.2
Require Go 1.22+
Maintenance

bump go to v1.22 [a671816]

onsi/gomega (github.com/onsi/gomega)
v1.34.2
Compare Source
1.34.2
Require Go 1.22+
Maintenance

bump ginkgo as well [c59c6dc]
bump to go 1.22 - remove x/exp dependency [8158b99]

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

ℹ Artifact update notice
File name: go.mod
In order to perform the update(s) described in the table above, Renovate ran the go get command, which resulted in the following additional change(s):

2 additional dependencies were updated

Details:

Package
Change

github.com/google/pprof
v0.0.0-20240727154555-813a5fbdbec8 -> v0.0.0-20240827171923-fa2c70bbbfe5

golang.org/x/sys
v0.23.0 -> v0.24.0

Pull Request Test Coverage Report for Build<PHONE_NUMBER>6
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 63.209%

Totals

Change from base Build<PHONE_NUMBER>6:
0.0%

Covered Lines:
1237

Relevant Lines:
1957

💛  - Coveralls


Update README

First, have a look on oemof's README are make structure similar.
TODOs

[x] remove/replace announcement of planned oemof release
[x] Redesign installation section: installation via pip and developer istallation
[x] Update configuration
[x] Remove required packages section

There could be more improvement, but not for now...


Move Image from Bess' private repo to quay

Once we can have private repositories in Quay move the circle ci mediflux build from @bess's private repo to where we store our other repos (in quay)

Closing in favor of a different solution: https://github.com/pulibrary/tiger-data-app/issues/871


Implement parameter navpanes and toolbar

See: https://bugzilla.mozilla.org/show_bug.cgi?id=844350

with the acrobat pdf plugin i can pass navpanes=0 and toolbar=0 to hide the toolbars

The parameters are specified here: http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/pdf_open_parameters_v9.pdf#page=7

https://github.com/mozilla/pdf.js/issues/9414 was closed with a reference to this issue, but I'm not sure if the parameters are the same as "auto-hide" toolbar. In case someone is searching for a workaround to gain space, create a userContent.css in profile>chrome with the following content (instead of hiding the toolbar completely, it's just made transparent:

/* https://support.mozilla.org/de/questions/1119523 */
#outerContainer #mainContainer div.toolbar {
/display: none !important; / hide PDF viewer toolbar /
opacity: 0.5 !important;
}
#outerContainer #mainContainer #viewerContainer {
top: 0 !important; / move doc up into empty bar space */
}

I opened this: https://bugzilla.mozilla.org/show_bug.cgi?id=1807490
And was directed here - what's the status of the various unimplemented pdf parameters in pdf.js?

/display: none !important; / hide PDF viewer toolbar / opacity: 0.5 !important;

Thanks, but strange that display: none and opecity: ... in one block?

Thanks, but strange that display: none and opecity: ... in one block?

Corrected, just seen that the * were not displayed. Learned now that one can use triple ``` to insert code in comments that includes new line characters ;)

Improved - to reveal the UI on mouse hover:
/* https://github.com/mozilla/pdf.js/issues/2784#issuecomment-366480669 */
#outerContainer #mainContainer div.toolbar {
	opacity: 0;
}
#outerContainer #mainContainer div.toolbar:hover {
	opacity: 1;
}

#outerContainer #mainContainer #viewerContainer {
	top: 0 !important; /* move doc up into empty bar space */
}

#viewerContainer {
	overflow-x: hidden !important;
}


Compiling WindbgExtension in release (Mixed mode) does not generate exports

Hi,
When compiling the windbgextension sample in Visual Studio 2015 release mode (Mixed mode) does not appear to generate exports of methods marked with [DllExports], this work in debug mode.  The missing exports makes it impossible for windbg to find the extension point.
To reproduce simply compile in release.
When running dumpbin /exports windbgextension.dll the exports are correct in debug:
Dump of file WindbgExtension.dll
File Type: DLL
Section contains the following exports for \WindbgExtension.dll
00000000 characteristics
560E2497 time date stamp Fri Oct 02 08:30:47 2015
0.00 version
0 ordinal base
2 number of functions
2 number of names
ordinal hint RVA      name
0    0 00004DE6 DebugExtensionInitialize
1    1 00004DF2 heapstat   <== all good
Summary
2000 .reloc
2000 .rsrc
2000 .sdata
4000 .text
Release mode will give you the following:
Dump of file WindbgExtension.dll
File Type: DLL
Summary
2000 .reloc
2000 .rsrc
4000 .text
If you disassemble the 2 output files in ilspy, you get the following for debug:
[DllExport("heapstat")] <== all good
public static void HeapStat(IntPtr client, [MarshalAs(UnmanagedType.LPStr)] string args)
{
if (!DebuggerExtensions.InitApi(client))
{
return;
}
...omitted..
For release you get the following:
<== missing attribute
public static void HeapStat(IntPtr client, [MarshalAs(UnmanagedType.LPStr)] string args)
{
bool flag = !DebuggerExtensions.InitApi(client);
if (!flag)
...omitted..
The same goes for DebugExtensionInitialize of course.

Ok, played a little more with it and it turns out if you change the platform target under release(mixed mode) from any CPU to X64 the exports show up.


Fix window in vscode

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version:
OS Version:

Steps to Reproduce:
1.I want to when I write code
Do not move from the width I set for that window and do not scroll my page
2.

@HasanKanani1 I am not sure I understand the problem correctly. Can you please attach a screen shot showing your problem?

I want some of this page to overflow, I want the line to be cut

@HasanKanani1 You can use View > Word Wrap.


[#219] Install pgmoneta

Follow these steps to install pgmoneta.

You can modify the pgmoneta/doc/tutorial/01_install.md file to serve as a configuration file.

Move this to a doc/DEVELOPERS.md file

Start by squashing your commits, and force push.
https://www.git-tower.com/learn/git/faq/git-squash
https://github.com/progit/progit2/releases/tag/2.1.423
Then we can take a closer look

Sorry about that, this is my first time making a pull request.

We should fix #209 first

Sorry about that, this is my first time making a pull request.

It is ok, normally we want only one commit for each PR. Perhaps we could add that reminder in DEVELOPERS.md.

Hi, you should make a separate PR for the fix for #209 . And remember to use uncrustify to format your code. You can add a reminder to that in DEVELOPERS.md as well :)

"Could you please close this pull request? I made a mistake. My apologies.

I will submit the pull request again, this time separating [#209] and [#210], and adding a reminder in the file. If for [209], it's just assigning a value to slot_status, I would prefer to do that.

Just use this pull request, and force push your changes to your branch

Finally, I've fixed it and learned so much. Thank you!

When I attempted this, I encountered two problems:

Yesterday, when I added the master key, there were no issues. However, today when I tried to add it, I had to input a string longer than 9 characters to proceed. Otherwise, it kept asking for the master key.

During the backup, there was no influence on the details; the used space always showed as 0.

update database:

check backup:

Could you help me understand the possible reasons for these issues?

When I attempted this, I encountered two problems:

Yesterday, when I added the master key, there were no issues. However, today when I tried to add it, I had to input a string longer than 9 characters to proceed. Otherwise, it kept asking for the master key.

During the backup, there was no influence on the details; the used space always showed as 0.

update database:

check backup:

Could you help me understand the possible reasons for these issues?

@GuChad369 For the issues you need to open a forum discussion about them

Please, rebase

Done.
I'm not sure if this approach is correct.

You need to squash

Done

Now the 2 PRs are the same...

My bad. How should we handle this? I am currently attempting to undo the squash.

Take your patches, split them in to the 2 pull requests. Delete the old branch, create a new branch with the same name, apply the patch and force push

Look in the ProGit book - the command you are looking for is git reset --hard

AFTER you have got your patch with f.ex. git format-patch

Done.
I've acquired new knowledge. Thank you for your patience and guidance.

Hi, seems that a lot of people are not very familiar with git, which is totally normal with new contributors. Could you add a git workflow with command examples in the doc?

Done

We need to add the new file to pgmoneta.spec as well...

fantastic!
a new thing!

You need to rebase

Done

Merged.
Thanks for your contribution !


Enable persistence for pulsar-manager

Motivation
Currently, pulsar-manager volume is not persistent, which means that after a pod restart admin accounts must be recreated every time. I want to change that.
Modifications
Switch pulsar-manager from a Deployment to a StatefulSet and add settings to enable persistence. The settings are inspired by those offered for other components.
Also, the Docker image will always reinitialize the DB, so I made a PR to fix that too.
Verifying this change

[ ] Make sure that the change passes the CI checks.

The pulsar-manager persistence fix has been merged, so it would be nice if this fix was merged too :smile:

Any chance we can get this merged? LGTM!

@vfauth Thanks for the contribution. Do you have a chance to rebase the changes and ensure that this works with recent versions?

@Mortom123 Please check this PR, is something like this needed after #457?

Yes this is an open issue. However I think it could be quite charming to use the Zookeper as store for the Pulsar-Manager configuration data. @lhotari

Thanks for the contribution @vfauth ! This is merged now and will be included in the next release of the chart.


⚠ Bofrost.at has degraded performance

In c11e51c, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2132 ms

Resolved: Bofrost.at performance has improved in d1f31a2 after 6 minutes.


Add inference part of PSPNet

Reopens https://github.com/chainer/chainercv/pull/388

@Hakuyume
Please review.
Also this PR depends on https://github.com/chainer/chainercv/pull/441.

Thanks

some executable scripts don't use main().

How about making examples/semantic_segmentation/README.md? And examples/segnet/eval_camvid.py should be moved under semantic_segmentation.


install: fixed typo

Thanks @novoid  ! I'll take a look at this soon.

I appreciate the help @novoid !

LMAO! This one slipped past me 🤣


Get client and server semver only

This is a quick question. It seems like you can run helm version --server/--client --short to get the version of server or client component. It also seems to support a --template flag, but there is no documentation of how to construct the string. I assume there is a way to get just the semver value using this flag but can someone provide an example?

Hi @eroji :
Thanks for your question. The --template flag simply accepts a golang text/template that is then used to format the output. Using this, you could get just the server/client semvers by running something like the following:
$ helm version --template "{{ .Client.SemVer }}"
v2.13.1

$ helm version --template "{{ .Server.SemVer }}"
v2.13.1

Hope this helps.

Closing this issue. Feel free to re-open if things are still unclear. Thanks!


llvm: stop running strip-preserve-atime.test

This test fails in the presence of anti-virus software or other intrusion-detection software that modifies the atime when run.
See nixos/nixpkgs#284056 and llvm/llvm-project#82372.
Description of changes
Cherry picked this commit from nixpkgs-unstable.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

This is a MR against the wrong branch, reopening on the right one.


format code style of [package import]

format the code style of package import

When builded the docker images , occured [Error: Cannot perform an interactive login from a non TTY device]. I cannot get the reason,would you like to help me. @netshade @jpkrohling @Dieterbe @MiLk

Mocks are auto generated, we do not format them.


Suggestion to improve snapping

https://github.com/android/compose-samples/blob/dcc400da3fb8e09b99b5fb21fced77c60e04393d/Jetcaster/app/src/main/java/com/example/jetcaster/util/Pager.kt#L111-L113
can we do this instead:
    _currentPageOffset.fling(velocity) { _, _, _ ->
        if (reason != AnimationEndReason.Interrupted) {
            _currentPageOffset.animateTo(currentPageOffset.roundToInt().toFloat()) { _, _ ->
                selectPage()
            }
        }
    }

So it doesn't just jump onto the next view after flinging?

I tried reproducing quickly the known behavior for the ViewPager, I ended up doing this
fun snap(velocity: Float) {
    val targetValue = if (abs(velocity) > 1.5f) {
        // High velocity, go to previous/next page if the velocity is in the same direction and traveled a minimum distance
        if (abs(currentPageOffset) > 0.15f && velocity * currentPageOffset > 0) {
            velocity.coerceIn(-1f, 1f)
        } else {
            0f
        }
    } else {
        // Small velocity, round from current position
        currentPageOffset.roundToInt().toFloat()
    }
    _currentPageOffset.animateTo(targetValue) { _, _ ->
        selectPage()
    }
}

I played a bit with the values to get what I wanted, it works fine until we get a proper Pager composable.
It does behave weirdly when you try to swipe too fast between pages though.


[WIP] New Compile Architecture

[x] Achieve the plug out of totum
[x] App should be compilable after totum plug out
[x] App should be loadable from parcer, serve or any other server and should open fine

This PR changes 617 files and it's not clear what it's doing, with no forward progress.
Closing this because it's not clear what this is or if it will ever merge.


feat: show warning if gas exceeds 500_000, refactor state

closes UMA-2168
closes UMA-2336
motivation
oSnap will only subsidize 500_000 gas and execute transaction automatically. We want to show users a warning if their transaction is exceeding this limit.

Note: This screenshot quotes the limit to be 50_000, but this is only to force showing this warning. The actual figured hardcoded is 500_000.

screenshots

If the tenderly project is not public, don't share link. (this will likely not happen since we control the project and are always interacting with the OG contract)

@Reinis-FRP can you verify that this gas amount returned from sim is the same gas the bot uses to threshold automatic execution

@Reinis-FRP can you verify that this gas amount returned from sim is the same gas the bot uses to threshold automatic execution

yes, 500K is default value used by execution bot


Add config flag for boto client defaults for s3 filestore

hey CCCS folks, this change is to allow the boto client to defer to the default credential load order which will be convienient for stuff like EKS Pod Identity
thank y'all for maintaining this awesome open source project!

Seems fine to me, but I'll merge it into a dev release and try it on our staging cluster but I don't think it should interfere with existing deployments once merged.

awesome, thank you!


[FIX] - Correction of total list count with pagination.

Can you add context to your PR description? that helps the maintainers determine whether they should merge it

Tests are failing. There is a specific test to avoid this change, as it has been attempted numerous times across the years, and always breaks some people's apps.
Take a look at the past attempts:
https://github.com/laravel/framework/pulls?q=is%3Apr+pagination+distinct


python3Packages.factory_boy: 3.2.0 -> 3.2.1

Motivation for this change
Update to latest upstream release 3.2.1
Change log: https://github.com/FactoryBoy/factory_boy/blob/master/docs/changelog.rst
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

@ofborg eval
@ofborg build


implement the "where have i finished?" feature

– remembering the last viewed entry (not only last edited, but last viewed)

for now at least last edited available, implemented in eb68acf8e63154561caacb9f5c97caf89d428dc9


fix: compiler errors when using avx512f

Fix some compiler errors when compiling on nightly w/ avx512f feature

closing -- I had the wrong version of lance going against wrong verison of nightly


Fix/914/incorrect path for converter example

Thank you for opening a pull request and contributing to AsciidoctorJ!
Please take a bit of time giving some details about your pull request:
Kind of change

[ ] Bug fix
[ ] New non-breaking feature
[ ] New breaking feature
[X] Documentation update
[ ] Build improvement

Description
What is the goal of this pull request?

Correct minnor error seen in converters examples

How does it achieve that?

Corrects path in the example label
Also removes some unnecessary linebreaks in some other examples. Given that most don't have them, I assume those linebreaks are not on purpose.

Are there any alternative ways to implement this?
Are there any implications of this pull request? Anything a user must know?
Issue
If this PR fixes an open issue, please add a line of the form:
Fixes #914
Release notes
Please add a corresponding entry to the file CHANGELOG.adoc

Don't think such a minor correction is worth mentioning.

Thank you!


Check that AdGuard works when you enable filtering of itself in the Apps Mgmt

// Note that some old Android custom firmwares might ignore addDisallowedApplication call
// That's why we check the Android version additionally
May be there are buggy firmwares exist even with android 7.0+ ?

Was checked on nexus 6p (android 8.1.0) and nexus 5 (android 6.0.1) in both cases AdGuard works well

IMHO that it is easier just not to emit socket protection for any firmwares and settings.
Or create the list of buggy firmwares that ignores adguard in disallowedApplications.


listen does not recover from server restart

I'm debugging an issue where an active listener is not re subscribed when the deepstream server restarts. I think I've nailed down the issue to here:
https://github.com/deepstreamIO/deepstream.io/blob/master/src/listen/listener-registry.js#L485
I believe it should check for === -1 instead of !== -1. Please correct me if I'm missing something?

Basically what I'm seeing is that the listener client does not receive a SUBSCRIPTION_FOR_PATTERN_FOUND. I have to restart it after restarting the deepstream server to make it live.

    if (subscribers && subscribers.indexOf(provider.socketWrapper) !== -1) {
      this._triggerNextProvider(subscriptionName)
      return
    }

What this is saying is that if the provider is also a subscriber then ignore it and ask another subscriber to provider the data.
I can see this being an issue if your only provider is already subscribed to the subject and sends both the subscribe and listen down at the same time.
Commenting out the check as opposed to changing the number results in all the tests passing, so I'm assuming its a edge case that we covered due to paranoia lol.
I would suggested a test that has a single provider for a subscription, gets a match, subscribes and then after server restarts gets a subscription found again. We haven't ran into this issue because we always had more than one listener running redundant, but I'm not too sure why it was there to begin with!

Yes. Removing it seems to resolve my issue. Will do more testing tomorrow.

Cool. I need to write some micro-benchmarks around listening, I'm not sure how it scales when it comes to thousands of connections.
Keep me posted on if you see any odd side effects!

This fix had an unfortunate regression which is that providers get told to subscribe immediately after discarding occasionally. We might need to put the line back in, write some tests to guard around it and find another solution.

@yasserf Any update on this?

Hey Ronag,
So I got a fix for this, but its not the.. sanest.
Basically the issue we have is when the server restarts we have no idea who is a subscriber and who is a publisher
The two solutions are:

Client listens to connection, when its lost unsubscribes to things it has provided.

Server finds a provider than can provide a subscriptionName, but the provider is already subscribed.. odd! So it unsubscribes the record on the provider on behalf of the client and then tells it to continue.

Problem is to unsubscribe a client on its behalf AND allow the client to continue working we need to sort of get that logic into the client too, so its all a bit messy.
Open for suggestions!

Just found out we can pretty much solve this by conditionally notifying records (which emits a discarded event) and force the issue only on events. Given events should NOT subscribe to the event ( its a one way emitter pattern ) this should work quite nicely.

Fixed in v3 when we dropped the support of getting a record to update and instead using setData


How do I use this?

Do I flash it with the USB port on the smartpad or do I need an adapter?

Hi @vektrix1,
sorry for my late reply. I by mistake did not activate watching this repository so I got no notification for your issue.
Unfortunately I did not understand how the stock bootloader works so you have to flash the HID bootloader mentioned in README with SWD interface on the board (CN2) first. After this it will be possible to flash via USB.
Feel free to ask if further questions come up.

Ah, thank you.


[Bug] Random crash

A user had a random crash, but I have no idea what was being done and how to reproduce, I got the error log.
You can just close this issue if there aren't enough to go on, just thought I would report it as it might be obvious for you what went wrong. :)
https://gist.github.com/olivernybroe/0b7f057cf30a49eba4a3035faee21b51

Got a random crash again, might be related?
https://gist.github.com/olivernybroe/c54110439899a1c105f8fa0feb823bfe

I found some places where local references should/could be deleted from memory sooner. Especially with the image loading this has an impact on the memory and can cause the crashes. Will fix it and commit as soon as possible.

Should be fixed in current 0.2.0 snapshot. Closing this now because unable to reproduce locally and plugin has been discontinued.


Investigate possibility of higher-level animation mechanisms

Currently you need to use the animationProgress and define the animation as a Modifier function, but theoretically there's also Transition and stuff.
I recall AnimatedVisibility not working well initially, but that was before I had the loop' goin'.
I wonder if it is as simple as allowing a custom composable to happen around the composable switch.
Alternately, the current animation progress could be exposed through a CompositionLocal so that composables can "initialize their animated state" through it.

This was actually provided in 0.5.0 although the composition local animation progress is tracked in https://github.com/Zhuinden/simple-stack-compose-integration/issues/11

Irrelevant because the animation APIs in Compose are experimental and probably will be for a while, this is technically already supported with the content wrapper. Closing


Pumps?

This isnt an issue, but I would love to see a pump or liquid extractor come in to this mod! Most of the big tech mods dont seem to have them (anymore...) Something like the ender-thermic Pump from extrautils back in 1.7.10.

I will think about it


implement mnemonic-based change password operation (shelley)

User facing changes

we implement passphrase reset via mnemonic for shelley only wallets

Issue Number
ADP-1658

bors r+

bors r-

bors r+


add width and height propertty picker

Hello, thanks for the input. I'm very excited to see the first pull request for this project. I would like to check if I understand correctly what this code does. Is it intended to resize the button (width and height) using predefined names like "sm", "lg", "2xl", etc.?

yes, change width and height with absolute size(16px,15rem,etc) or using predefined names(sm,xl,etc)

All is great, the only changes I think should be done are:

change width / ** height** attributes to button-width and button-height instead, because in the future I'm planning to add possibility to change popup width too with popup-width attribute.
and the same change with css variable: change --tool-cool-width-picker-span to --tool-cool-color-picker-btn-width, and --tool-cool-height-picker-span to --tool-cool-color-picker-btn-height

Sounds reasonable?

seem right.
add commint in pr, changing the display of elements to flexbox so that relative sizes can also be used and make it more attached to the page

Awesome. I'll review and approve the PR tomorrow as it's getting late here. Good night


Expose force option in SetTransformIdByIfName()

TwoStageL3ClosClient.SetTransformIdByIfName() always sets force = false today.
https://github.com/Juniper/apstra-go-sdk/blob/c876ae578e53e43d8549302e0ba2374494efb5f9/apstra/two_stage_l3_clos_interface_transformation.go#L31
terraform-provider-apstra #341 requests the ability to set the force flag.
At a minimum, we'll need to make that option accessible in SetTransformIdByIfName()

The linked provider issue has been closed without needing access to the Force attribute.
Looks like we don't need it.

The linked issue was closed without access to the Force attribute.
Looks like we don't need it for now.

The linked provider issue doesn't need this feature after all


Incorrect balance in placeholder field

Placeholder on the transaction page shows incorrect balance in the amount field:

Solved.


DOC: Update donation link to Open Collective

NumFOCUS has moved to Open Collective to handle donations / accounts.

ITK.macOS.Python failure is unrelated. Merging.


Reordering related entities in the submission

Internal PR to visualize the differences between PR 534 and PR 531

@benbosman : This diff PR doesn't look correct to me.  Could you double check it?  It looks to include a LOT of unrelated changes to the collection-page, community-page and homepage components/modules that don't look to have anything to do with the Submission process.
It may be that it just needs a rebase (as there are a lot of code conflicts listed below)

@tdonohue It seems GIT sometimes has problems updating the list of differences if the base branch gains additional commits
I've updated this PR (based on https://stackoverflow.com/a/46782679) to force GIT to update the commit list (which has drastically reduced the amount of commits displayed here), but the commit https://github.com/atmire/dspace-angular/pull/2/commits/dc316dc6cbad86ed95d4ffe26410c0c1b2aad5af still causes this large amount of files to be displayed.
I'm not sure whether we can create a better diff until https://github.com/DSpace/dspace-angular/pull/531 is merged (which seems to be almost ready now, Lotte was going to perform a few more tests tomorrow)


v4.0 release

Wait for JavaScriptEngineSwitcher 3.0 to be released as a stable version (currently just an RC)
Ensure all JavaScriptEngineSwitcher references are updated to 3.0 stable
Write a blog post with all the changes

cc @dustinsoftware

Yep, I also need to finish updating the docs for the CSS in JS support (and
add some for react router)
On Sun, Oct 14, 2018 at 12:15, Daniel Lo Nigro<EMAIL_ADDRESS>wrote:

Wait for JavaScriptEngineSwitcher 3.0 to be released as a stable
version (currently just an RC)
Write a blog post with all the changes

cc @dustinsoftware https://github.com/dustinsoftware
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/reactjs/React.NET/issues/609, or mute the thread
https://github.com/notifications/unsubscribe-auth/AA5hFtXdTdsazX1IrNMhc0KWebF3hpAfks5uk404gaJpZM4XbSG7
.

And it's released! :)

Awesome! I'm currently in vacation until next week, so I'll look into this next week, unless @dustinsoftware gets to it first. I want to release a new version of JsPool too.

I'm unsure if I'll get to it either, but here's the major changes to cover in blog post:

CSS-in-JS supported. Docs
React Helmet supported. Docs
Lots of fixes
MIT licensed
React 16.7.0 bundled
Updated to JavaScriptEngineSwitcher 3.0, including script precompilation. JS engines no longer automatically registered. Getting started pages for NetFx and NetCore updated to reflect this change.


Support for UInt Model Variable Identifiers

... some FMUs are using model varibale identfiers larger than<PHONE_NUMBER>, which results in failures during parsing the model description. This will be solved in the next update.

Solved in commit ba33121


using shuffled cards loses entropy

A recent reddit post caused me to notice how you're generating entropy from cards, and how it is throwing away some entropy.
The last 12 or so cards in a full 52 card shuffled deck have no effect on the generated seed because of this code:
// Calculate the new value by scaling the original value down
var withoutReplace = entropyInt.multiply(maxWithoutReplace).divide(maxWithReplace);

The entropyInt is made by spacing all the individual cards out equally (log(52) bits per card), but the 'scaling' will effectively chop off the end of the string, allowing only the first N cards to have any effect on the outcome. Rather than scaling like this, where only the leading bits affect the outcome, I would use an SHA-2 hash of all the bits and take the first N bits of the result, so that all the input bits affect the outcome.

Thanks for this.
To better understand how the raw entropy is expected to work, I made a table of how the bits should be distributed as cards are consumed. It shows with 0-3 remaining cards there should be a 21 word mnemonic, with 4-14 remaining cards there should be an 18 word mnemonic. Initially I thought this would account for the behavior because of the small amount of total entropy added by the last 12 cards, but it's clear there's still an error with the 'scaling' process as currently implemented.
I'm going to fix the scaling process, but using cards + raw entropy will still have a 'thinning' of bits at the end of the deck.
I concede that using the raw entropy to generate mnemonics often gives unexpected results for users, and possibly shouldn't be the default selection. But having it available is good for compatibility with bip32jp.github.io, and I think having alternative interoperable implementations is important for the ecosystem.
Changes to make:

Fix scaling of entropy from cards
Change default Mnemonic Length value to 15 words

Cards Remaining
Bits Contributed
Bits Accumulated
Bits Remaining
Words

52
5.70
5.70
219.88
0

51
5.67
11.37
214.21
0

50
5.64
17.02
208.56
0

49
5.61
22.63
202.95
0

48
5.58
28.22
197.36
0

47
5.55
33.77
191.81
3

46
5.52
39.29
186.29
3

45
5.49
44.79
180.79
3

44
5.46
50.25
175.34
3

43
5.43
55.67
169.91
3

42
5.39
61.06
164.52
3

41
5.36
66.42
159.16
6

40
5.32
71.74
153.84
6

39
5.29
77.03
148.55
6

38
5.25
82.28
143.30
6

37
5.21
87.49
138.09
6

36
5.17
92.66
132.92
6

35
5.13
97.79
127.80
9

34
5.09
102.87
122.71
9

33
5.04
107.92
117.66
9

32
5.00
112.92
112.66
9

31
4.95
117.87
107.71
9

30
4.91
122.78
102.80
9

29
4.86
127.64
97.94
9

28
4.81
132.44
93.14
12

27
4.75
137.20
88.38
12

26
4.70
141.90
83.68
12

25
4.64
146.54
79.04
12

24
4.58
151.13
74.45
12

23
4.52
155.65
69.93
12

22
4.46
160.11
65.47
15

21
4.39
164.50
61.08
15

20
4.32
168.83
56.76
15

19
4.25
173.07
52.51
15

18
4.17
177.24
48.34
15

17
4.09
181.33
44.25
15

16
4.00
185.33
40.25
15

15
3.91
189.24
36.34
15

14
3.81
193.05
32.54
18

13
3.70
196.75
28.84
18

12
3.58
200.33
25.25
18

11
3.46
203.79
21.79
18

10
3.32
207.11
18.47
18

9
3.17
210.28
15.30
18

8
3.00
213.28
12.30
18

7
2.81
216.09
9.49
18

6
2.58
218.67
6.91
18

5
2.32
221.00
4.58
18

4
2.00
223.00
2.58
18

3
1.58
224.58
1.00
21

2
1.00
225.58
0.00
21

1
0.00
225.58
0.00
21

See

https://github.com/iancoleman/bip39/commit/886f06ee6bbca2e6e05109dbd7e264bdea35238f - Card entropy calculation bugfix
https://github.com/iancoleman/bip39/commit/55b592b498a57465b91bd8402d0d91324cc34cf9 - Entropy defaults to generating a 15 word mnemonic

The fix was to scale each event individually, rather than concatenate them and then scale. So each event contributes the correct number of bits as indicated in the table above, rather than all contributing 5.7 bits and then being scaled uniformly.

I've not had a chance to fully understand your changes yet, but I'm hoping you can clarify a couple of things:

Are you assuming that the total number of decks I have shuffled is ceiling(cards/52)? Because I could shuffle 8 decks together and only pick 52 cards, but you would set totalDecks to 1 in that case:
var totalDecks = Math.ceil(base.parts.length / 52);

What is remainingDecks for? It seems to me like it will always be zero:
var remainingDecks = Math.floor(totalRemainingCards / 52);

Total decks is calculated incorrectly, and would be better calculated by looking for max(duplicates). There's currently a small discrepancy between calculating total number of bits and checking how much entropy each individual event adds.
A good test would be to use AS AS AS AS; by the first method it would calculate log2(52!/48!) = 22.63 bits of entropy; by the second method it would calculate 4*log2(52) = 22.8 bits of entropy. This discrepancy is rarely noticed because of flooring, but may create issues so I will fix it. Use of a full deck (or using any multiple full decks) will not have this discrepancy.
I'll fix this it to correctly tally the number of bits on a per event basis.
I'm still unclear about one aspect: shuffling two decks together should have slightly more entropy than shuffling one deck twice in a row. But my code doesn't account for that. Any ideas how this might work? My probability theory isn't especially strong.

remainingDecks is used to calculate remainingCombos, which ultimately determines the numberOfBits. If there are 2 decks in use (104 cards) and only 5 cards have been removed, there is still one full deck and 47 cards remaining, hence the remainingDecks variable.
But there is an error here, since if the first two cards are AS and AS there are only 51 cards remaining from each deck. 51!*51! combos remain (439.76 bits), not 52!*50! (439.78 bits). Once again it's a tiny error which is almost never visible due to flooring, but should be corrected.
I need to change the calculation of numberOfBits to work better with multiple decks.
Note that this test for 2 decks passes - it has 451 bits, ie 2*log2(52!).

As an aside, a single deck provides more than enough entropy, and just using a single deck would greatly simplify the code. However, I plan to make the entropy library standalone so would like it to be as generic as possible for applications that may require more entropy than a single deck.

It doesn't feel right to see you looking at the cards in turn to decide how many bits of entropy it provides. It seems to me that picking 10 cards from 4 decks shuffled together should give a certain amount of entropy, regardless of what those 10 cards happen to be, but you're looking at the cards and somehow deciding that the first 6 came from the first deck and the 2nd 6 came from the 2nd deck. I don't think that's a valid assumption to make.
I can't find a good way of calculating the amount of entropy when drawing N cards from M decks. I don't think you even know how many decks I'm drawing from. Just because max(dups) is 2 doesn't mean I didn't use 4 decks. I just didn't get more than 2 dups from those 4 decks. In fact I might even be drawing with replacement from a single deck, shuffling it after every draw. This would be the same as using an infinite number of decks shuffled together.
In general I think you're trying to solve a hard problem that doesn't really need solving. All this appears to be motivated by the requirement of compatibility with bip32jp.github.io, but it looks to me like that site doesn't even allow you to enter cards. So let's assume max(dups) decks, and conservatively assume 52! permutations for each of the (length/52) decks, and (length%52)! permutations for the leftovers. The binary representation can be the first log(permutations,2) bits of the sha-2 hash of the normalized deck string. Then you have a 1-1 correspondence between the binary representation and the mnemonic, compatible with bip32jp.
Am I missing something?

To be clearer:
You are using linear scaling to convert an M bit number which you believe to have only N bits of entropy into an N bit number, but the entropy isn't all at the left hand edge of the M bit number, so you are losing some of the entropy. Your solution is to try to understand more clearly how the entropy is distributed, and to use that understand to improve your linear scaling algorithm.
I propose that using first_N_bits(SHA-2(M bit number)) instead of linear scaling will do what you're looking to do without loss of entropy, and without you having to know how the entropy is distributed throughout the M bit number.
You can then use that N bit number in just the same way as bip32jp uses it, to maintain compatibility.

It doesn't feel right to see you looking at the cards in turn to decide how many bits of entropy it provides.

Agreed. To rephrase it, a card sitting on the top of the deck provides a specific number of bits of entropy before being turned over. I agree with this, but don't know how to calculate that value. With a single deck it's easy to calculate, but with multiple decks, I'm unsure.

I think you're trying to solve a hard problem that doesn't really need solving. All this appears to be motivated by the requirement of compatibility with bip32jp.github.io

Yeah, I think this has gone too far down the rabbit-hole for what it's worth. Better to have a naive-but-clear solution than a complex-but-still-incorrect solution.
If anyone in the future with a better understanding of stats / probability can illustrate how to make this work as desired, then I'll implement it. But for now I'll make it good-enough with clearly separated concerns in case of future improvements.
Changes to make:

Amount of entropy added should not depend on the specific card drawn
totalDecks calculation should use max(dupes)
Hash the supplied card values and take the first N bits according to log2(permutations)
Structural changes to facilitate simpler upgrades in the future

Appreciate the clarity you've bought to this topic.

I've asked a question at https://crypto.stackexchange.com/questions/41886/entropy-measurement-from-shuffled-cards

I can't see any way to improve this feature. The current implementation using 'average entropy' is suitable and is based on the best information currently available.
I'll close this for now since it a) works and b) has no clear direction for potential improvement. Always happy to have it reopened if more information comes to light.
Great feature too, @dooglus; I'm very glad to have it available in the tool.

I didn't receive notification of your commit (87ad2c6) which uses hashing instead of scaling to convert the cards to binary, and only just noticed the commit now. I'm happy to see you implemented my suggestion.
This comment and corresponding code concerns me:
    // If the number of bits is more than 256, multiple rounds of hashing
    // are used until the required number of bits is reached.

        for (var j=0; j<iterations; j++) {
            hashedCards = sjcl.hash.sha256.hash(hashedCards);
        }

You don't add any extra entropy by using multiple rounds of hashing. There are 2^256 possible outputs from sha256. If I need 512 bits of entropy, I can't take a text string, hash it once, create a 256 binary digit string, then hash it twice, append another 256 binary digit string, and expect that the output has 512 bits of entropy. While it's true that I'll have a 512 digit binary string, there are at most 2^256 different 512 digit binary strings I could end up with, and so the 512 digit binary string has at most 256 bits of entropy.
In other words, suppose normalizedCards has 512 bits of entropy.
Then sha256(normalizedCards) has 256 bits of entropy,
and sha256(sha256(normalizedCards)) has the same 256 bits of entropy.
Concatenating those two outputs gives a string with 256 bits of entropy.
A solution is to use sha256(normalizedCards + ':0') and sha256(normalizedCards + ':1') (where + is string concatenation), since concatenating those two gives you an output with 512 bits of entropy.
Consider that once I know just sha256(normalizedCards) and don't know normalizedCards, I can trivially compute sha256(sha256(normalizedCards)). But if I know just sha256(normalizedCards + ':0') and don't know normalizedCards, then I have no idea of the value of sha256(normalizedCards + ':1').

Maybe it's clearer in code:
diff --git a/src/js/entropy.js b/src/js/entropy.js
index 24fc21a..918f567 100644
--- a/src/js/entropy.js
+++ b/src/js/entropy.js
@@ -293,15 +293,12 @@ window.Entropy = new (function() {
         // Create a normalized string of the selected cards
         var normalizedCards = cards.join("").toUpperCase();
         // Convert to binary using the SHA256 hash of the normalized cards.
-        // If the number of bits is more than 256, multiple rounds of hashing
+        // If the number of bits is more than 256, multiple hashes
         // are used until the required number of bits is reached.
         var entropyBin = "";
         var iterations = 0;
         while (entropyBin.length < numberOfBits) {
-            var hashedCards = sjcl.hash.sha256.hash(normalizedCards);
-            for (var j=0; j<iterations; j++) {
-                hashedCards = sjcl.hash.sha256.hash(hashedCards);
-            }
+            var hashedCards = sjcl.hash.sha256.hash(normalizedCards + ':' + iterations);
             var hashHex = sjcl.codec.hex.fromBits(hashedCards);
             for (var i=0; i<hashHex.length; i++) {
                 var decimal = parseInt(hashHex[i], 16);

Also, I wonder if it's worth pointing out on the page that the algorithm used to create the seed from the supplied entropy isn't yet set in stone.
I can imagine someone using a deck of cards as their backup, "knowing" that your tool will always be able to regenerate their BIP39 seed words from the order of the deck if they need it. Changing how the supplied entropy is converted to a BIP39 seed will break his assumption.

Great suggestions.

I can imagine someone using a deck of cards as their backup

I see your comment in How can I encode my BIP39 seed into the order of cards of a deck and agree it's worth noting that the entropy feature is under development and may change.

Fix hashing iterations
Add warning to the ui about possible changes to the entropy feature

See

https://github.com/iancoleman/bip39/commit/9d33c8925dc07783e2cd819c91eee1144598fcf2 - Card entropy uses secure hashing when > 256 bits
https://github.com/iancoleman/bip39/commit/bf9bdfd20ff1997dc5c53ba5a25f6441b9edd3b7 - Note that entropy must not be stored

I figured there is already a strong warning above the entropy field, and adding more text there was not as helpful as a detailed explanation with the rest of the entropy details.
Since there are many reasons not to store entropy (algorithmic changes during development, lack of standardization, interoperability, security etc) I don't explicitly warn of the reasons.

This is an interesting android app using cards to generate addresses - deckwallet
Source code for MainActivity.java
Doesn't seem to be BIP32 compatible so this tool will probably never be compatible with deckwallet.
See getAddress which uses getSeed which uses generateBrainWifKey

@dooglus, are you content for this issue to be closed now? I think the feature is well designed and robust thanks to the conversation that came from these issues.