Enabling trim mode breaks program at runtime (v2 only)

Describe the bug
Enabling trim mode when publishing a project using Terminal.Gui will throw this error when run. Disabling trim mode or reverting to v1 resolves this issue:
Unhandled exception. System.InvalidOperationException: Reflection-based serialization has been disabled for this application. Either use the source generator APIs or explicitly configure the 'JsonSerializerOptions.TypeInfoResolver' property.
   at System.Text.Json.ThrowHelper.ThrowInvalidOperationException_JsonSerializerIsReflectionDisabled()
   at System.Text.Json.JsonSerializerOptions.ConfigureForJsonSerializer()
   at System.Text.Json.JsonSerializerOptions.MakeReadOnly(Boolean)
   at System.Text.Json.JsonSerializer.GetTypeInfo(JsonSerializerOptions, Type)
   at System.Text.Json.JsonSerializer.GetTypeInfo[T](JsonSerializerOptions)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](Stream , JsonSerializerOptions )
   at Terminal.Gui.SettingsScope.Update(Stream, String)
   at Terminal.Gui.SettingsScope.UpdateFromResource(Assembly, String)
   at Terminal.Gui.ConfigurationManager.Reset()
   at Terminal.Gui.ConfigurationManager.get_Settings()
   at Terminal.Gui.ConfigurationManager.Load(Boolean )
   at Terminal.Gui.Application.InternalInit(ConsoleDriver , String , Boolean )
   at Terminal.Gui.Application.Init(ConsoleDriver , String )
   at Program.<Main>$(String[]) in C:\Users\Admin\Workshop\FFBatchConverter\FFBatchConverter\Source\Program.cs:line 4

To Reproduce
Steps to reproduce the behavior:

Create a sample project, Something like this will suffice:

Application.Init();

try
{
	Application.Run(new Toplevel());
}
finally
{
	Application.Shutdown();
}

Publish the project with trim enabled.

Expected behavior
The program should work even with trim enabled, as it did for v1.
Screenshots

Desktop (please complete the following information):

OS: Windows 10
Browser N/A
Version 23H2, OS build 22631.37374

Additional context
This link recommends setting this flag in the .csproj file:
  <JsonSerializerIsReflectionEnabledByDefault>true</JsonSerializerIsReflectionEnabledByDefault>

as a workaround. This does not work, as it will give this error message at runtime:
Terminal.Gui ConfigurationManager encountered the following errors while deserializing configuration files:
Error deserializing resource://[Terminal.Gui]/Terminal.Gui.Resources.config.json: Unknown property name "$schema".

You would need to disable all reflection-based serializers to be able to trim.
However, we do know, right now, that V2 is not yet trim-friendly. It will be, before release.

Hello @dodexahedron! That's good to hear! Is "disabling all reflection-based serializers" something one could reasonably do now (that is, it's exposed via an API) or will it require digging through the guts of the project?
And regarding resolving this, is the plan to move more towards .NET's source generators, or is it headed another direction?

I don't think so - at least not in a way that will make TG work with trim outside of non-View types, which of course isn't all that helpful.
There are other non-serializer uses of reflection-related functionality, so it's just not really anything we can give much support to at all, until we get to addressing reflection and trim compatibility, specifically.

Now... If you did want to dig around and fix it all, we certainly wouldn't complain. But I suspect it's going to be a non-trivial task.

However, I suspect some of the PInvoke that is still the DllImport form will break it anyway, because the marshaling for that is not trim-compatible.

This has already been discussed in #3544. It's worth reading to find out what I've already discovered. All that remains is to find an alternative to resolve and read the configuration files. I also advise using trim on all operating systems, especially win-x64, linux-x64 and osx-x64, because I also found other problems with win-x64 related to the WriteConsole method.

With osx-x64:

Alright, thanks! Might look into it. I suspect there's no concrete answer yet, but do you know when, roughly, Terminal.Gui v2 will be released officially? Would it be in 2025 or 2026?

Alright, thanks! Might look into it. I suspect there's no concrete answer yet, but do you know when, roughly, Terminal.Gui v2 will be released officially? Would it be in 2025 or 2026?

Unfortunately I'm not able to debug the self-container after the publish to find out what's going on. With the Debug configuration the ConfigurationManager works well. With the Release configuration and publishing the self-container it doesn't work well and I can't even debug it to discover the cause. It seems that there is a bug in the runtime that is not allowing debugging of the release version.
I have no idea when the official version will be released but I know that a beta version will be released before then.

There are still some usages of things like GetType(), which will cause issues too.
RuntimeType, which is what that actually returns (just up-cast to Type) doesn't exist at runtime in an AoT assembly. It works in non-AoT trimmed assemblies - just not AoT.
That's one of the nitpicky things we have to be careful about, around Trim-compatibility: What works in Self-Contained, Framework-Dependent, and AoT are mostly overlapping but not identical circles on a venn diagram.

Alright, thanks! Might look into it. I suspect there's no concrete answer yet, but do you know when, roughly, Terminal.Gui v2 will be released officially? Would it be in 2025 or 2026?

Unfortunately I'm not able to debug the self-container after the publish to find out what's going on. With the Debug configuration the ConfigurationManager works well. With the Release configuration and publishing the self-container it doesn't work well and I can't even debug it to discover the cause. It seems that there is a bug in the runtime that is not allowing debugging of the release version. I have no idea when the official version will be released but I know that a beta version will be released before then.

Did you explicitly tell the Debug config to publish trimmed, too? IIRC, the default if not explicitly specified is for Debug to not trim[^1], but don't quote me on that.
[^1]: The reason being that the debug symbols themselves would also be removed, if not configured to not do that, because they're "unreferenced" code.

@dodexahedron if you create a Self-Contained project in Terminal.Gui and manage to generate a single file that can be debugged please let me know. I can only generate a single file with Publish but unfortunately I can't debug it to find out how to get around the fact that the config file settings are not being loaded and if I could debug maybe I could find an alternative to read this configuration. In the Debug and Release configuration I can debug but the config file is being properly read. With the single file the config file is not being properly loaded.
From the error below you can see that it is the Sequence field that is not being properly loaded with the config file configuration, but without being able to debug it is difficult to find a solution.
4>ILLink(0,0): Error IL1012: IL Trimmer has encountered an unexpected error. Please report the issue at https://aka.ms/report-illink
4>Fatal error in IL Linker
4>Unhandled exception. System.InvalidOperationException: Sequence contains no elements
4> at System.Linq.ThrowHelper.ThrowNoElementsException()
4> at System.Linq.Enumerable.Last[TSource](IEnumerable`1 source)
4> at Mono.Linker.MessageOrigin.ToString()
4> at Mono.Linker.MessageContainer.ToMSBuildString()
4> at Mono.Linker.MessageContainer.ToString()

There's more than just the serialization mode and whatnot that would need to be done for CM to work without reflection dependencies.
But even if you made CM fully trim-safe, any other code path in the library that requires use of unreferenced code will still cause a crash until all of them are addressed.
I don't really think it's worth your time to continue trying to debug that, right now, especially since I'll be addressing many of the core problems with it after some other in-progress work is done that I'm getting back to actually working on, today. I've decided to bump my priority for addressing trim problems in light of the recent uptick in people wanting to avail themselves of that, so I might even start on that next, after the build-related stuff.
As a note, though, there are also built-in analyzers that can help to catch some of the trim problems, where they fit the heuristics those analyzers use. I'll turn those on in the process of addressing that stuff and put it in a PR, early on, so everyone will at least get warnings if they add something that trips one of the rules.

I've already managed to attach Self-contained to the debugger using win-x64. However, I cannot attach using linux-x64 and osx-x64 to the debugger, as it gives the following error. If anyone knows how to solve it, I'd be grateful in advance.

This is the right place to state this situation.

I've already managed to attach Self-contained to the debugger using win-x64. However, I cannot attach using linux-x64 and osx-x64 to the debugger, as it gives the following error. If anyone knows how to solve it, I'd be grateful in advance.

This is the right place to state this situation.

I reported this issue on Visual Studio Developer Community (https://developercommunity.visualstudio.com/t/Failed-to-attach-to-process:-Unknown-Err/10694351).

That sounds like something I vaguely remember seeing documented actually. But I am not sure and I'm gonna be busy reviewing #3571 and working on finishing up splitting the analyzers off, so probably not gonna try to hunt that up. I might be thinking of something else anyway. 🤷♂


First branch

What is this spam? Why do I keep getting these? Who are you bots and what is your goal?

It’s ok if this was an accident, but I seem to be getting lots of very similar PRs from lots of different users, so I assumed bots were involved. Were you directed here from another website?


Insecure Randomness

Can we implement the more secure implementation?
Current Insecure   implementation
Math.random
More  secure implementation
/* Assume that window.cryto.getRandomValues is available   */     function secureMathRandom() {       return   window.crypto.getRandomValues(new Uint32Array(1))[0] /<PHONE_NUMBER>;     }
Reference   URL
https://developer.mozilla.org/en-US/docs/Web/API/Crypto/getRandomValues
Current Insecure   implementation
Random rnd = new Random();
More  secure implementation
rnd.setSeed(seed); | SecureRandom   random = new SecureRandom();           byte bytes[] = new   byte[20];           random.nextBytes(bytes) |
Reference   URL
https://docs.oracle.com/javase/7/docs/api/java/security/SecureRandom.html

It would be great if you could fill out the Bug template, which will provide us the information we need to address your concern.
Could you elaborate on what this is in reference to? Please also link JavaScript documentation since this is a JS repo. Thank you!

We included a library called math-random that do secure random if the browser support P/RNG.
The only production code which use random() is packages/core/src/utils/uniqueID.js. And we already secured it using math-random.
We have insecure random but they are used in sample (19 and 22) and test harness. So they do not impose any security risks on Web Chat itself.
Please let us know by opening a new issue if you observe any of our code is using insecure random.
Thanks!


Multiple Handler configuration sections

The following is really a request to get feedback on an idea I have, before I begin any work implementing it.  This all depends on #55 being accepted, i.e allowing handlers to accept command-line options.
I've been thinking about how to allow handlers to have multiple configuration sections defined in the config json.  This would be useful for handlers like mailer or xmpp, where you might want to have different handlers alerting different users, but not want to have to modify the handler code each time.
I'd like to propose a command-line option, set in sensu-plugin, which is then inherited by handlers - lets call this 'jsonconfig' for now (name isn't important). It's default value would be the name of the handler.  It can then be overridden to point to a differently named section when the handler is used, e.g:
/etc/sensu/handlers/notification/mailer.rb --jsonconfig mailer-devteam

'mailer-devteam' in the json config would contain email configuration options for the development team, while 'mailer' (the default) would email the production team.
It would be easy to change handlers on a case-by-case basis, switching them from having hard-coded config paths to using this variable.
For example, mailer.rb currently has:
 mail_to = settings['mailer']['mail_to']

This would change to something like:
mail_to = settings[config[:jsonconfig]]['mail_to']

I'm happy to write the code for this, and also to update any documentation.  I'd also be making changes to at least some of the community handlers to support this for my own use, mailer and xmpp, but perhaps others if the changes are minimal.
Thoughts, comments, input etc appreciated!

I would like to propose a new idea to this problem.
Instead of using a command-line option to find the hander configuration, the hander definition could be passed through STDIN together with the event data. This would require changing sensu-server.
By handler configuration I mean settings[:<handler name>] and by hander definition I mean settings[:handlers][:<handler name>].
The handler name could then be used to locate the handler configuration, or the handler options could be within the handler definition.
Thoughts?

That PR has been merged, should we re-open this up for discussion?


Demo - bug - runtime error

After successful or blocked transactions, click on new debitor, click send without clicking on new transactions
This doesn't happen if you click on send the first time when a new debtor is created, only subsequent

Managed to re-create this issue.
Applying error handling fix.

Fix will be in v1.0.12

Tested and confirmed fixed


Support more than one rejected responses

Hi,
I'd like to know if DPOTrainer supports more than one rejected sample. As the original DPO paper mentioned, they can support Plackett-Luce model for a set of possible responses.

the loss funtion implementation in DPOTrainer is based on logsigmoid, so I think it doesn't support P-L model

Hi, I'd like to know if DPOTrainer supports more than one rejected sample. As the original DPO paper mentioned, they can support Plackett-Luce model for a set of possible responses.

@wyh2000 the loss funtion implementation in DPOTrainer is based on logsigmoid, so I think it doesn't support P-L model

Is there a plan to include this generalization? This could be helpful in many different settings

That's an interesting point. Do you have examples of datasets with several rejected answers?
I wonder if it is possible to pre-process such a dataset to go from
chosen A, [rejected A1, rejected A2]

to
chosen A, rejected A1
chosen A, rejected A2

and obtain similar results.

Closing because this feature hasn't received any PR, and doesn't seem to be widely requested by the community. This issue may be reopened if that changes.


$scope - updated but carousel appears as a list

Hi,
I have my slick carousel looking like:
<slick id="regionCaro" init-onload=true data="questionsMost" dots=true arrows=false
      autoplay=true speed=500 autoplay-speed=4000>

However  - when the data questionsMost scope variable gets updated, the items in the carousel appear as a list on my page, not as an updated carousel.
So the error in the console i get is slider.unslick is not a function and on reading here I tried using slider.slick("unslick"); which gets rid of the message however it doesn't show the correct/update data.
Any ideas what I'm missing or is this a issue?
Thanks.

Hi,
I have the same error. Do you find any solution? :)

Hi,
I actually stopped using a carousel on my site - however from what I very briefly remember it was related to some dependency or the angular-slick library that I was using being the wrong version.
I just tried a few different combinations and one of them worked.


Improve serialization for notes in section page

Who is this PR for?
students, HS educators
What problem does this PR fix?
Improves the serialization so that restricted notes aren't serialized.

selfie


Update Kitura-CredentialsFacebook,Kitura-CredentialsGoogle, and Kitura-Credentials-Sample to the latest versions.

Additionally fix issues on Linux

@irar2 Please validate

Reviewed.


How good is it with the session management?

Let's say I logged into a site using Scrapy and then the spider crawls its way into a page where I  need Selenium for rendering, Would I still have to log into the site yet again to complete the request successfully? OR will it share Cookies and session information with the browser?

hi @gerosalesc,
First of all, sorry for the really late answer.
About your question, you can look at the code here: https://github.com/clemfromspace/scrapy-selenium/blob/develop/scrapy_selenium/middlewares.py#L80
It's really basic, the middleware just pass along the cookies names/values from Scrapy to Selenium.


Technical Documentation Improvments

Problems to solve for:
Examples:

Kyle from CZ Biohub: He really likes the MUI has coded examples and understanding what is leading to a component rendering. Component recipes could be very helpful here and enable engineers to copy and paste, play with the code, and then run it in their own local.
Sydney: Keeping the API for components the same as MUI, especially when it comes to data input formats, would alleviate a lot of headaches. Those APIs have been developed and tested by hundreds of people, and a lot of devs on our team already know them well.
If it is necessary to change the API, please please please provide full working code vignettes for each use case, instead of just . Showing the full import path, how the different components are meant to fit together, etc., in a block that I can copy and paste like in the MUI docs would be a HUGE HUGE HUGE speedup.

What to do:
Right now there is a delay between when the code is updated and zeroheight is updated. Should we create a release checklist to have updated zeroheight links?

[ ] #407
[ ] #409
[ ] #408
[ ] #411
[ ] #410

Must have to launch:

if we deviate from MUI API design make sure there is a statement that it is deviated and why

Must have to have finished documentation:

Feedback on things to think about, I know we are covering these but adding this here in case there are things to add on:

[Lower lift, big impact] - An API reference for each component that lists out which props are available and what they do, along with which CSS classes are available and what they do. These are most important for complex components.
[Bigger lift, huuuge impact] - “Full source” code examples of each component variation in use, such as the attached image from MUI.
One other piece of feedback that might be specific to me as I’m a newbie — I wanted to update the docs for the dropdown, but am confused as to how the “storybook” thing works and didn’t have time to figure it out. It’s also what’s blocking me from putting in a PR with the PanelBasic component. A lower lift docs option might make this easier? I don’t care much about being able to “configure” the component on the docs page, I just care that all the options are clearly listed out. (edited)

Thanks for logging this, @cathystoli ! One other note -- I care a lot less about having a code sandbox than just having a complete plain text example and a clearly defined API. I never use the sandbox option for MUI, semantic UI, etc. I just need the reference.


USHIFT-510: Clean up references to removed auditLogDir option.

Mop-up for https://github.com/openshift/microshift/pull/1051.

/hold

/lgtm

/hold cancel


Enteroendocrine cell subclasses

Make subclasses for enteroendocrine cells based on peptide expression maps
Look up Veenstra papers for larva (and FBrf0211737 for DH31)
Park + Kwon for adult

Plan of action for this issue:
For the adult gut
Based on the consensus that EE cells fall into two major classes: class I cells expressing Allatostatin C and class II cells expressing Tachykinin, with other hormones being expressed dependent on the localization within the midgut (FBrf0226048, FBrf0232200, FBrf0244352).

Define a adult class I enteroendocrine cell as equivalent to adult enteroendocrine cell that expresses AstC;
Rename Tachykinin cell of the adult midgut (FBbt:00110036; that term is not used anywhere) into adult class II enteroendocrine cell for consistency and define it as equivalent to adult enteroendocrine cell that expresses Tachykinin.
For each class, define three sublasses, one for each major region of the midgut (anterior, middle, posterior).

We ultimately end up with the following hierarchy:

adult enteroendocrine cell

adult class I enteroendocrine cell

adult anterior midgut class I enteroendocrine cell (:= adult class I enteroendocrine cell, part of adult anterior midgut, and expressing Orcokinin);
adult middle midgut class I enteroendocrine cell (:= adult class I enteroendocrine cell, part of adult midgut region R3, and expressing Orcokinin);
adult posterior midgut class I enteroendocrine cell (:= adult class I enteroendocrine cell, part of adult posterior midgut, and expressing Mip and AstA);

adult class II enteroendocrine cell

adult anterior midgut class II enteroendocrine cell (:= adult class II enteroendocrine cell, part of adult anterior midgut, and expressing NPF and Dh31);
adult middle midgut class II enteroendocrine cell (:= adult class II enteroendocrine cell, part of adult midgut region R3, and expressing NPF and Mip);
adult posterior midgut class II enteroendocrine cell (:= adult class II enteroendocrine cell, part of adult posterior midgut, and expressing NPF and Dh31).

For the larval gut
Remains to be determined, but we may not be able to define precise subtypes of larval enteroendocrine cell as proposed above for the adult gut, based on the available data.

looks good, best to keep the logical definitions (EquivalentTo) minimal, so for "adult anterior midgut class II enteroendocrine cell" you will have:
EquivalentTo:
'adult class II enteroendocrine cell' and 'part of' some 'adult anterior midgut'
SubClassOf:
expresses some NPF
expresses some Dh31


[bug] useAppKitProvider returning walletProvider as undefined when a solana wallet is connected

Link to minimal reproducible example
https://github.com/DaveTheFrontEndDev/reown-solana-transaction-bug
Summary
Scenario
Using the React reown libraries in a simple vite setup, implementing the wagmi+solana multichain setup
Documented in the example here
Expected Behavior
useAppKitProvider hook should return a walletProvider value as documented here (smart contract interaction section). This example expect the walletProveder to contain a publicKey property.
What is Actually Happening
walletProvider is being returned as undefined despite a solana wallet being connected.
Producing the Bug
Simple Vite React app example
Bug 1 in repo

Copy .env.example into new file .env and replace with project id from here

Run code example:

yarn install && yarn dev

Example preview
https://github.com/user-attachments/assets/525c0a5a-3ee5-42d7-8d4a-d3154893d339
List of related npm package versions
@reown/appkit: ^1.0.6
@reown/appkit-adapter-solana: ^1.0.7
@reown/appkit-adapter-wagmi: ^1.0.6
@reown/appkit-solana: ^1.0.6
@solana/wallet-adapter-wallets: ^0.19.32
@solana/web3.js: ^1.95.3
@tanstack/react-query: ^5.59.2
viem: ^2.21.15
react: ^18.2.0
wagmi: ^2.12.16

Please check this example -> https://github.com/reown-com/web-examples/blob/feat/create-appkit-examples/dapps/appkit/react-solana/src/components/hooks.tsx

Please check this example -> https://github.com/reown-com/web-examples/blob/feat/create-appkit-examples/dapps/appkit/react-solana/src/components/hooks.tsx

This is now a dead link, but was able to find the example here https://github.com/reown-com/web-examples/blob/main/dapps/appkit/react-solana/src/components/hooks.tsx and it worked now


WIP pagination refactor and datagrid

Codecov Report

Merging #287 (c884100) into dev (ef9231c) will decrease coverage by 0.18%.
The diff coverage is 90.51%.

@@            Coverage Diff             @@
##              dev     #287      +/-   ##
==========================================
- Coverage   92.50%   92.32%   -0.19%
==========================================
  Files         128      130       +2
  Lines        3295     3361      +66
  Branches     1096     1123      +27
==========================================
+ Hits         3048     3103      +55
- Misses        112      117       +5
- Partials      135      141       +6

Impacted Files
Coverage Δ

...magma-dom/src/components/Pagination/Pagination.tsx
93.02% <70.00%> (-6.98%)
:arrow_down:

...ackages/react-magma-dom/src/hooks/useControlled.ts
85.71% <85.71%> (ø)

...act-magma-dom/src/components/Datagrid/Datagrid.tsx
93.02% <89.47%> (-3.95%)
:arrow_down:

...ackages/react-magma-dom/src/hooks/usePagination.ts
89.79% <89.79%> (ø)

...magma-dom/src/components/Table/TablePagination.tsx
100.00% <100.00%> (ø)

...ges/react-magma-dom/src/hooks/useDataPagination.ts
100.00% <100.00%> (+14.28%)
:arrow_up:

...act-magma-dom/src/components/Dropdown/Dropdown.tsx
82.60% <0.00%> (-2.90%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef9231c...c884100. Read the comment docs.

In the section for "Disabled Selectable Row", I don't see any disabled rows in the example

In the "InverseContext" section, the pagination isn't showing inversed.  I'm pretty sure we had that working before, but if not, I can get a visual design worked up for that quickly.

also, changeset


[WIP] Feature/react settings

This is a constant WIP for upgrading our current Settings panel to be React based.
This brings the ability to properly test a crucial part of our UI.  Increasing readability of our code.  And getting one step closer to removing jQuery (which I don't know why I used in the first place 👍 )

Current coverage is 65.54% (diff: 70.58%)

No coverage report found for master at 16b5f85.

Powered by Codecov. Last update 16b5f85...f044763

Changes Unknown when pulling 69ebc7fb33622841815037a93aee3da7bda91bf6 on feature/react-settings into ** on master**.

Changes Unknown when pulling 69ebc7fb33622841815037a93aee3da7bda91bf6 on feature/react-settings into ** on master**.

Changes Unknown when pulling 16e14950f18a32ece0206cc11b004bdffc2fe36a on feature/react-settings into ** on master**.

Changes Unknown when pulling 85104c0a4b1b48c96402f1249b92308d05a6d20f on feature/react-settings into ** on master**.

Changes Unknown when pulling 3f12c466d1a6ce5ea15f82c14f79939f6e6729a1 on feature/react-settings into ** on master**.

Changes Unknown when pulling 06e372b45a6fdbcf0c57655c4cdb287fa72b0606 on feature/react-settings into ** on master**.

Changes Unknown when pulling 0989aa5ac7f28fb162124097b3959e03e19006f2 on feature/react-settings into ** on master**.

Changes Unknown when pulling b8bb92170dcb4deec1ecbc6bf7bb8efa4e295d36 on feature/react-settings into ** on master**.

Changes Unknown when pulling 73de4e81b55b3976991ac9c8e75103f3e26be5a2 on feature/react-settings into ** on master**.

Changes Unknown when pulling 53f2998c93b61ba4cf152ba682466c931e4d804f on feature/react-settings into ** on master**.

Changes Unknown when pulling 7a29f7eca108dc8aaa92a2d99cd04c60d9f9c98a on feature/react-settings into ** on master**.

Changes Unknown when pulling aaa8109dfa694ab8974bfea23057defd8d6c79aa on feature/react-settings into ** on master**.

Changes Unknown when pulling aaa8109dfa694ab8974bfea23057defd8d6c79aa on feature/react-settings into ** on master**.

Changes Unknown when pulling 92c6a0a5b217b48778c2f6f293ee69b8dd56a546 on feature/react-settings into ** on master**.

Changes Unknown when pulling 92c6a0a5b217b48778c2f6f293ee69b8dd56a546 on feature/react-settings into ** on master**.

Changes Unknown when pulling 8c4d2c31a90a2dbad0da535566f078f9c727c4a5 on feature/react-settings into ** on master**.

Changes Unknown when pulling b5fe10404a314d1069946fbed96fdf5d5bd49956 on feature/react-settings into ** on master**.

Changes Unknown when pulling 9ae508b52d9f45131525c925c7b51392a81f7f7d on feature/react-settings into ** on master**.

Changes Unknown when pulling 9ae508b52d9f45131525c925c7b51392a81f7f7d on feature/react-settings into ** on master**.

Changes Unknown when pulling 9ae508b52d9f45131525c925c7b51392a81f7f7d on feature/react-settings into ** on master**.

Changes Unknown when pulling 41669c4da9e9354ce7e46cedf8734e39f5c143ef on feature/react-settings into ** on master**.

Changes Unknown when pulling<PHONE_NUMBER>bce39d14018309dcd31830761e5454 on feature/react-settings into ** on master**.

Changes Unknown when pulling 7d1cfb97f40fa037313c538ab6c1928a608908ae on feature/react-settings into ** on master**.

Changes Unknown when pulling afeca9a231d2c1b0e1819357c556b9773c2deb4e on feature/react-settings into ** on master**.

Changes Unknown when pulling 6c3d3f600c0b2c6d9cc09a52917e6ad81e3a0620 on feature/react-settings into ** on master**.

Changes Unknown when pulling 23b9b847b75b9e59da4266a555d4c7326594bb67 on feature/react-settings into ** on master**.


Unwrapping of cyclical data

I would love to give Kamui a try in my particular application where I am tasked with unwrapping weighted gridded data with one cyclical axis. However, I do not understand the algorithm well enough to figure out how to implement cyclical axes. An example of how to unwrap 2D data with one cyclical axis would be highly appreciated. Ideally, this functionality would be part of the library itself with an optional cyclical_axis keyword.

Hi @nup002 , thanks for asking. That sounds like a good idea and can be easily added.
The type of cyclical_axis will be Union[Int, Tuple[Int]], in the case of multiple cyclical axes.
I'll find a time to do it.

Hi @nup002,
The cyclical axis feature is added to the dev branch (5ea1b10f0dee2a3700bc55cbe5d95bcdb2c785dc).
If you have time, could you help me test if it work in your case.

Hello @yoyololicon ,
I am really excited to try this out! I will get back to you once I've had a go at using it, to provide feedback. It may take a few weeks, as we have a lot of higher-priority tasks.


chore(circleci): remove duplicate compile step

Which problem is this PR solving?

WIP

Codecov Report

Merging #510 into master will decrease coverage by 33.86%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master     #510       +/-   ##
===========================================
- Coverage   88.54%   54.67%   -33.87%
===========================================
  Files         116       43       -73
  Lines        5561     1112     -4449
  Branches      502      120      -382
===========================================
- Hits         4924      608     -4316
+ Misses        637      504      -133

Impacted Files
Coverage Δ

packages/opentelemetry-web/src/WebTracer.ts
100% <0%> (ø)
:arrow_up:

packages/opentelemetry-tracing/src/Span.ts

...pe-async-hooks/test/AsyncHooksScopeManager.test.ts

...ntelemetry-node/test/instrumentation/utils.test.ts

...postgres/opentelemetry-plugin-pg/test/testUtils.ts

...ges/opentelemetry-exporter-jaeger/src/transform.ts

...ges/opentelemetry-exporter-zipkin/test/e2e.test.ts

packages/opentelemetry-metrics/src/Utils.ts

...telemetry-tracing/src/export/BatchSpanProcessor.ts

...lemetry-tracing/src/export/InMemorySpanExporter.ts

... and 64 more


Bump protobuf-java to 3.21.7 in repository-gcs and repository-hdfs

Signed-off-by: Ryan Bogan<EMAIL_ADDRESS>Description
Backport 4790 to 1.x
Check List

[X] New functionality includes testing.

[X] All tests pass

[X] New functionality has been documented.

[X] New functionality has javadoc added

[X] Commits are signed per the DCO using --signoff
[X] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #4890 (156f879) into 1.x (a8573af) will increase coverage by 0.03%.
The diff coverage is n/a.

:exclamation: Current head 156f879 differs from pull request most recent head 7697ab9. Consider uploading reports for the commit 7697ab9 to get more accurate results

@@             Coverage Diff              @@
##                1.x    #4890      +/-   ##
============================================
+ Coverage     77.82%   77.86%   +0.03%
- Complexity    63367    63425      +58
============================================
  Files          4453     4453
  Lines        275011   275011
  Branches      41172    41172
============================================
+ Hits         214026   214132     +106
+ Misses        44029    44007      -22
+ Partials      16956    16872      -84

Impacted Files
Coverage Δ

...nsearch/search/dfs/DfsPhaseExecutionException.java
0.00% <0.00%> (-66.67%)
:arrow_down:

.../index/shard/IndexShardNotRecoveringException.java
50.00% <0.00%> (-50.00%)
:arrow_down:

...n/indices/upgrade/post/UpgradeSettingsRequest.java
30.00% <0.00%> (-45.00%)
:arrow_down:

...java/org/opensearch/script/ScriptContextStats.java
33.33% <0.00%> (-40.01%)
:arrow_down:

...regations/metrics/AbstractHyperLogLogPlusPlus.java
51.72% <0.00%> (-37.94%)
:arrow_down:

...java/org/opensearch/threadpool/ThreadPoolInfo.java
56.25% <0.00%> (-37.50%)
:arrow_down:

.../opensearch/index/stats/IndexingPressureStats.java
61.03% <0.00%> (-35.07%)
:arrow_down:

...search/index/stats/ShardIndexingPressureStats.java
42.30% <0.00%> (-30.77%)
:arrow_down:

...pensearch/index/shard/SearchOperationListener.java
75.53% <0.00%> (-23.41%)
:arrow_down:

...ch/action/support/replication/ReplicationTask.java
46.15% <0.00%> (-23.08%)
:arrow_down:

... and 464 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Backport mumble fix

Motivation for this change
Fixes https://github.com/NixOS/nixpkgs/issues/51196
I have cherry-picked the 2 commits from master that fixes the problem.
This is my first backport, so please review extra thorough.
cc maintainers @jgeerds @wkennington
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

I have used git cherry-pick commit1 commit2 and not git cherry-pick -xe, so it's not in line with CONTRIBUTING.md.
I'm also not sure if the first commit (c++11) is even needed, but it works.
Hope that's OK.

@GrahamcOfBorg build mumble


stevedore: add pbr to propagatedBuildInputs

This fixes a crash in flexget and possibly other packages. I think the
issue was introduced in fc1165b0d94ec8b1b01b7522ed3143bd3dd03e5c
Without this patch, flexget gives an error at start about not being able to locate pbr.  This fixes it on my system.  I don't know enough about these packages or Python to be completely sure whether this is the correct fix.
cc @domenkozar @jagajaga @mbakke

Interesting. Was this a runtime crash or build time crash? I'm chasing a rabbit hole of updates in #9549 and it's possible that will solve some problems.

@mbakke It was a runtime crash:
~  nix-shell -p pythonPackages.flexget --run flexget                                                 ~
Traceback (most recent call last):
  File "/nix/store/3j6zxfpxagn0d841192dq6f7m5llzjw0-python2.7-FlexGet-1.2.337/bin/.flexget-wrapped", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/nix/store/1axqs4cf1rrf7kdvfm705vlis98n8xc7-python2.7-setuptools-18.2/lib/python2.7/site-packages/setuptools-18.2-py2.7.egg/pkg_resources/__init__.py", line 3084, in <module>
    @_call_aside
  File "/nix/store/1axqs4cf1rrf7kdvfm705vlis98n8xc7-python2.7-setuptools-18.2/lib/python2.7/site-packages/setuptools-18.2-py2.7.egg/pkg_resources/__init__.py", line 3070, in _call_aside
    f(*args, **kwargs)
  File "/nix/store/1axqs4cf1rrf7kdvfm705vlis98n8xc7-python2.7-setuptools-18.2/lib/python2.7/site-packages/setuptools-18.2-py2.7.egg/pkg_resources/__init__.py", line 3097, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/nix/store/1axqs4cf1rrf7kdvfm705vlis98n8xc7-python2.7-setuptools-18.2/lib/python2.7/site-packages/setuptools-18.2-py2.7.egg/pkg_resources/__init__.py", line 651, in _build_master
    ws.require(__requires__)
  File "/nix/store/1axqs4cf1rrf7kdvfm705vlis98n8xc7-python2.7-setuptools-18.2/lib/python2.7/site-packages/setuptools-18.2-py2.7.egg/pkg_resources/__init__.py", line 952, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/nix/store/1axqs4cf1rrf7kdvfm705vlis98n8xc7-python2.7-setuptools-18.2/lib/python2.7/site-packages/setuptools-18.2-py2.7.egg/pkg_resources/__init__.py", line 839, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pbr<2.0,>=1.3' distribution was not found and is required by stevedore


[Backport 2.16] Create listener to refresh search thread resource usage (#14832)

Backporting https://github.com/opensearch-project/OpenSearch/pull/14832 to 2.16
Signed-off-by: Siddhant Deshmukh<EMAIL_ADDRESS>
[bug fix] fix incorrect coordinator node search resource usages

Signed-off-by: Chenyang Ji<EMAIL_ADDRESS>
fix bug on serialization when passing task resource usage to coordinator

Signed-off-by: Chenyang Ji<EMAIL_ADDRESS>
add more unit tests

Signed-off-by: Chenyang Ji<EMAIL_ADDRESS>
remove query insights plugin related code

Signed-off-by: Chenyang Ji<EMAIL_ADDRESS>
create per request listener to refresh task resource usage

Signed-off-by: Chenyang Ji<EMAIL_ADDRESS>
Make new listener API public

Signed-off-by: Siddhant Deshmukh<EMAIL_ADDRESS>
Add changelog

Signed-off-by: Siddhant Deshmukh<EMAIL_ADDRESS>
Remove wrong files added

Signed-off-by: Siddhant Deshmukh<EMAIL_ADDRESS>
Address review comments

Signed-off-by: Siddhant Deshmukh<EMAIL_ADDRESS>
Build fix

Signed-off-by: Siddhant Deshmukh<EMAIL_ADDRESS>
Make singleton

Signed-off-by: Siddhant Deshmukh<EMAIL_ADDRESS>
Address review comments

Signed-off-by: Siddhant Deshmukh<EMAIL_ADDRESS>
Make sure listener runs before plugin listeners

Signed-off-by: Siddhant Deshmukh<EMAIL_ADDRESS>
Spotless

Signed-off-by: Siddhant Deshmukh<EMAIL_ADDRESS>
Minor fix

Signed-off-by: Siddhant Deshmukh<EMAIL_ADDRESS>
Signed-off-by: Chenyang Ji<EMAIL_ADDRESS>Signed-off-by: Siddhant Deshmukh<EMAIL_ADDRESS>Signed-off-by: Jay Deng<EMAIL_ADDRESS>Co-authored-by: Chenyang Ji<EMAIL_ADDRESS>Co-authored-by: Jay Deng<EMAIL_ADDRESS>(cherry picked from commit 8ff3bcc4632287d8a784a1cba662957d6f921851)

Description
[Describe what this change achieves]
Related Issues
Resolves #[Issue number to be closed when this PR is merged]

Check List

[ ] Functionality includes testing.
[ ] API changes companion pull request created, if applicable.
[ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

❌ Gradle check result for 20b4370: FAILURE
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

https://github.com/opensearch-project/OpenSearch/issues/14292


Improve rate handling

Recently I ran into some trouble with using zmap with the rate-based packet limit via -r.  In debugging this I didn't see the rate in the logs, which was confusing as compared to what you see with -B (for bandwidth).  This patch fixes this such that a proper invocation with -r will log the rate.  Also, I've added some simple error handling with rate and bandwidth.
With just rate specified (-r 1) correctly:
Feb 05 09:37:58.754 [DEBUG] send: rate set to 1 pkt/s

With and invalid rate (-r 0):
Feb 05 09:37:50.202 [FATAL] send: rate impossibly slow

With rate and bandwidth (-r 10 -B 10M):
Feb 05 09:33:20.160 [FATAL] send: Must specify rate or bandwidth, or neither, not both.

i found error by default.
root@iZbp15nvuk3jmtwsnrjpo3Z:~/zmap-master/src# zmap --bandwidth=10M --target-port=80 --max-targets=10000 --output-file=results.csv
Jul 24 14:54:54.818 [INFO] zmap: output module: csv
Jul 24 14:54:54.818 [FATAL] send: must specify rate or bandwidth, or neither, not both.
root@iZbp15nvuk3jmtwsnrjpo3Z:~/zmap-master/src#


eval doesn't work correctly.

On CPython, eval works even if space exists at the beginning of the line.
>>> eval("   1")
1

But on RustPython, IndentationError occurs.
>>>>> eval("   1")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndentationError: unexpected indent at line 1 column 4
   1
   ^

Hi, I'd like to take a look at this. @DimitrisJim Could you assign this to me?

@yt2b has been opening issues and sending a PR at the same time so you'd have to get the ok from them! 🙂

#4365

Understood! I made a fix I thought it was suitable, but if @yt2b already made a PR it can be used instead.

@branai
OK! I will leave it for you.

There's another eval error if you want to tackle it @yt2b, it involves a custom locals object passed as the locals argument that raises something else than KeyError (see the final check if test_eval of test_builtins.py). A slight change of the LoadNameAny opcode functionality is required since it currently ignores the possible error when looking up a name in the locals.

@DimitrisJim
I'll tackle another eval error in #4375.


DevTestLabs: When using the CLI to create an environment fails with 'user_name' error

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR has modified HISTORY.rst describing any customer-facing, functional changes. Note that this does not include changes only to help content. (see Modifying change log).

[x] I adhere to the Command Guidelines.

View a preview at https://prompt.ws/r/Azure/azure-cli/5860
This is an experimental preview for @microsoft users.

abandon


Breaking posix_memalign precondition is not UB

The size==0 test here might be overtesting, but I figured might as well test it and leave a comment saying it is fine to remove it if the implementation changes.
Fixes #2099

Looks great, thanks a lot!
@bors r+

:pushpin: Commit e75b2c8543ee08abed6f3d072447afc514af13bb has been approved by RalfJung
It is now in the queue for this repository.

:hourglass: Testing commit e75b2c8543ee08abed6f3d072447afc514af13bb with merge 31b95dd74925dff6753a313bff5558056bf51eb0...

:sunny: Test successful - checks-actions
Approved by: RalfJung
Pushing 31b95dd74925dff6753a313bff5558056bf51eb0 to master...


k_thread_join() taking a very long time on qemu_cortex_m3

Describe the bug
When K_MSEC(5000) is used for the timeout parameter to k_thread_join() it takes 5 seconds to return, even when the thread has already run to completion.
However, when K_FOREVER is specified as the timeout paramter, it returns as soon as the thread has run to completion.
From what the documentation says, when specifying K_MSEC(5000) for the timeout parameter, k_thread_join() should return immediately as well if the thread in question has already run to completion.
To Reproduce
Steps to reproduce the behavior:

build and run the attached test case
see error

Expected behavior
As per the api documentation:
"This API returns immediately if the thread isn’t running"
Impact
unnecessary delays
Screenshots or console output
starting test - test_pipe_and_poll_write_block
3130 T: 0x200008cc test_pipe_and_poll_write_block(): 128: setting up context
3130 T: 0x200008cc test_pipe_and_poll_write_block(): 133: creating secondary thread
3140 T: 0x200008cc test_pipe_and_poll_write_block(): 139: created secondary thread 0x20000054
3140 T: 0x200008cc k_pipe_write_avail(): 33: pipe: 0x200000e4 N: 5 u: 0 r: 0 w: 0
3140 T: 0x200008cc test_pipe_and_poll_write_block(): 168: calling k_pipe_put()
3140 T: 0x200008cc test_pipe_and_poll_write_block(): 173: have written 1 bytes
3140 T: 0x200008cc k_pipe_write_avail(): 33: pipe: 0x200000e4 N: 5 u: 1 r: 0 w: 1
3140 T: 0x200008cc test_pipe_and_poll_write_block(): 168: calling k_pipe_put()
3140 T: 0x200008cc test_pipe_and_poll_write_block(): 173: have written 2 bytes
3140 T: 0x200008cc k_pipe_write_avail(): 33: pipe: 0x200000e4 N: 5 u: 2 r: 0 w: 2
3140 T: 0x200008cc test_pipe_and_poll_write_block(): 168: calling k_pipe_put()
3140 T: 0x200008cc test_pipe_and_poll_write_block(): 173: have written 3 bytes
3140 T: 0x200008cc k_pipe_write_avail(): 33: pipe: 0x200000e4 N: 5 u: 3 r: 0 w: 3
3140 T: 0x200008cc test_pipe_and_poll_write_block(): 168: calling k_pipe_put()
3150 T: 0x200008cc test_pipe_and_poll_write_block(): 173: have written 4 bytes
3150 T: 0x200008cc k_pipe_write_avail(): 33: pipe: 0x200000e4 N: 5 u: 4 r: 0 w: 4
3150 T: 0x200008cc test_pipe_and_poll_write_block(): 168: calling k_pipe_put()
3150 T: 0x200008cc test_pipe_and_poll_write_block(): 173: have written 5 bytes
3150 T: 0x200008cc k_pipe_write_avail(): 33: pipe: 0x200000e4 N: 5 u: 5 r: 0 w: 0
3150 T: 0x200008cc test_pipe_and_poll_write_block(): 146: the next call to k_pipe_put() would block
3150 T: 0x200008cc test_pipe_and_poll_write_block(): 154: calling k_poll()
3150 T: 0x20000054 reader_thread(): 101: secondary thread running
3150 T: 0x20000054 reader_thread(): 105: have written 5 bytes
4160 T: 0x20000054 reader_thread(): 109: calling k_pipe_get()
4160 T: 0x20000054 reader_thread(): 113: calling k_poll_signal_raise()
4160 T: 0x200008cc test_pipe_and_poll_write_block(): 157: returned from k_poll()
4160 T: 0x200008cc test_pipe_and_poll_write_block(): 161: calling k_poll_signal_check()
4160 T: 0x200008cc test_pipe_and_poll_write_block(): 165: received signal / correct result
4160 T: 0x200008cc test_pipe_and_poll_write_block(): 168: calling k_pipe_put()
4160 T: 0x200008cc test_pipe_and_poll_write_block(): 173: have written 6 bytes
4160 T: 0x200008cc test_pipe_and_poll_write_block(): 176: success!
4160 T: 0x200008cc test_pipe_and_poll_write_block(): 178: calling k_thread_join()
4160 T: 0x20000054 reader_thread(): 116: terminating..
9170 T: 0x200008cc test_pipe_and_poll_write_block(): 181: joined!
PASS - test_pipe_and_poll_write_block

Just looking at the log above again, it's possible that the thread had not run to completion.. I still think it's worth investigating though. I'm seeing consistent delays across several test cases.
Environment (please complete the following information):

OS: Ubuntu Linux (Bionic)
Toolchain: cross-compile (gcc-arm-none-eabi package in Ubuntu)
Commit SHA or Version used: 75b42d8a37b09cff7dc088515ad15f5f6657a8d0

Additional context

test_pipe_and_poll.c.txt

Confirmed also broken on qemu_x86, should be an arch-neutral issue. I'm investigating...

Hmm, this works flawlessly with a K_FOREVER timeout.
Whatever the root cause ends up being, need to update test cases to exercise this scenario.


Currency Input field

I am using AvField throughout my entire codebase and I am afraid if there is a currency field available in this package. If it's not available then is there any way we can pass custom input to the AvForm for the currency field? Any example will really help.
Thank you.

The same here!


zql: invalid zng panic with cut-put-cut flowgraph

I ran into this bug while finishing up #856 (rename proc):
$ zq -t "head 1 | cut id | put id2=id.orig_h | cut -c id.orig_h" conn.log
#0:record[id:record[orig_h:ip,orig_p:port,resp_h:ip,resp_p:port],id2:ip]
0:[[<ZNG-ERR type ip [%!s(PANIC=String method: runtime error: slice bounds out of range [:78] with capacity 30)]: failure trying to decode IP address that is not 4 or 16 bytes long>;<ZNG-ERR type port [%!s(PANIC=String method: runtime error: slice bounds out of range [5:4])]: port encoding must be 2 bytes>;<ZNG-ERR type ip [%!s(PANIC=String method: runtime error: slice bounds out of range [6:2])]: failure trying to decode IP address that is not 4 or 16 bytes long>;]<IP_ADDRESS>;]

Working on this right away the bug is showing up in my rename unit tests.

Not a panic, but incorrect and probably the same bug:
$ zq -t "head 1 | cut id | put id2=id.orig_h | cut id.orig_h" conn.log
#0:record[id:record[orig_h:ip,orig_p:port,resp_h:ip,resp_p:port]]
0:[[<IP_ADDRESS>;]]

Well, the bug comes from using different resolver.Contexts in the zio Reader and in the flowgraph. Kind of surprising that things have worked this long with this issue!

Verified in zq commit 5b3fc4.
The example that kicked off the issue no longer produces a panic:
~/work/zq-sample-data/zeek-default$ zq -t "head 1 | cut id | put id2=id.orig_h | cut -c id.orig_h" conn.log.gz
#0:record[id:record[orig_p:port,resp_h:ip,resp_p:port],id2:ip]
0:[[39681;<IP_ADDRESS>;3389;]<IP_ADDRESS>;]

The other example from the comment above now also produces correct output:
~/work/zq-sample-data/zeek-default$ zq -t "head 1 | cut id | put id2=id.orig_h | cut id.orig_h" conn.log.gz
#0:record[id:record[orig_h:ip]]
0:[[<IP_ADDRESS>;]]

Thanks @henridf!


Wrong date format.

In objectPresenter in LfmHelpers.php line 479 on version 1.8, the date format should be "Y-m-d h:i" instead of "Y-m-d h:m".

Thanks! Good catch! See #418 . If it's merged the next release will fix it.

Merged. Thank you @domialex


Using {T} in mapper type imported from file produces wrong import

Describe the bug
The problem is related to custom wrapper type imported from a file using ./someFile$SomeType<{T}>  syntax. This feature already works for global types #1631, but for types coming from file generates wrong imports.
To Reproduce
Steps to reproduce the behavior:

My custom mapper type

export type UserExtension<T> = Partial<T> & {
    someDatabaseRelationId: number
}

My codegen.yml config file:

plugins:
      - typescript
      - typescript-resolvers
config:
    mappers:
       User: ./userExtension#UserExtension<{T}>

Generated types

import { UserExtension<{T}> } from './userExtension; // wrong import

export type ResolversTypes = {
  Concept: UserExtension<User> //correct type
}

Expected behavior
Correct types are generated
import { UserExtension } from './userExtension; // wrong import

export type ResolversTypes = {
  Concept: UserExtension<User> //correct type
}

Environment:

OS: macOS 10.14.3
@graphql-codegen/1.1.0:
NodeJS: v11.12.0

Fixed at #1748

Fixed in 1.1.1 🎉


Model overfit on unbalanced outcomes

The reason that the model that has been trained in the notebook LOGISTIC REGRESSION MODEL.ipynb is because of imbalanced classes. I can fix the model if this issue is assigned to me.

Let's move this discussion to #4. I'd love to see your suggestions.


Installation Error "No module named cymem.cymem" El Capitan, Ananconda, spacy-0.99

I am on Mac OSX El Capitan and using Anaconda with a fresh install of python in my home dir.
MacBook-Pro:anaconda vikram$ which python
/Users/vikram/anaconda/bin/python
When running
python -m spacy.en.download all
I get the error:
/Users/vikram/anaconda/bin/python: No module named cymem.cymem
While installing spacy there was a message about possible conflict in cymem package which it automatically resolved.
MacBook-Pro:anaconda vikram$ conda install spacy
Fetching package metadata: ......
Solving package specifications: ......................................
Warning: 4 possible package resolutions (only showing differing packages):
[u'cymem-1.30-0.tar.bz2', u'murmurhash-0.24-0.tar.bz2'],
[u'cymem-1.30-py27_0.tar.bz2', u'murmurhash-0.24-0.tar.bz2'],
[u'cymem-1.30-py27_0.tar.bz2', u'murmurhash-0.24-py27_0.tar.bz2'],
[u'cymem-1.30-0.tar.bz2', u'murmurhash-0.24-py27_0.tar.bz2'],
Package plan for installation in environment /Users/vikram/anaconda:
The following packages will be downloaded:
package                    |            build
---------------------------|-----------------
python-2.7.11              |                0         9.9 MB
cloudpickle-0.1.1          |           py27_0          15 KB
conda-env-2.4.5            |           py27_0          24 KB
cymem-1.30                 |                0          39 KB
murmurhash-0.24            |                0          15 KB
plac-0.9.1                 |           py27_0          31 KB
setuptools-18.5            |           py27_0         343 KB
text-unidecode-1.0         |           py27_0          62 KB
conda-3.18.9               |           py27_0         176 KB
preshed-0.44               |           py27_0          85 KB
thinc-4.0.0                |           py27_0         207 KB
spacy-0.99                 |      np110py27_0         2.6 MB
------------------------------------------------------------
                                       Total:        13.5 MB

There seems to be references to python3.5 in the conda-meta data for this module...
Perhaps it needed to resolve to [u'cymem-1.30-py27_0.tar.bz2', u'murmurhash-0.24-py27_0.tar.bz2'] when it chose which package to install?
MacBook-Pro:anaconda vikram$ more conda-meta/cymem-1.30-0.json
{
"arch": "x86_64",
"binstar": {
"channel": "main",
"owner_id": "5617c51fb7eda558af8b2827",
"package_id": "562b7c76ab59365f7a302252"
},
"build": "0",
"build_number": 0,
"channel": "https://conda.anaconda.org/spacy/osx-64/",
"depends": [
"python >=2.7"
],
"files": [
"lib/python3.5/site-packages/cymem-1.30-py3.5.egg-info",
"lib/python3.5/site-packages/cymem/init.pxd",
"lib/python3.5/site-packages/cymem/init.py",
"lib/python3.5/site-packages/cymem/pycache/init.cpython-35.pyc",
"lib/python3.5/site-packages/cymem/cymem.c",
"lib/python3.5/site-packages/cymem/cymem.cpython-35m-darwin.so",
"lib/python3.5/site-packages/cymem/cymem.pxd",
"lib/python3.5/site-packages/cymem/cymem.pyx"
],
"license": "MIT",
"link": {
"source": "/Users/vikram/anaconda/pkgs/cymem-1.30-0",
"type": "hard-link"
},
"machine": "x86_64",
"md5": "3f646ce5d6b86c600b92a23697b96498",
"name": "cymem",
"operatingsystem": "darwin",
"platform": "osx",
"requires": [],
"size": 39690,
"subdir": "osx-64",
"target-triplet": "x86_64-any-darwin",
"url": "https://conda.anaconda.org/spacy/osx-64/cymem-1.30-0.tar.bz2",
"version": "1.30"
}

Problem seems to be in the way the packages are listed in the repodate file.
https://conda.anaconda.org/spacy/osx-64/repodata.json
has an entry for
"cymem-1.30-0.tar.bz2": with "depends": [
"python >=2.7"
],
before this one.
"cymem-1.30-py27_0.tar.bz2":  with "depends": [
"python 2.7*"
],
The first one seems to be selected during installation and it installs the python 3.5 libraries.
/Users/vikram/anaconda/pkgs/cymem-1.30-0/lib/python3.5/
Same for murmurhash,
"murmurhash-0.24-0.tar.bz2" is listed before "murmurhash-0.24-0.tar.bz2" and it is also installing python 3.5 libraries.
/Users/vikram/anaconda/pkgs/murmurhash-0.24-0/lib/python3.5/

Thanks for this, this obviously doesn't look right. We are not that experienced in building conda packages yet, in case you want to help out can you take a look at our conda build script at https://github.com/spacy-io/spaCy-conda/blob/master/.travis.yml and see whether there is something to improve?

I am brand new to Anaconda (just installed to it to get spacy going) and its past bed time here in the US, but here is what a quick reading suggests (I have not tried to build it yet).
http://conda.pydata.org/docs/spec.html#repository-structure-and-index
-The command conda index can be used to create such an index from the conda packages within a directory._
It seems like the director(ies) which contains the python version specific packages, also contain the  package without the versions and when the "conda index" runs, they too get included. I haven't figured out where the conda index runs in the build process yet (or there may be some other command which also has a similar effect).
The unversioned packages appear for all of the following:
https://conda.anaconda.org/spacy/osx-64/repodata.json
cymem-1.30-0.tar.bz2
headers_workaround-0.18-0.tar.bz2
murmurhash-0.24-0.tar.bz2
preshed-0.44-0.tar.bz2
thinc-4.0.0-0.tar.bz2
The order in which they get included  in the repodata, may depend on how they get listed in the dir.
I think if those unversioned package files are located in a different dir, they would not end up in the repodata.

This is some great debugging — we really appreciate this, thanks!
If you just want to get up and running, it seems easier to use pip instead of conda, as it seems our package on conda isn't correct. While we sort the issue out, I suggest the following:

Start with a clean conda environment
conda create --name myenv numpy

The conda create command requires you to specify at least one package to install. I always find this super confusing. The important thing here is that you can't write spacy! If you do, spacy will be installed from the conda package, when we want to install it from pip.

Activate the environment

source activate myenv

Install with pip

pip install spacy

Install data

python -m spacy.en.download --force all

Thanks. That worked.
Did have to add the unicode conversion for the 1st example
doc = nlp(unicode('Hello, world. Here are two sentences.'))
Was Getting a type mismatch error
tokens = self.tokenizer(text)
TypeError: Argument 'string' has incorrect type (expected unicode, got str)

All spaCy documentation assumes Python3 unicode conventions - thanks, we should make that clearer. Adding a from __future__ import unicode_literals makes your Python2 code future-proof.

Thanks for the pointers on __future__
Working fine now.


Draggable path

Opening this as an issue, but really just an idea to think about.
Being able to click and hold on a line segment on the path and drag the entire connected path to a new location would facilitate centering a region on an image feature.  Would be useful for multiple similarly shaped region statistics computation.

if i understand correctly what you are asking for, this is already supported: select a point by clicking on it, select any point after it by shift+click, which let's select multiple nodes. you can then drag all selected segments as a group. similar behavior is supported on the node table in the measure control: click a node, then shift+click on any node below it to do the multimode selection. you can then drag the selected portion of the path.
let me know if this covers what you had in mind.

Yes, indeed.  Already implemented.  A little cumbersome if there are many nodes, and I also found that I inadvertently moved two nodes when shift-clicking, but I understand there needs to be some selection mechanism, so this is servicable.  Thanks.


cannot get module info

I am getting error
$ gohack get -vcs gopkg.in/danilopolani
cannot get module info: go list -m: can't compute 'all' using the vendor directory
	(Use -mod=mod or -mod=readonly to bypass.)

Could you provide a minimal reproducible example here, please? Otherwise it's very hard to know what's going on. For example, I can't tell if you've got a go.mod file.

Here is a an example repo https://github.com/gadelkareem/test
Just clone it and do a go mod vender then gohack get -vcs gopkg.in/danilopolani and you should see the error.

@gadelkareem Just to point out, the repo you're pointing to is empty.  No code in it whatsoever.

I think this is not relevant anymore, thanks!


Random seek point can overshoot

Hi there. I was getting lots of "Ran out of frames. Looping." messages in my application, even though I never requested more frames than available. The following test showed that the errors were not deterministic, because even for the same video and the same number of frames the message sometimes occurred and sometimes not:
def test_ran_out_of_frames(width, height, duration, framerate):
    for idx in range(50):
        print('Run {}'.format(idx))

        with open('test_video.mp4', 'rb') as f:
            encoded_video = f.read()

        num_frames = int(duration * framerate)
        video, _ = lintel.loadvid(encoded_video,
                                  height=height, width=width, num_frames=num_frames)

Output:
Run 0
Run 1
Run 2
Ran out of frames. Looping.
Run 3
Run 4
Run 5
Run 6
Ran out of frames. Looping.
Run 7
Run 8
Run 9
.
.
.

Doing the same with should_random_seek=False solved the problem, so it very much looks like the random seek point does not truly respect the space to leave for the requested frames.
Can you recreate the issue and if so take a look into this?

Hi! I think respecting the number of frames in the video is a tricky business, for example due to placement of the keyframes, the seek can overshoot. But I can take a look, since I agree this is less than satisfactory. Perhaps a "strict" mode is called for.
What is the length of video you are using? I will try to reproduce with a similar length video.

Thanks for your quick reply! The length of my video is about 10 seconds.

Hello again, I was wondering if you might be able to give the seek-point-overshoot branch a try (https://github.com/dukebw/lintel/tree/seek-point-overshoot).
I changed the seeking in loadvid to work based on AVStream.nb_frames, rather than some poor approximations in seconds. This is more similar to how the loadvid_frame_nums seeking works now.
I have still found at least one video where AVStream.nb_frames is wrong, in this case AVStream.nb_frames reported 168, but using receive_frame() I could only decode 166 frames. So the method will still overshoot by however much AVStream.nb_frames overestimates the number of frames in the video. If this extremely precise accuracy is required, I wonder if it is better to preprocess all the videos by counting frames with receive_frame(), store metadata about how many frames can really be decoded, then use loadvid_frame_nums to only get frames within those bounds. I'm also not sure if the estimate here: https://github.com/dukebw/lintel/blob/5af35bd3a80b9012b43801142235d301122d2cb0/lintel/py_ext/lintelmodule.c#L134-L186 maybe already accounts for when AVStream.nb_frames from the container is wrong, and we should just use this estimate always.
Anyway, please let me know what you think, and if you get a chance to try the fix. Thank you for pointing out the bug!

Thanks for your work! I gave the branch a try, but as you have already noticed there are some problems concerning nb_frames. For the videos in my database I need to reduce nb_frames reported by ffprobe by exactly 3 to get the number of frames that can actually be decoded. That is also why I am not able to check, if the modifications improved the random seek point placement.
Do you think it might be possible that the nb_frames offset somehow comes from lintel itself? I find it hard to imagine that so many of our videos have wrong meta information. The reported nb_frames also matches duration*avg_frame_rate, so the information seems to be consistent.
My preferred solution to all of this would be to allow an argument such as -1 for num_frames. Lintel could then return just as many frames as it is able to decode. The downside here of course is that the output stream can not easily be preallocated. This would require the preprocessing that you mentioned.
Side note: Thanks for removing fps_cap. I was disabling it so far by setting it to an extremely high number. If you want to reintroduce it, I would similarly propose to add the option -1 for disabling.

Okay great, thank you for the feedback about both APIs, and I agree those would be improvements, at least it would be good to have some interface to allow reporting of how many frames were decoded successfully when decoding fails.
I will think about how to incorporate a solution that matches the frame count reported by ffprobe. I'm pretty sure that receive_frame (https://github.com/dukebw/lintel/blob/456d211e8a9a91aae0cd33801e85622996289cf2/lintel/core/video_decode.c#L51) is correctly using the send/receive packet API, so I will have to dig into the ffprobe code to see what the heck it is doing to count frames, and this may take some time.

Hi again! It appears you were right, and there was a bug in receive_frame. I was neglecting to "drain" the codec, as described here:
https://github.com/FFmpeg/FFmpeg/blob/fe06ed22e6e0a8c2995818c4532eb6f4ec9320b9/libavcodec/avcodec.h#L122-L133
I was wondering if you might be able to give commit ca3e1dedd3c3aeba9263e3ea7b33f45263ca34cb a try.


jvm-opts on uber

Is there any way to set up jvm-opts for an uberjar build?
I'm trying to move off depstar to this helper lib per the README's recommendation but neither passing :compile-opts {:jvm-opts ["-Doption"]} to opts nor :jvm-opts ["-Doption"] seem to work.

Per the readme:
As noted above, run-task takes an options hash map and a vector of aliases. The following options can be provided to run-task to override the default behavior:

:java-opts -- used instead of :jvm-opts from the aliases,
:jvm-opts -- used in addition to the :java-opts vector or in addition to :jvm-opts from the aliases,
:main -- used instead of 'clojure.main when building the java command to run,
:main-args -- used instead of :main-opts from the aliases,
:main-opts -- used in addition to the :main-args vector or in addition to :main-opts from the aliases.

See https://clojure.github.io/tools.build/clojure.tools.build.api.html#var-compile-clj
:compile-opts is a hash map of compiler options, not JVM flags.

If you can provide more detail about how you invoked depstar, I might be able to give you better guidance on how to invoke tools.build via build-clj.

@seancorfield Sorry, I'm probably missing something obvious, but my understanding of that readme snippet is that it was for the function run-task, not uber. Passing jvm-opts to uber is what I tried among other things.
This is my depstar alias:
:uberjar {:replace-deps {com.github.seancorfield/depstar {:mvn/version "2.1.278"}}
                     :exec-fn hf.depstar/uberjar
                     :exec-args {:jar "api.jar"
                                 :aot true
                                 :jvm-opts ["-Dmalli.registry/type=custom"]
                                 :main-class reverse.domain.application.core}

Thanks for such a quick response!

Ah, it turns out I was thinking about this https://ask.clojure.org/index.php/10894/create-basis-java-command-do-not-handle-jvm-opts which is about java-command (& process), not about compile-clj.
So you'll need to post on ask.clojure.org asking for compile-clj to support JVM options so that the core team can make a JIRA issue for it for tracking.
In the meantime, I guess you'll have to stay on depstar until that functionality is added to compile-clj (at which point, I'll expose it in build-clj).

https://ask.clojure.org/index.php/11078/tools-builds-compile-clj-does-not-allow-setting-jvm-options
Once TBUILD-18 is completed, I'll make sure build-clj supports it.

tools.build's compile-clj supports :java-opts now and build-clj will just pass that straight through. I'll cut a new release of build-clj that depends on the latest tools.build.

@seancorfield Thank you for the fix and the heads up!

Well, I didn't have to fix anything 😄 I just waited for @puredanger to do his thing!


[Drilldown] Add an 'else' clause to not let the $wrapper property undefined

If the parent of the drilldown menu already has the 'is-drilldown' css class, the $wrapper property used everywhere in the plugin stays undefined. Fixes that issue.

Hi @mlang38
What do you think about the changes I mentioned above ?

No pb. 'else' clauses are evil :wink:


Structured subproperties of is about

Description of the issue
From #1328 and OEO dev meeting 48:

"isAbout" relation has to be replaced by more specific relations

The object property is about is very generic. In a lot of cases, it would be helpful have a subproproperty of is about with a limited range.
The following subproperties exists, but they are rather unstructured.

'economic value of'
'has considered region'
'has interacting region'
'has scenario year'
'has study region'
'has study subregion'
'quantity value of'

E.g. given that we already have four regional subproperties, an intermediary property like has spatial region might be useful. Then axioms like 'is about' some 'spatial region' can be refined to 'has spatial region' some 'spatial region'.
Ideas of solution
If you already have ideas for the solution describe them here
Workflow checklist

[ ] I discussed the issue with someone else than me before working on a solution
[ ] I already read the latest version of the workflow for this repository
[ ] The goal of this ontology is clear to me

I am aware that

[ ] every entry in the ontology should have a definition
[ ] classes should arise from concepts rather than from words

Proposal for has spatial region: An is-about-relation between an information content entity and a spatial region that describes that the information content entity is about a spatial region.

No objections against my proposals and at least one thumbs up. I interpret this as consensus and will implement. 🚀


Add more typings:

sql.elements
sql.functions
sql.selectable
sql.sqltypes
sql.type_api

Sorry for a delay. I probably will not have time to review this today, but will try to do this tomorrow.

Also, I finally got time to run this against our internal codebase. This gives six extra errors, among those:
* `Argument "convert_unicode" to "UnicodeText" has incompatible type "str"; expected "bool"`

* `Argument "convert_unicode" to "Unicode" has incompatible type "str"; expected "bool"`

I've fixed the signature of __init__ for these types and have added tests for each of them locally.

* `"Alias" has no attribute "__iter__" (not iterable)`

I couldn't find an __iter__ in the class hierarchy of Alias, so I tried the following in a REPL:
from sqlalchemy import MetaData, Table, Column, Integer

m = MetaData()
t = Table('foo', m, Column('id', Integer))
a = t.alias('bar')
for o in a:    # Type Error: 'Alias' object is not iterable
    print(o)

Is your codebase adding something to Alias?

Are any tests needed for this PR, what do you think?

I've been adding tests for the failures you've found in the codebase you're checking. I can add more if you think we need more.

Is your codebase adding something to Alias?

Maybe, I didn't look in detail, so let us ignore this for now.

I can add more if you think we need more.

No, I think just a few about the __init__ is enough, thanks.

I've fixed the signature of __init__ for these types and have added tests for each of them locally.

You can push the latest updates, hopefully we can merge this today.


FormItem: Form item emits array of errors when an error occurs

Please make sure these boxes are checked before submitting your PR, thank you!

[x] Make sure you follow Element's contributing guide (中文 | English | Español).
[x] Make sure you are merging your commits to dev branch.
[x] Add some descriptions and refer relative issues for you PR.

This is to fix this issue I posted here earlier
https://github.com/ElemeFE/element/issues/12732

@ziyoung This is the payload emitted by the Form component at the moment. It does not emit the current error message.

Personally, I think both Form and FormItem should emit that error message to enable developers to wrap these components in any way they want.
I disagree with how the payload is being evaluated right now on the Form (as an array). An object of declared keys would be the way to go usually, but for the sake of backwards compatibility, I think It should be added as the third entry in that array anyway.

@ziyoung I agree with @YamenSharaf that we need more flexibility with the error messages to give the developers more power to control how they show the errors

@YamenSharaf Is it better to put the error message in the validate event of form?

If validate event emits error message, is it necessary to add error event?

@ibufu @ziyoung Personally I think we should have both, because:

The bigger use case for this is listening to a specific error on a specific form-item. Probably to take action on such error.

Each form item could have multiple errors associated with it at the time (Although it's showing one at a time). having a list of errors would be useful to trigger an action on specific error

The current validate event on form should only care about the errors visible to the end-user, and the way its payload is structured is not ideal (but it makes sense for it to emit the error itself anyway).

@ziyoung Also made this PR for having form emit the actual error on validate events. This doesn't seem very wieldy since ElForm emits that event all the time for each individual form element. I still think it can be useful though. https://github.com/ElemeFE/element/pull/12860

errors are actually a list of new Error(message). These messages are meant to be read by user (not to be used in program logic).
What would you plan to do with these strings? For example?
if (errors.find(e => e.message.includes('Field A is required'))) {
    blame_user_with_dialog(...)
}

String comparison (on error message) does not make sense to me.

@wacky6 String comparison is icky, but sometimes you wanna respond to a certain error in a certain way, let that be dimming a button or popping an alert. It could be that I want errors to appear on the side of the form as opposed underneath inputs.
The way things are set up now there's no way to access these errors for any reason which I don't think is very flexible.

In that case, I would append additional codes after calling validate(), and check the v-model / form's value directly.
Validators are supposed to show a message, but nothing more.


Added example demonstrating creation of MNE objects from numpy arrays.

Issue #1382. Now it uses just sin and cos for the data. Should it be random data or something else? Is plotting the right way to demonstrate functionality?

you should start your filename with plot_ if you want it to be included in the example gallery.

I would rather use the data from the sample dataset and use it. But let's see what the others think

You need to check your document using a flake8 tool

I actually think it's better if the data does not come from a sample dataset -- using np.random should be a good way to go. We want people to know they can put in anything they want...

+1
you can use scipy.signal gaussian pulse for example:
http://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.gausspulse.html

I actually think it's better if the data does not come from a sample dataset -- using np.random should be a good way to go.

I agree. But maybe we can use a more simple example.

I feel that the raw plot is not meaningful if you are using arbitrary data. What do the channel names mean?

How about data with a few different channels with pulled from different random distributions, e.g. uniform, gaussian, etc.? Sounds like a neat example.

It should be simple and direct the attention on how to create such objects, not the data.

I feel we should address the most common use case for this which is when people load data in some format that MNE does not support and construct MNE objects from it.

Maybe then some of the NEO files / file formats?

good idea -- one more thing for @mluessi to need to install on the build bot :)
But seriously, I think an example loading data with NEO would be great.

+1 for a NEO file!

@jaeilepp : https://github.com/NeuralEnsemble/python-neo for your reference :)

FYI: https://github.com/NeuralEnsemble/python-neo/issues/108

So should there be another example for NEO? I have a feeling that just a NEO example won't reduce the confusion with the users. Or am I the only one unaware of this NEO?

No, just change this one example to use NEO and real data instead of fake data.
The idea is that we want people to know that they can use arbitrary other data sources by creating them from Numpy arrays. NEO is a good python library for reading file formats, so an example that uses it in together with RawArray or EvokedArray would make it clear how users can extend MNE's functionality. Make sense?

So can I add a NEO sample data file to the sample data set somehow? Or should I create NEO files from the existing sample data and then extract the data and convert it back to MNE objects? I'm confused...

don't add data to the repo. You can use an edf file from eegbci
datasets. Neo can read EDF data.

Got it! Thanks.

Now that I look at it, it seems that EDF is not on the list of supported formats list (https://pythonhosted.org/neo/io.html#list-of-io). I was able to get the data in NEO reader with ExampleIO, but the channel names don't transfer. I can't seem to read the matlab file with NEO either. Any suggestions?

I could of course set the channel names by hand, but I think it wouldn't be as neat as an example. Should I do that?

looking at the example. I like it much. it's simple and gets the message clearly.
I would use a demo with NEO in another example.
how do the plots look like?

Well, the plots look a bit rough, since there are so little data for epoching and averaging, but I think they get the message through. I could add some titles and such to make it look better.
I've played a bit with the NEO libraries, and I could add a simple demo on extracting raw data from the NEOs ExampleIO object. The eventarrays in it are just oscillating signals, which seems weird to me, since it should be points of time, rather than data, if I've understood correctly.

let us know when to review and don't hesitate to share screen shots

The plots could be prettier, but I wanted to use the object's own plotting function to demonstrate, that they actually are the objects everyone knows.

This is ready for review, if everyone is okay with the example cases...

In this particular case, no. I could use channel indices though. Let me fix it...

@jaeilepp how ist going in this end. Keep in mind that we need to merge some contributions for the GSOC application.

This is ready for merging from my part. I should have probably notified earlier. I'm not too familiar with the Github workflow yet.

This is ready for merging from my part.

Ahh! Good to know :)

What did we decide? Splitting this example or having NEO and the array stuff in one example?

One example is fine with me.

I am wondering if we should be IO examples in a separate folder in
examples...

Fine with me as well, and the example runs well over here after pip installing neo

I'll let you merge if you're happy

@jaeilepp almost there :-)

@jaeilepp feel free to list your contribution in our whats_new.rst file . Just copy the patterns you find there.

No worries. It's the only way I'll learn :)

merged by rebase after addressing the nitpick
thanks @jaeilepp !


Get username in client side

Logged in user is connected with WS without problem. But now I want to get username in client side. But if I publish using js, I only get the message. How can I get the username and other related information. I have tried to modify MyTopic class. But it seems topic method(onPublish) does not work in case I send message through js. Also, if I want to add some other information with message how can I do that?
Here is the code for client side:
        var _WS_URI = "ws://{{ websocket_host }}:{{ websocket_port }}/";
        var webSocket = WS.connect(_WS_URI);
        webSocket.on("socket/connect", function(session){
            session.subscribe("app/notification", function(uri, payload){
                console.log("Received message ", uri, payload);
            });
            session.publish("app/notification", "I want to send more info including username with this message");

        });

Never mind. solved.

I am looking for the solution to this, can you please explain how did you solve it?


Integrate ELK

Use

[x] elasticsearch
[x] kibana
[x] logstash
[x] minimal setup

Write docs for

[ ] minimal setup
[ ] filebeat

Resolved by introducing:

https://github.com/stefa2k/prysm-ansible/blob/master/elknode.yaml
https://github.com/stefa2k/prysm-ansible/blob/master/filebeat-docker.yaml
https://github.com/stefa2k/docker-elk
https://github.com/stefa2k/prysm-docker-compose#elk


Fix MLIR CAPI DLL build

Recent added MLIR CAPI library breaks windows build, this PR fixes it.

@hawkinsp The story of building shared library on Windows with Bazel is totally s**tshow nowadays. The workaround is pretty weird, pls dont try to understand it 🤣

linking error in<PHONE_NUMBER>
xutility C2440 error in<PHONE_NUMBER>
Built successfully in<PHONE_NUMBER>

Wonderful! I had a sneaking suspicion that the Windows support wouldn't be that easy. Thanks so much for the fix!


left_sidebar: Align more-topics indicator on topic grid.

This PR gets the .searching-for-more-topics indicator participating in the same grid used on other topic rows. A prep commit just flattens a selector in the area.
CZO discussion

Screenshots and screen captures:

Before
After

Self-review checklist

[x] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[x] Explains differences from previous plans (e.g., issue description).
[x] Highlights technical choices and bugs encountered.
[x] Calls out remaining decisions and concerns.
[x] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[x] Each commit is a coherent idea.
[x] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[x] Visual appearance of the changes.
[x] Responsiveness and internationalization.
[x] Strings and tooltips.
[x] End-to-end functionality of buttons, interactions and flows.
[x] Corner cases, error conditions, and easily imagined bugs.

Merged, thanks @karlstolley!


FEATURE: Support ARM64 build for Windows

New devices are coming to market in a couple months. These are ARM64 devices running the Windows 10 OS. Can we get an officially supported build of VS Code that runs native on ARM64?

Electron would have to support ARM binaries on Windows first, which it currently does not.

For reference, you should follow along with the Electron issue https://github.com/electron/electron/issues/9456 as well as the upstream Chromium issue https://bugs.chromium.org/p/chromium/issues/detail?id=893460 as these are both blockers for VSCode to be able to run on Windows for ARM64..

As an update, somebody successfully built VSCode for Windows ARM64.
http://github.com/ljsabc/Code-OSS-Win32-arm64

As I know, electron started to support windows 10 on arm64 in stable version. Is there an official support of vs code?

@rcjsuen @joaomoreno
Official Windows ARM64 Electron release: https://electronjs.org/blog/electron-7-0
https://github.com/electron/electron/releases/tag/v7.0.0
https://github.com/electron/electron/pull/20112

Surface Pro X is released (I'm using it now), is there any timeline for VS Code for Windows Arm64?

I tracked down their booth and asked them about it at Ignite, they would not give a timeline but basically said that the primary blocker at this point is having the man hours to run the testing to make sure that it works as expected with all of the third party extensions.

That's a fair concern but I'd rather have vscode with no extensions than no vscode.

Sure having ARM64 might make current extensions incompatible, but we need native VSCode to kick start native extension development isn't it?
People can always install both x86 and ARM64 vscode if they have to.

Just Checking on this, purchased a surface pro X and super excited about vscode on this device but would rather have ARM64 than x86

Same is there anything we could do that would help?

Note that if you have a WSL-based workflow, the remoting server in VS Code Insiders now supports ARM64 Linux. This allows using the official x86 frontend app with ARM64 Linux extensions running in the WSL environment, and works well for my PHP+JS projects for now.
However as with a potential ARM64 Windows frontend, not all extensions work yet. I found that the Rust (RLS) and PHP intellisense extensions work ok, while the C/C++ one explicitly fails out saying it's without support for ARM64 Linux.

 I can use a terminal based workflow if I can use vscode Win Arm 64.

@brion  @voronoipotato  How did you even get  an ARM64 front end?  Teach me

Surface Pro X, it's pretty great actually. The major hurdle is just the software side, but otherwise it's very good.

I'm still using the x86 frontend in emulation, so only the extensions inside the WSL Linux sandbox are running natively ARM64 for me. But I've heard some folks have had success manually building the frontend for ARM64?

https://code.headmelted.com/ Perhaps this is a good starting point.

+1, I would love to use VSCode on my Surface Pro X

Any news on this? Both electron and nodejs are building in Windows for ARM now.

For anyone interested, I've put some instruction on how to build vscode for Windows/ARM64, and also a prebuilt binary package of version 1.41.1. Use this until Windows/ARM64 build becomes officially supported.
https://github.com/dsugiyama/build-vscode-win-arm64

With WoA getting pushed by MSFT shouldn't this be a priority? I as a dev would love to have a sleek arm machine running VSCode.

@sunipkmukherjee I agree with you, because if vscode is correctly ported to WoA, then it becomes a real good option for development when combined with vscode remote development. I've been using the 32-bit intel build of vscode for the past two weeks on my Surface Pro X + remote development over SSH to macOS, Linux and Windows, and it works, but it does crash about once every few days, unfortunately.

@awakecoding perhaps we should be filing issues for those to Microsoft support I guess? At least then it could get fixed. It appears that the Code team may be understaffed. I'm not seeing any indication that we will be getting vscode for arm64 any time soon, the pull request appears to be completed and it's not being reviewed/merged. I believe the reason is they aren't sure they have the manpower to review, test and support WoA. While completely understandable from a peer to peer relation (I don't blame vscode devs at all), it's very hard to understand in terms of a customer to corporation relation. This is a tremendous disappointment to me, especially because the surface pro x is a fantastic laptop for the current crisis. We all figured that some apps wouldn't be compiled for it, but it's a bit crushing that Microsoft's own flagship development applications aren't being compiled for it, especially when there is a build ready. Seems like a truly astronomical amount of work and love just being set on fire because of a lack of coordination within Microsoft. By contrast the dotnet core team is building an WoA installer and have a build already available. Why does it appear there was nobody tasked with making sure this initiative was followed through, and resources allocated to make it attainable across the business? It seems like they just said "We're doing WoA" and then tried to put pressure on the devs as though they weren't already doing a job. Maybe they didn't even bring it up with other teams as a thing that mattered.
Who knows... I feel really sad. Sorry to be a bummer in here, I know we're all really excited. I still love my surface pro x, but I'm not convinced that Microsoft feels the same way.

Despite the lack of activity on this thread the ARM64 client is in progress. Maybe it will be released in time for the three year anniversary of my opening this ticket. 😀

Thank you for having greater patience in 3 years than I had in 3 months haha. I suppose all we can do is keep our chin up.

My jaw is dropped with what Edg/Chromium are doing on a Snapdragon 850... Got one of these surprisingly unquirky ARM64 convetible tablets. Still figuring out how to properly get git working here...
This process has many steps:

Build: This has been somewhat achieved in https://github.com/microsoft/vscode/pull/85326, still some work to be done on our side.
Test: We have a test machine on the way, so this will happen soon. I'll post results as soon as possible.
Integrate: This is the hardest part. We rely on Azure DevOps for building and there aren't any hosted ARM64 build agents. A few options here: roll our own build agent or cross-compile this on x64.
Collaborate: We've seen this happen when we released remote for ARM64: our partner extensions are not prepared for this. These are extensions which rely on native components to operate (C#, C++, Rust, etc) and they need to release those components for the ARM64 architecture. We should do a better job this time and let them know ARM64 for Windows is coming, otherwise we're just dumping an onslaught of issues on them as soon as we release. This might take a while.

@joaomoreno for step 3 - integrate, it should be much easier than you think, since the best approach would be to keep your intel x64 build agents and cross-compile to ARM64 using the MSVC build tools. I am not familiar with your build process but for me it was just a matter of modifying my build container to add the MSVC ARM64 components, similar to how to add x86 or x64 build components.

It's not only a matter of compilation but of running as well. We want to run all our test suites on the built bits.

@joaomoreno that's a good point, maybe the upcoming Hyper-V support on Windows ARM64 support (currently in the insider preview) could help instead of using physical hosts to run the tests.

What Electron do is they run the Azure DevOps device runner in x86 mode - seems to work fine for them and they use that to test the ARM64 binary.

Thanks for the update. A few questions:

Just spitting out what needs to happen here.
This process has many steps:

Build: This has been somewhat achieved in #85326, still some work to be done on our side.
Test: We have a test machine (Surface Pro X) on the way, so this will happen soon. I'll post results as soon as possible.

I have a machine. Can we get an officially-built VSCode for ARM64 (even an "alpha"/preview/whatever version) to test too? My machine is from a previous generation (Asus NovaGo, SD835) and I'd like to see that it works there too.
(If you have a test suite you'd want me to run I think I could do that too, if you don't have special environment requirements.)

Integrate: This is the hardest part. We rely on Azure DevOps for building and there aren't any hosted ARM64 build agents. A few options here: roll our own build agent or cross-compile this on x64.
Collaborate: We've seen this happen when we released remote for ARM64: our partner extensions are not prepared for this. These are extensions which rely on native components to operate (C#, C++, Rust, etc) and they need to release those components for the ARM64 architecture. We should do a better job this time and let them know ARM64 for Windows is coming, otherwise we're just dumping an onslaught of issues on them as soon as we release. This might take a while.

I don't know how this works now, with VSCode being available for Windows, Linux and macOS, but assuming you have some method of specifying platform and extensions bring the appropriate native components for each platform, sounds like all you need to do is to add win-arm64 to win-x86-32, win-x86-64, linux-x86-64 and macos-x86-64.
What's the difference? It's completely expected that certain extensions will only work on certain platforms. A WPF designer will only work on Windows. A Swift extension will work on macOS and Linux, but not Windows. That's fine. I don't think any reasonable person expects all extensions to work on all platforms. I'd be satisfied with extensions with native components that can't run on a certain platform to give an early error before downloading and trying to install.
That sounds like something that should already be in place. I don't see why we should wait for extensions X, Y and Z to support ARM64 when other things are already working.

I don't know how this works now, with VSCode being available for Windows, Linux and macOS, but assuming you have some method of specifying platform and extensions bring the appropriate native components for each platform, sounds like all you need to do is to add win-arm64 to win-x86-32, win-x86-64, linux-x86-64 and macos-x86-64.

We don't build the C#, C++, Rust extensions. Our partners do that. Extension authors have to be onboard, otherwise their features will just be broken. This has happened before when releasing the ARM64 remote for Linux. We don't have to wait for that to happen, but we should to a better job at collaborating with them to let them know a torrent of issues is about to fall on top of them.

@joaomoreno : It would be great if Microsoft could release the Win Arm64 version of VS Code soon. Having it working on Surface Pro X would be the last program I need on this machine to make it 100% usable for my daily work. Pls make it happen soon. Many thanks.

@joaomoreno To be clear the x86 emulation is good enough their extensions will "work" provided it pulls x86 instead of x64. I agree though that it is up to the extension authors and their community (us) to help update everything for ARM should we need to.

@joaomoreno To be clear the x86 emulation is good enough their extensions will "work" provided it pulls x86 instead of x64. I agree though that it is up to the extension authors and their community (us) to help update everything for ARM should we need to.

Clarification, Windows on ARM supports emulating x86_32, but does not support emulating 64 bit x86_64 apps. Native apps are either ARMv8_64 or ARMv7_32 apps, publishing ARM32 is usually unnecessary... unless you're targeting older Raspberry Pi's which don't have the memory to run VS Code properly anyway or Windows RT  which is dead. Windows 10 on ARM doesn't come in a 32 bit variant.

It's not only a matter of compilation but of running as well. We want to run all our test suites on the built bits.

Would it help if a Qemu Windows ARM64 environment was set up to run the tests?

What Electron do is they run the Azure DevOps device runner in x86 mode - seems to work fine for them and they use that to test the ARM64 binary.

I wanted to understand how exactly the Electron team did this, and it seems like they literally set up a Surface device (I guess a Surface Pro X) which they use for Azure DevOps. Like @richard-townsend-arm mentioned, they should be running the self-hosted x86 Azure DevOps agent as there's no arm64 version for it, but it works for them apparently.
I'm working on adding Windows ARM64 support to GitHub Desktop and wanted to try the same, so I installed the Azure DevOps x86 agent on my Surface Pro X and created an Azure Pipeline. So, the setup is:

Windows 10 ARM64 host (Surface Pro X)
32-bit NodeJS + VS Build Tools (ARM64 build for NodeJS is in the works)
Set npm_config_arch=arm64 and TARGET_ARCH=arm64 so that ARM64 binaries (like Electron ARM64) are downloaded

This seems to work well. An ARM64 build is created without issues, I only need to make sure that the tests will also pass, but that's unrelated to ARM64 specifically.

Would it help if a Qemu Windows ARM64 environment was set up to run the tests?

That might work, but I expect it to be incredibly slow. Running on a native ARM64 device already timed out after 1hr because the 32-bit emulation on Windows ARM64 slows things down significantly. Also, there's no Windows 10 ARM64 ISO files available via Microsoft's official channels AFAIK, so you'll need to do some magic yourself :) Instructions to create a Windows 10 ARM64 ISO can be found on this page, just ignore the Raspberry Pi-specific bits.

@dennisameling That is pretty much my plan as well, once we get that Surface Pro X. Glad to know you had success. Good news for this plan item.

Squirrel is now looking into this: https://github.com/Squirrel/Squirrel.Windows/issues/1616

And... we have liftoff!

User Setup: https://update.code.visualstudio.com/latest/win32-arm64-user/insider
System Setup: https://update.code.visualstudio.com/latest/win32-arm64/insider
ZIP: https://update.code.visualstudio.com/latest/win32-arm64-archive/insider

Announcement: https://twitter.com/joaomoreno/status/1266005299661545474

Special thanks to @richard-townsend-arm for pushing this forward in https://github.com/microsoft/vscode/pull/85326 👏

@deepak1556 Care to verify this one?

Sure happy to 👍

FWIW,  perhaps the team at Microsoft can look into Microsoft-made extensions and ARM support...  -- e.g. https://github.com/microsoft/python-language-server/pull/2073


Empty package in npm

Hey! I believe it's error, that last version in npm just have one file in it with package.json.

See #1. Contributions are welcome!


As a developer I want to make sure that I am leveraging the linter applied on the master branch to cleanse my code.

As a developer I want to make sure that I am leveraging the linter applied on the master branch to cleanse my code.

@devlace to comment on which linter we'll be using

Linter used: https://github.com/tcort/markdown-link-check
Github action that does the linter: https://github.com/gaurav-nelson/github-action-markdown-link-check
Github workflow: https://github.com/Azure-Samples/modern-data-warehouse-dataops/blob/master/.github/workflows/documentation.yml


feat(compiler): allow custom transformers to access internal Program

Summary

Allow custom transformers to access internal ts-jest TypeScript Program
Test plan

Adjusted unit tests, green CI
Does this PR introduce a breaking change?

[x] Yes
[ ] No

ts-jest custom AST transformer function signature has changed to
import type { TsCompilerInstance } from 'ts-jest/dist/types'

export function factory(compilerInstance: TsCompilerInstance) {
  //...
}

Other information
N.A.

Pull Request Test Coverage Report for Build 510474279

20 of 22 (90.91%)  changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.1%) to 93.515%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/compiler/ts-compiler.ts
3
5
60.0%

Totals

Change from base Build 508753501:
-0.1%

Covered Lines:
943

Relevant Lines:
975

💛  - Coveralls

Pull Request Test Coverage Report for Build 510474279

20 of 22 (90.91%)  changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.1%) to 93.515%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/compiler/ts-compiler.ts
3
5
60.0%

Totals

Change from base Build 508753501:
-0.1%

Covered Lines:
943

Relevant Lines:
975

💛  - Coveralls