[semantic highlight] Add basic language tokens

Looking at HighlightingKind, clangd's semantic highlighting seems to be missing a lot of basic tokens (e.g. strings, numbers, operators). Because of that, highlighting based on this alone looks very barren:

VSCode doesn't look as broken, because it seems to combine semantic highlighting with its regular textmate highlighters. But it would be nice if just clangd would be enough for fully-featured highlighting, especially for use with other editors.
For reference, Rust Analyzer has a fairly extensive list of tokens/modifiers.

Based on previous discussions, e.g. https://github.com/clangd/clangd/issues/352#issuecomment-619442147:

Things that are easily lexically derived (such as preprocessor defines and doxygen tags) don't need to be in the language server I think. At least, we haven't tried so far to make the coloring exhaustive, because we assume it's always a layer over syntax highlighting. Compared to syntax highlighting, LSP based syntax highlighting is asynchronous and imposes extra costs like serialization.

it seems unlikely that we're going to do this in clangd.

we assume it's always a layer over syntax highlighting

I see.
The flip-side is that in order to get this "lower layer syntax highlighting" an editor can either

pull in a second instance of the dedicated language parser, e.g. tree sitter, which then has the same latency 'issues' as clangd, but cannot share with it
have a generic, approximate parsing system (textmate, regex, ...) which, in my experience, doesn't get to the same level as a dedicated parser

But I guess I'll have to look into implementing tree-sitter syntax highlighting for this, then.

have a generic, approximate parsing system (textmate, regex, ...) which, in my experience, doesn't get to the same level as a dedicated parser

In my experience, the C++ TextMate grammar at https://github.com/jeff-hykin/better-cpp-syntax (used by vscode) is pretty accurate for the things clangd's semantic tokens don't cover.
There are a few template edge cases like https://github.com/jeff-hykin/better-cpp-syntax/issues/370 that it gets wrong, but they're actually caused by that grammar trying too hard (wanting to provide reasonable heuristic colors for identifiers for non-LSP use cases).
I'm pretty sure a C++ TextMate grammar designed to only highlight non-semantic token types could nail it and be very accurate.

pull in a second instance of the dedicated language parser, e.g. tree sitter, which then has the same latency 'issues' as clangd, but cannot share with it

IMO tree-sitter is overkill in comparison to TextMate, but tree-sitter will still have better latency than clangd's semantic tokens, at least on file-open, since by design it does not need to process the contents of #included files.

Do you have tests/estimates of how much additional tokens "cost"? Rust Analyzer provides tokens for pretty much every character in a file, and it seems to be doing just fine on performance in my experience.

I don't; it would be interesting to see some numbers.
I personally have no objection to giving clangd an option to provide semantic tokens for all C++ syntax. From an implementation point of view, I expect the missing tokens are all "lexical" (i.e. correspond to tokens in TokenBuffer, with token types no more granular than C++ lexical token types), and so could be provided relatively easily. May want to run this by @sam-mccall before spending time on it though.

we assume it's always a layer over syntax highlighting

I see.
The flip-side is that in order to get this "lower layer syntax highlighting" an editor can either

pull in a second instance of the dedicated language parser, e.g. tree sitter, which then has the same latency 'issues' as clangd, but cannot share with it
have a generic, approximate parsing system (textmate, regex, ...) which, in my experience, doesn't get to the same level as a dedicated parser

But I guess I'll have to look into implementing tree-sitter syntax highlighting for this, then.

I have an experimental vscode extension Minimal C++ Syntax which does exactly what @HighCommander4 mentioned above, which works pretty well for me (though it's not that mature and well-tested). Excluding semantic tokens that are already colored by the clangd, the remaining tokens are pretty easy to recognize with regex. Basically, what we need to match are mostly context-insensitive tokens like comments, strings, constants, keywords and .etc, which could be done with a few hundred lines of textmate rules.
I find that the pain points are preprocessors, especially function-like macros, and obfuscated program with backslash or other preprocessing tricks. But they are really minor in day-to-day sources.

I have an experimental vscode extension Minimal C++ Syntax

That's great for you, but doesn't help me or lots of other people. Going through this thread, I always see "VScode has this", "VSCode can do that". But VSCode is not the only text editor with LSP support out there. And not all of them support TextMate.
As far as I'm aware, the reason why there is a "P" in LSP is because its goal is to provide IDE-like features to any editor that has a client for it, while also making features and configuration uniform across editors. And syntax highlighting is one of those features that the protocol supports.
But the way clangd currently implements it requires every one of those editors to also implement syntax highlighting. And every theme creator/user has to do extra work to configure both and make them play nicely together.

I agree that clangd should follow the LSP, and cater to all clients not just VSCode (and we do try -- we have users happily using other clients such as vim, emacs, Sublime Text, and others).
However, the name of the coloring feature in the LSP is "semantic tokens", and its description is as follows (emphasis mine):

Semantic tokens are used to add additional color information to a file that depends on language specific symbol information.

My understanding is that this has been fairly widely interpreted in the LSP community to mean color information that requires the full semantic model built by a language server (e.g. classification of identifier tokens as types vs. variables), which are applied on top of client-side coloring of basic language syntax such as keywords.
It's possible I'm off base here, and perhaps it's worth seeking clarification in the LSP repo. If the LSP maintainers clarify that the intention is for semanticTokens to cover all tokens in a file, I think that would be a strong argument for clangd to do so by default.
Otherwise, my personal opinion remains that it may be appropriate for clangd to have an option to do this, but should not do it by default.

Thank you for the quote, I wasn't aware of that specific wording.
But in the same section, the "predefined values" already include some of the token types I felt missing in clangd, such as string or number, which would be some of those "just syntax" kind.
As for the LSP maintainers, I just found an existing discussion at https://github.com/microsoft/language-server-protocol/issues/1063, but without response.
An off-by-default option in clangd would be fine for my needs.

I just noticed: The upcoming spec version has a capability that's very close to the mentioned option:
/**
 * Whether the client uses semantic tokens to augment existing
 * syntax tokens. If set to `true` client side created syntax
 * tokens and semantic tokens are both used for colorization. If
 * set to `false` the client only uses the returned semantic tokens
 * for colorization.
 *
 * If the value is `undefined` then the client behavior is not
 * specified.
 *
 * @since 3.17.0
 */
augmentsSyntaxTokens?: boolean;

I'd be happy to see this in as an option if someone wants to contribute it. On the other hand I/others can't really prioritize it if no widely used editors will benefit. So I think we call this one "patches welcome".
I agree augmentsSyntaxTokens sounds exactly like the option we need, so I don't think we need to add an explicit flag.
When augmentsSyntaxTokens is unset we should keep the current behavior.
Potential costs are:

server-side complexity: likely to be smallish as we can basically just dispatch on token-type, right?
server-side performance: I doubt 3x more tokens or so will make a difference
client-side performance: manipulating large highlight sets for big files can be really slow (I know this is true in vim). big JSON-wrangling can be somewhat slow too in (bad) extension languages.
bandwidth: we and others run browser-based monaco with clangd, so request/response sizes matter.
maintenance: there's bound to be edge-cases or disagreements about how we classify tokens.

Only the last is really a problem if editors are opting in.

I just noticed: The upcoming spec version has a new client capability that's very close to the mentioned option:

Nice find, I think this makes the spec's intention clear.
I gave a suggestion for a high-level implementation approach in a previous comment:

From an implementation point of view, I expect the missing tokens are all "lexical" (i.e. correspond to tokens in TokenBuffer, with token types no more granular than C++ lexical token types), and so could be provided relatively easily.

Happy to provide more detailed guidance if someone is interested in working on this.

@HighCommander4 Is this being worked on by anybody? If not, I can spend some time on this.
Here's my random ideas:

As the missing tokens are lexical and well-defined by itself or its immediate surroundings, we should have a dedicated scanner which directly operates source text instead of AST. The semantic tokens could be computed from the scanner and merged into current semantic highlighter. Maybe this could run in parallel with the semantic highlighter.
To reduce the amount of redundant computation, we should implement provideDocumentRangeSemanticTokens so those lexical tokens could be computed within a small range of source code instead of the entire source. My understanding on the current semantic highlight implementation is that we iterate the entire AST and trim the tokens by doing diff. Am I correct?
We should still leave some tokens to syntax highlight to allow users to take advantage of language injection. For example, people may write glsl/hlsl shader code in string literals. And some extension may color text in comments accordingly. An option is to have each token type configurable.

@HighCommander4 Is this being worked on by anybody?

Not that I'm aware of.

1. As the missing tokens are lexical and well-defined by itself or its immediate surroundings, we should have a dedicated scanner which directly operates source text instead of AST. The semantic tokens could be computed from the scanner and merged into current semantic highlighter. Maybe this could run in parallel with the semantic highlighter.

Agree that we don't need the AST to provide the missing tokens. We have the token stream from the lexer available as a TokenBuffer, I would recommend using that. I believe providing lexical highlighting tokens should be a fairly simple matter of iterating through the tokens in the TokenBuffer, selecting the subset for which we want to provide a lexical highlighting, and performing a simple mapping from the clang token kind to the highlighting kind.

2. To reduce the amount of redundant computation, we should implement `provideDocumentRangeSemanticTokens` so those lexical tokens could be computed within a small range of source code instead of the entire source. My understanding on the current semantic highlight implementation is that we iterate the entire AST and trim the tokens by doing diff. Am I correct?

LSP provides two different ways to reduce the overhead of semantic tokens requests:

"Delta" requests, where the server only sends the diff between the last request and the current one, and the client applies this diff to the result of the last request which it retains.
"Range" requests, where the client only requests tokens for a subset of the file (presumably, the portion currently visible) and the server only sends tokens in this range.

Both are optional and will be used if available. Clangd supports (1) but not (2) (I believe we felt (2) would not bring a significant performance improvement for the complexity.) I don't think this needs to change for providing lexical tokens.

3. We should still leave some tokens to syntax highlight to allow users to take advantage of language injection. For example, people may write glsl/hlsl shader code in string literals. And some extension may color text in comments accordingly. An option is to have each token type configurable.

I don't have a strong opinion on this; having knobs to disable particular tokens like string literals to allow another plugin to provide finer-grained tokens for their contents (like your mentioned shader example) sounds fine to me.

I believe providing lexical highlighting tokens should be a fairly simple matter of iterating through the tokens in the TokenBuffer, selecting the subset for which we want to provide a lexical highlighting, and performing a simple mapping from the clang token kind to the highlighting kind.

I think LSP expects tokens to be on a single line, so we'll probably need to split multi-line tokens into multiple single-line tokens.

I think LSP expects tokens to be on a single line, so we'll probably need to split multi-line tokens into multiple single-line tokens.

It's up to the client to support (and report) that:

Whether a token can span multiple lines is defined by the client capability multilineTokenSupport. If multiline tokens are not supported and a tokens length takes it past the end of the line, it should be treated as if the token ends at the end of the line and will not wrap onto the next line.
https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_semanticTokens

As the missing tokens are lexical and well-defined by itself or its immediate surroundings, we should have a dedicated scanner which directly operates source text instead of AST. The semantic tokens could be computed from the scanner and merged into current semantic highlighter. Maybe this could run in parallel with the semantic highlighter.
Agree that we don't need the AST to provide the missing tokens. We have the token stream from the lexer available as a TokenBuffer, I would recommend using that. I believe providing lexical highlighting tokens should be a fairly simple matter of iterating through the tokens in the TokenBuffer, selecting the subset for which we want to provide a lexical highlighting, and performing a simple mapping from the clang token kind to the highlighting kind.

Yes, it would be great if spelled TokenBuffer could be reused for this purpose. However, there's a problem with the approach because of preprocessor. It seems raw lexer will provide incorrect token type for preprocessing directives and eliminate the boundary of that, which makes it hard to highlight the preprocessors.

To reduce the amount of redundant computation, we should implement provideDocumentRangeSemanticTokens so those lexical tokens could be computed within a small range of source code instead of the entire source. My understanding on the current semantic highlight implementation is that we iterate the entire AST and trim the tokens by doing diff. Am I correct?

LSP provides two different ways to reduce the overhead of semantic tokens requests:

"Delta" requests, where the server only sends the diff between the last request and the current one, and the client applies this diff to the result of the last request which it retains.
"Range" requests, where the client only requests tokens for a subset of the file (presumably, the portion currently visible) and the server only sends tokens in this range.

Both are optional and will be used if available. Clangd supports (1) but not (2) (I believe we felt (2) would not bring a significant performance improvement for the complexity.) I don't think this needs to change for providing lexical tokens.

I experimented with a hack into clangd to support range request. it looks it's not worth it to support the range request because:

The major bottleneck of a semantic token request is waiting for building AST instead of traversal.
The client may not take advantage of range request at all. VSCode will only issue a couple of range requests at opening the file. After that, only full request will be sent as long as it's supported.

Here, I'd like to list and clarify the features that may/may not be wanted.
Lexical Tokens
There are roughly the following lexical tokens in the language

comments
keywords
number literal
char literal
string literal
label

Furthermore, it might be helpful to add modifier to refine a lexical token for better theming:

Preprocessor
Preprocessor definition is technically not that lexical because its definition is more structured. However, since they are part of the language (or a meta language on its own), I expect they should be properly highlighted as well. Including:

preprocessor directive
pragma commands
quoted/unquoted string
defined(...), __has_include(...) and .etc
line-continuation symbol
...

And maybe we could implement something regex-based syntax highlight cannot do. Like highlight macro parameters in the macro definition (can X in "X::Y()" be assigned with a semantic token?).

Notably, those are features that can't be easily implemented with the existing TokenBuffer.

Lexical Tokens
There are roughly the following lexical tokens in the language
* comments

* keywords

* number literal

* char literal

* string literal

* label

Furthermore, it might be helpful to add modifier to refine a lexical token for better theming:

This sounds reasonable to me, and should be implementable using TokenBuffer.

Preprocessor
Preprocessor definition is technically not that lexical because its definition is more structured. However, since they are part of the language (or a meta language on its own), I expect they should be properly highlighted as well. Including:
* preprocessor directive

* pragma commands

* quoted/unquoted string

* defined(...), __has_include(...) and .etc

* line-continuation symbol

* ...

For this, I think we can use clangd's existing PPCallbacks. We can store any additional information we need for highlighting (e.g. the bodies of macro definitions) in MainFileMacros, which is then available in the semantic highlighter as AST.getMacros().

And maybe we could implement something regex-based syntax highlight cannot do. Like highlight macro parameters in the macro definition (can X in "X::Y()" be assigned with a semantic token?).

This sounds useful, and it should be possible. High-level sketch:

Given a MacroInfo representing a macro definitions, loop through its ReplacemenTokens
If it's an identifier token, check if it corresponds to a macor parameter by calling getParameterNum(IdentifierInfo*) and if so highlight it

Lexical Tokens
There are roughly the following lexical tokens in the language
* comments

* keywords

* number literal

* char literal

* string literal

* label

Operators and punctuation should be added as well.


turtlebot3 Burger wifi not connecting, Noetic.

ISSUE TEMPLATE ver. 0.4.0

Which TurtleBot3 platform do you use?

[ x ] Burger
[ ] Waffle
[ ] Waffle Pi

Which ROS is working with TurtleBot3?

[ ] ROS 1 Kinetic Kame
[ ] ROS 1 Melodic Morenia
[ x ] ROS 1 Noetic Ninjemys
[ ] ROS 2 Dashing Diademata
[ ] ROS 2 Eloquent Elusor
[ ] ROS 2 Foxy Fitzroy
[ ] etc (Please specify your ROS Version here)

Which SBC(Single Board Computer) is working on TurtleBot3?

[ ] Intel Joule 570x
[ x ] Raspberry Pi 3B+
[ ] Raspberry Pi 4
[ ] etc (Please specify your SBC here)

Which OS you installed on SBC?

[ x ] Raspbian distributed by ROBOTIS
[ ] Ubuntu MATE (16.04/18.04/20.04)
[ ] Ubuntu preinstalled server (18.04/20.04)
[ ] etc (Please specify your OS here)

Which OS you installed on Remote PC?

[ ] Ubuntu 16.04 LTS (Xenial Xerus)
[ ] Ubuntu 18.04 LTS (Bionic Beaver)
[ x ] Ubuntu 20.04 LTS (Focal Fossa)
[ ] Windows 10
[ ] MAC OS X (Specify version)
[ ] etc (Please specify your OS here)

Specify the software and firmware version(Can be found from Bringup messages)

Software version: [x.x.x]
Firmware version: [x.x.x]

Specify the commands or instructions to reproduce the issue.

I followed the SBC Noetic setup on the emanual.

Copy and Paste the error messages on terminal.

HERE

Please describe the issue in detail.

rpi doesn't connect to wifi. I use the same wifi for my turtlebot3 waffle pi (rpi4, ROS2 Foxy), and it works fine. I have made the regular adjustments to  "50-cloud-init.yaml" file.

Imager =>           "

imager_1.7.2_amd64.deb"

@mayBeAutomata
In case the Raspberry Pi and your wifi router uses different WiFi channel, this could happen.
Please try setting your country code in the /etc/default/crda file as below.
By default the REGDOMAIN is empty.
REGDOMAIN=US

Noted. Will try that and update you ASAP.

Issue Resolved.


Add a transparent, fixed overlay over screen when tooltip is opened

When a tooltip is opened, we'll create a transparent, fixed overlay over the entire viewport to make sure outer clicks don't do anything but hide the tooltip.
.overlay::before {
  position: fixed;
  top: 0;
  right: 0;
  bottom: 0;
  left: 0;
  z-index: 80;
  display: block;
  cursor: default;
  content: " ";
  background: transparent;
}

Closed as you can easily add this css to your custom content.


add lsu to list of data_type

A user requested to add LSU as a possible data type.
We will need to:

[ ] Add entry to database: qiita.data_type
[ ] Add to target gene processing (not sure about this one?)
[ ] If the previous one is needed, also change documentation and code (TARGET_GENE_DATA_TYPES in qiita_db.metadata_template.constants)

LSU needs to be added to target gene. This is the reason why I was proposing to leve it as a target gene in general and provide a different way of subsetting it. The main issue is that there are a lot of different primers targeting different genes and in the current behavior we need to list all of them, whether the only thing that we care about is that it is target gene.

Yes, I see your point. Other simpler options are have it in the config file but the downside is that we will need to restart the system to reactivate a new option; or as a new column in the data_type, for example: data_type_type.

I think we should step back and ask ourselves the following question:
Do we care which target gene was used?
In my opinion, no we don't. What we actually care for analysis is the parameters that where use for processing. For example, if you do a closed reference using GG but one of your runs was v4 and the other was v3-5 you can still compare them, since they've been generated closed reference. Then in your analysis you'll need to address if the different primers is introducing a bias or not.

For the way it currently stands, no we don't, and I think is fine. However, it's a nice separation to have as users will like to know which one is 16S, 18S, etc. Moreover, this is information we need to send to EBI. Thus, we need to keep track of it but not sure if having the separation is needed.

I think it is ok to provide a way for the user to differentiate the different datatypes. As far as I know, the information in the data type column from the database is not submitted to EBI, as that information is in theory represented with the primer used. It may be worth double checking.
That being said, I think the current data type should be changed to be only target gene, and then use something else (open to ideas) to keep track of the specific subtype. One idea might be to create another column "sub-type" to keep track of these, but I'm unsure if we will need more recursive subtypes for this and then everything gets messy.
Another idea is to represent this data as an ltree in the database, and just look at the root of the ltree to knew which pipeline to use. This will still require us to add a new row for each new subtype that we might need, but is just data added to a table and no changes in code are required. I think I will vote for this option, but I'm open to more ideas.


Enhanced permission management

Actions before raising this issue

[X] I searched the existing issues and did not find anything similar.
[X] I read/searched the docs

Is your feature request related to a problem? Please describe.
I think administrators can control the management of dataset download switches on the UI interface of the management backend

Describe the solution you'd like
Perhaps it can be implemented in two forms - some ideas

Administrators can set the available or unavailable status of the permission button switch for each user to download the dataset
Administrators can globally set the available or unavailable status of the export button for the entire client UI interface dataset

Describe alternatives you've considered
No response
Additional context
No response

This will have flexible management methods for some data security issues

@KTXKIKI Hello, did you find another solution to restrict downloading data for worker roles?
Thank you in advance.


Changed structure readme and install option

@floriansemm Ok, removed the instructions for use in Symfony 2.1. https://github.com/royopa/SolrBundle/commit/cb7241401f3465e481bc71210d8f66ee7cbe8e1c

Thanks! Great work!


Failed to compile when using ArweaveSigner

Hi there,
Failed to compile Typescript-React project.
Reason:
const signer = new ArweaveSigner(ARWEAVE_WALLET_KEY);
Failed to compile
./node_modules/arbundles/build/signing/keys/Rsa4096Pss.js 16:6
Module parse failed: Unexpected token (16:6)
File was processed with these loaders:

./node_modules/babel-loader/lib/index.js
You may need an additional loader to handle the result of these loaders.
|
| class Rsa4096Pss {

_key;
|   pk;
|   signatureType = 1;

Stupid me, when compiling for node instead of browser everything is great!


No FSR - NIS option available.

There's a tool to improve even better the VR performance in rFactor2 (or almost any OpenVR game), Open Fsr (https://github.com/fholger/openvr_fsr)
Given that's a dll and a config file if can be easily integrated to this wonderful tool, and improve the performance in VR even further.

"Easily" is a bold claim ;P But here you go:
github.com/tappi287/rf2_video_settings/releases/latest


Improve italian translation

"da ora" is a very literal translation of "from now", and not much used in this sense - strictly speaking, it could even mean "ago".

Thanks!


⚠ Secret Website 3 has degraded performance

In 7cad13b, Secret Website 3 ($SECRET_WEBSITE_3) experienced degraded performance:

HTTP code: 200
Response time: 2765 ms

Resolved: Secret Website 3 performance has improved in 90b0d10.


[5.10] Update allowed years in check-source script

Bug/issue #, if applicable:
Summary
This updates the allowed years in check-source script to include 2024.
Note that this isn't a cherry-pick because this change was made in #778 which we didn't cherry-pick onto the 5.10 release.
Dependencies
None.
Testing
Run check-source, it shouldn't warn about the year in the check-source file itself.
Checklist
Make sure you check off the following items. If they cannot be completed, provide a reason.

~[ ] Added tests~
[x] Ran the ./bin/test script and it succeeded
~[ ] Updated documentation if necessary~

@swift-ci please test


Add support for VK_KHR_external_memory for inter-device memory access (depends on VK_KHR_external_memory_capabilities)

https://www.khronos.org/registry/vulkan/specs/1.0-extensions/html/vkspec.html#VK_KHR_external_memory

An application may wish to reference device memory in multiple Vulkan logical devices or instances, in multiple processes, and/or in multiple APIs. This extension enables an application to export non-Vulkan handles from Vulkan memory objects such that the underlying resources can be referenced outside the scope of the Vulkan logical device that created them.

The different logical devices part makes it possible for Vulkano users to implement multi-device renders etc without performing on-CPU inter-device buffer copies which are only suitable in very few cases and impose high load on CPU.

I think this should be low priority

For those still interested, here's the first stab at this:
https://github.com/vulkano-rs/vulkano/pull/1467

For those still interested, here's the first stab at this:
https://github.com/vulkano-rs/vulkano/pull/1467

This is now more thoroughly implemented, although there's still room for adding more platforms and memory types.


error running notification service

Error running notification service: ...opt/nvim-notify/lua/notify/service/buffer/highlights.lua:118: a
ttempt to call method '_get_hl_from_capture' (a nil value)

Any chance you're using Tokyonight? I merged some PR's yesterday and one of them caused this issue for me. Fixed my issue already on Tokyonight.
The cause for me was TelescopeNormal having None as bg.

It's due to a breaking change in nightly, should be working now :smile:


Tempest compatibility issue.

ServersClient does not provide get_client anymore. I changed to call to
show_client. This was causing smoke.test_hostname_set to fail.

Tested and this does fix the issue.


propojení Genetiky

We want to add another MMCI workspace - Genetics.
Things we know:

They use MiSeq
They have a different analysis tool (FinalistDX)
They want to store the data (grouped by sequenced families)
They want to join catalogue

Things we don't know:

How to connect the sequences with bbm and clinical data
How much they want to access the data once they are on SC
What is their workflow
@RadoslavaKacova will ask these questions on next meeting

News from Meeting 31.1.2024:

they work with blood/DNA (saliva too, but this is not in BBM so we wont have consent)
They use CZECANCA gene panel (only one - easier then in patology)
Capa hypercap target enrichment kit
The computer for analysis (FinalistDX) is outside of MOU net
Can we separete Seq data from
We are still not sure how to connect clinical data/patient with seq data
They want to use the data storage as archive, they would need to access the data once every 3  months let's say
We need to find out, if RATIONAI group finally made sence of their data. Ask HOPET
We would want to logically separate Pathology and Genetics


Webpack code splitting with require.ensure()

Is it currently possible to build an application that makes use of webpack code splitting with require.ensure() to render routes and be able to do static rendering of these routes as well? If not, what is needed to make this happen and is it planned? Is there a workaround?
I saw #31 but this seems to only make sure require.ensure() can occur in compiled code at all - but using it inside the render method used by the static rendering plugin is not possible?

We're trying to implement server-side rendering on top of code splitting inside React Router but running into the same issue.
I noticed the React Router example was removed recently -- is code splitting still supported by this library to achieve statically rendered markup?

It's possible; I'm using code splitting with react-router, i run two times webpack, the first for build the client, and the second for build all static pages. how it works? by using LimitChunkCountPlugin.
Project with code splitting


Validate Slot address in storage contracts

fix #48
This PR uses the AST already found in hardhat dbg artifacts, and parsing the AST manually.
A second iteration would be to "recompile" the contracts to get the AST from solc and use some AST parsing library.

Several non-minor comments. My recommendation is that you use this as the entry point sub-task for a series of AST validations, clearly enumerated in this entry point. And that you don't write the AST traverser yourself.


Make it easier to specify a FieldMask for CallSettings

(The user doesn't need to know the right header.)
Fixes #189 (although we may want more later.)

Whoops - looks like the tests fail because I didn't know about lower-casing of headers. Will fix that.


Slack bot app creation has changed such that screenshots and steps are no longer valid

Starting at the step "Create a Slack Bot User" the screenshots and steps are no longer valid.
The following steps are now inconsistent with the Slack interface:

Where "Bot Users" was an option on the left hand menu, there is now a "Where's Bot User" option. Clicking this explains that Bot User has moved to "App Home".
At this point the user is required to make a "Bot Token" and create Scopes (which is reference in #1831)
The bot is then given a default display name and default name based on the name of the app.

The remainer of the documentation seemed to go as expected.
I suggest the note at the end of the document related to scopes be moved higher to draw users attention to it. I didn't read it as I thought it was related to the "Create an Add to Slack button" section.
Thank you for all your hard work keeping the MS docs awesome!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 97b7eb7d-843a-774e-3ee4-8f3126cfea20
Version Independent ID: 5d501ffc-819f-0634-671c-13d432aa7f28
Content: Connect a bot to Slack - Bot Service
Content Source: articles/bot-service-channel-connect-slack.md
Service: bot-service
GitHub Login: @arturl
Microsoft Alias: arturl

I'm hoping to get to this article in the next week or two, to address this issue, as well as issue 1831.

The fix will be available in the next documentation update.


Fix file limit for future parser

The validate_re() function takes a string as first argument, so passing in an integer is invalid.

that feels… wrong…
i'm sure that @hlindberg would be (or was) against adding another validate_number() or validate_integer() to puppetlabs-stdlib, but this just feels wrong.

So consequently the correct solutions is to set file_limit to be a string as I'm suggesting in this PR. This is also consistent with how the ports and all other parameter with numeric values are configured.
In the end this is about setting a value in a file, so passing it in as a string containing a numeric value seems correct.

Actually, it makes total sense that you can't pass an integer to validate_re. You can't use regexes on anything but strings so the behaviour is pretty correct.
The nice way to solve this in Puppet right now is to leave it as an integer and validate it as such:
unless is_integer($param) {
  fail('$param must be an integer')
}

Again, I think setting this as a string containing a double is the correct thing to do here.

A double is a double, not a string containing a double and should be validates as being a double. This isn't (supposed to be) a stringly typed language.
A good argument would be "because any item in limits can also take the values of -1, unlimited or infinity except for priority and nice" as pointed out by man limits.conf.

I agree with @daenney

This isn't (supposed to be) a strongly typed language.

That is the direction we're going though. https://github.com/puppetlabs/puppet-specifications/blob/master/language/types_values_variables.md

No we're not. If anything we're moving away from it. We now actually have a language with a Type system, that respects and enforces the types given. See the Puppet Types section alone.

I see, I misinterpreted 'stringly' as a typo for 'strongly'.

So according to the conversation above I changed the following:

apart from a positive integer  the values -1, infinity and unlimited should be allowed as valid parameters
consequently file_limit has to be checked via validate_re and the default can't be an integer

How is infinity different from unlimited? (We use 'unlimited' for things like environment timeout). If there is no difference between what they do I suggest dropping 'infinity' since all that can be achieved is 'no limit imposed' vs 'to the end of time' (which hopefully will be long after the machine has turned to silicon dust).

From my understanding they are functionally the same. Citing man 5 limits.conf
All items support the values -1, unlimited or infinity indicating no limit, except for priority and nice.

But I think going with exactly what ulimit actually supports as the better approach than arbitrarily choosing a subset.

Yes agree, add both since the thing being modeled has both values.

closed by #319


docs: Suggest an branch for users upgrading to get latest features.

Still waiting to decide on what to call the branch; put weekly for now.

Closing, since doesn't seem like a direction we're moving towards.


should prefercliruntime be added?

See https://github.com/fable-compiler/Fable/issues/961

@ctaggart your deps for https://www.nuget.org/packages/dotnet-sourcelink/ is Microsoft.NETCore.App (>= 1.0.4), so shouldnt matter atm.
Just check if you update the sdk you use to build dotnet-sourcelink because with .NET Command Line Tools (1.0.4) the bundled is Microsoft.NETCore.App 1.0.5 (so deps generated will be Microsoft.NETCore.App (>= 1.0.5) , like in fable case), and that's when the issue happen


Support alternate OAuth2 token passing via  "Authorization: Token " and "Authorization: Basic "

Looks like a hack, but is needed to support Docker client auth in Pier One

Works for Pier One :-)


VRAM memory measurements should be process specific

Currently GPU (nvidia and amd) memory measurements reflect the usage of the whole system, however using nvml and amdsmi it should be possible to only capture a process's vram memory usage.

done in #124


firewallbypass patched (x2)

:sob: :sob: fairewalbipas no woirk 😱

does work, however it no longer gives a speed boost. only float disabler.

oh ok


Absolute taxonomic abundance analysis

For new term requests, please provide the following information:
Term Label [required]
Absolute taxonomic abundance analysis
Synonyms [optional]
Definition [required except for organism name]
it is an internal node so we don't show the definition.  You can use something similar to what is there for "relative taxonomic abundance analysis"
Suggested Parent Term [required]
Please look in the ontology hierarchy in a browser such as OLS or choose one from following list:
Everywhere we currently have: "Relative taxonomic abundance analysis", please make this a sibling term
Examples [optional]
Contributor [optional]
If you would like to be a contributor of the term or anyone you think that contributes to the term, such as label, definition, please provide the name and ORCID ID.
Requester [optional]
Please indicate which resource/database/project/ request the term.

Term Label:
absolute taxonomic abundance analysis
Definition:
A data transformation of input sequence data to obtain the fraction of absolute abundance of organisms.
Parent:
data transformation
Definition source: MicrobiomeDB
Requester: MicrobiomeDB
Reserved ID: EUPATH_0009350
Also need following related terms, which are subClassOf 'data item' and is_specified_output_of some 'absolute taxonomic abundance analysis'
Term Label: absolute abundance of kingdom data
Definition: A data item that is the output of an absolute taxonomic abundance analysis for organisms grouped at the level of kingdom.
Definition source: MicrobiomeDB
Requester: MicrobiomeDB
Reserved ID: EUPATH_0009351
Term Label: absolute abundance of phylum data
Definition: A data item that is the output of an absolute taxonomic abundance analysis for organisms grouped at the level of phylum.
Definition source: MicrobiomeDB
Requester: MicrobiomeDB
Reserved ID: EUPATH_0009352
Term Label: absolute abundance of class data
Definition: A data item that is the output of an absolute taxonomic abundance analysis for organisms grouped at the level of class.
Definition source: MicrobiomeDB
Requester: MicrobiomeDB
Reserved ID: EUPATH_0009353
Term Label: absolute abundance of order data
Definition: A data item that is the output of an absolute taxonomic abundance analysis for organisms grouped at the level of order.
Definition source: MicrobiomeDB
Requester: MicrobiomeDB
Reserved ID: EUPATH_0009354
Term Label: absolute abundance of family data
Definition: A data item that is the output of an absolute taxonomic abundance analysis for organisms grouped at the level of family.
Definition source: MicrobiomeDB
Requester: MicrobiomeDB
Reserved ID: EUPATH_0009355
Term Label: absolute abundance of genus data
Definition: A data item that is the output of an absolute taxonomic abundance analysis for organisms grouped at the level of genus.
Definition source: MicrobiomeDB
Requester: MicrobiomeDB
Reserved ID: EUPATH_0009356
Term Label: absolute abundance of species data
Definition: A data item that is the output of an absolute taxonomic abundance analysis for organisms grouped at the level of species.
Definition source: MicrobiomeDB
Requester: MicrobiomeDB
Reserved ID: EUPATH_0009357

@johnwjudkins We have added the terms for relative abundance analysis. I made the labels and definitions for absolute abundance analysis according to those terms. I don't think we need microbiome team to review the terms. Could you please review the labels and definitions before I make the pull request? Thanks!

These look good @zhengj2007


Add links to the logs

Currently getting to the logs is a pain, as far as I know the only way is by knowing the url. Adding a link to them would help (maybe spiders.html on the features count?)

Or maybe it's time to get a new page that highlights spider errors, and then maybe one for data errors we've identified.

Yep, getting to logs should be easier.
The same "spiders.html" page could optionally show the count of errors instead of places scraped?

We could toggle to elapsed time, too, to highlight spiders that are bumping into the ~4~ 8 hour limit.
Looks like the spider file name ended up in there this time around, too, so we can include links to the spider in Github: https://data.alltheplaces.xyz/runs/2022-09-03-13-31-55/stats/_results.json

Closable? I mean this could use a bit more UX love; but it works.
We also changed to have the "latest" links... somewhere!

Yep, logs are much easier to get to now.


better match keybindings to what vscode has

Fixes #3507

CHANGELOG entry needed?

Build already succeeded, merging.


tweak(fxmanifest): add ScaleformUI_Lua dependency

without this dependency you will need to ensure ScaleformUI_Lua is started before starting HandlingEditor, adding it as a dependency both ensures it's started in the right order and that the user does not forget to install it.

done!


IndexError: list index out of range while upgrading tools in bcbio

Version info

bcbio version (bcbio_nextgen.py --version): 1.2.9
OS name and version (lsb_release -ds): Scientific Linux release 7.5 (Nitrogen)

To Reproduce
Exact bcbio command you have used:
bcbio_nextgen.py upgrade --tools --tooldir /gpfs/igmmfs01/eddie/aitman-lab/Prasun/bcbio_nextgen/bcbio_tools/bin

Your yaml configuration file:

Log files (could be found in work/log)
Please attach (10MB max):  bcbio-nextgen-commands.log, and bcbio-nextgen-debug.log.
[pdutta@node2f24(eddie) bcbio_nextgen]$ bcbio_nextgen.py upgrade --tools --tooldir /gpfs/igmmfs01/eddie/aitman-lab/Prasun/bcbio_nextgen/bcbio_tools/bin
Upgrading bcbio
Upgrading third party tools to latest versions
Reading packages from /gpfs/igmmfs01/eddie/aitman-lab/Prasun/bcbio_nextgen/tmpbcbio-install/cloudbiolinux/contrib/flavor/ngs_pipeline_minimal/packages-conda.yaml
Creating conda environment: python3.6
Package Plan
environment location: /gpfs/igmmfs01/eddie/aitman-lab/Prasun/conda/envs/python3.6
added / updated specs:
- nomkl
- python=3.6
The following NEW packages will be INSTALLED:
_libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
_openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-1_gnu
ca-certificates    conda-forge/linux-64::ca-certificates-2021.10.8-ha878542_0
ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.36.1-hea4e1c9_2
libffi             conda-forge/linux-64::libffi-3.4.2-h7f98852_5
libgcc-ng          conda-forge/linux-64::libgcc-ng-11.2.0-h1d223b6_13
libgomp            conda-forge/linux-64::libgomp-11.2.0-h1d223b6_13
libnsl             conda-forge/linux-64::libnsl-2.0.0-h7f98852_0
libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-11.2.0-he4da1e4_13
libzlib            conda-forge/linux-64::libzlib-1.2.11-h36c2ea0_1013
ncurses            conda-forge/linux-64::ncurses-6.3-h9c3ff4c_0
nomkl              conda-forge/noarch::nomkl-1.0-h5ca1d4c_0
openssl            conda-forge/linux-64::openssl-1.1.1l-h7f98852_0
python             conda-forge/linux-64::python-3.6.15-hb7a2778_0_cpython
readline           conda-forge/linux-64::readline-8.1-h46c0cb4_0
sqlite             conda-forge/linux-64::sqlite-3.37.0-h9cd32fc_0
tk                 conda-forge/linux-64::tk-8.6.12-h27826a3_0
xz                 conda-forge/linux-64::xz-5.2.5-h516909a_1
zlib               conda-forge/linux-64::zlib-1.2.11-h36c2ea0_1013
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Traceback (most recent call last):
File "/gpfs/igmmfs01/eddie/aitman-lab/Prasun/bcbio_nextgen/bcbio_tools/bin/bcbio_nextgen.py", line 228, in
install.upgrade_bcbio(kwargs["args"])
File "/exports/igmm/eddie/aitman-lab/Prasun/bcbio_nextgen/bcbio_data/anaconda/lib/python3.6/site-packages/bcbio/install.py", line 93, in upgrade_bcbio
upgrade_thirdparty_tools(args, REMOTES)
File "/exports/igmm/eddie/aitman-lab/Prasun/bcbio_nextgen/bcbio_data/anaconda/lib/python3.6/site-packages/bcbio/install.py", line 510, in upgrade_thirdparty_tools
cbl_conda.install_in(conda_bin, args.tooldir, package_yaml)
File "/gpfs/igmmfs01/eddie/aitman-lab/Prasun/bcbio_nextgen/tmpbcbio-install/cloudbiolinux/cloudbio/package/conda.py", line 97, in install_in
conda_envs = _create_environments(conda_bin, packages)
File "/gpfs/igmmfs01/eddie/aitman-lab/Prasun/bcbio_nextgen/tmpbcbio-install/cloudbiolinux/cloudbio/package/conda.py", line 287, in _create_environments
out[addenv] = [x for x in conda_envs if x.endswith("/%s" % addenv)][0]
IndexError: list index out of range

Closing this issue as I am performing a fresh install of the latest bcbio version (v1.2.9)


Prowler Scan Freezing During Process

Steps to Reproduce
Command: prowler aws --security-hub
Cloud Provider: AWS
Scenario: I am running Prowler to send findings to AWS Security Hub.
Issue: During the scan, Prowler freezes.
First Test: I ran Prowler on a Lubuntu operating system hosted in Oracle VirtualBox directly on my laptop. In this scenario, the scan always froze during the CloudTrail check.
Second Test: I created an Ubuntu EC2 instance (t2.small) directly in AWS and installed Prowler via pip in a Python venv environment. In this scenario, the scan froze during the MemoryDB check. I noticed the EC2 instance became very slow, and upon checking the monitor, I observed the CPU utilization was at 100%.
Expected behavior
That the scan completes 100% and the findings are sent to AWS Security Hub.
Actual Result with Screenshots or Logs
Screenshot of the frozen terminal. The scan did not complete, the terminal is unresponsive, and the scan execution time is not incrementing:

AWS Monitoring:

How did you install Prowler?
From pip package (pip install prowler)
Environment Resource
EC2 Instance - t2.small
OS used
Ubuntu
Prowler version
Prowler 4.6.0
Pip version
pip 24.0 (python 3.12)
Context
I am writing an article about the integration of Prowler and Security Hub. I was using the latest version of the tool when I encountered the issue.

Hello @stenionljunior we will review this and get back to you once we have an update. I think the scan will finish but it'll require time to analyse all the resources in your account.
In the meanwhile, could you please review the following questions:

Could you execute it again using the --log-level ERROR and paste the output here?
How many resources do you have in your AWS environment?

Prowler requires a different amount of CPU and Memory based on the number of resources with a minimum of 2 CPU and 4GiB of RAM memory.

Thanks for using Prowler 🚀

Hello @jfagoagas!
After changing the instance to t2.large, the scan finished! Thank you for the feedback.

That's fantastic, thank you 🚀


Complete on pressing spacebar

Hello,
This package seems to be exactly what I need! So thank you for making it.
One feature I'd like to implement in my own version or get merged in is the ability to complete on the press of "space" instead of just tab.
Have you looked into this at all?  Would this be a welcome feature to get implemented?
If so can you point me as to how I would do it?
Thank you

Setting your own keybinds to activate autocompletions isn't enough for you ?
I really don't think activating completions on each space hitting is what everyone wants.

I was wondering if we could potentially only complete on space press for this specific package. Right now because we are using autocomplete, there could be more than one completion provider at the same time.  I also would like to introduce this feature behind a config flag that can be opt in, in case this isn't the behavior everyone wants.
Thoughts?


chore(dal): Document dynamic funcs

I'd merge but there appears to be a CI error.

bors merge

bors merge


Spyder IDE GUI elements don't reset scaling with View->Window Layouts->Reset to Spyder default

Problem Description
I just upgraded Laptop to a model with 14" screen and an 4k panel.
My external Monitor is a 24" model with 1080p panel.
After unplugging the monitors HDMI cable (while running Spyder on the monitor)
the Spyder IDE application window jumped back to the internal display and
the scaling of the GUI elements (pane borders, text in the status bar) where
grotesquely enlarged. (screenshot attached)
The text itself is correctly scaled, for example in the coding pane, but the way too big GUI elements make
the whole IDE unusable.
This behaviour stayed, no restart changes the scaling. (regardless of which monitor spyder now runs on)

As I understand it, this problem should be fixable by using the reset of the pane layout:
View->Window Layouts->Reset to Spyder default
But using this function correctly resets the spacing between the panes but leaves the scaling of the GUI untouched.
I found several older questions where this was mentioned as the solution to scaling problems, so I assume this
is a bug worth reporting.
What steps reproduce the problem?

Working on 1080p monitor
disconnecting 1080p monitor so that windows 10 sends the application to the internal 4k display
trying to reset the layout to no avail

What is the expected output? What do you see instead?

Following a hunch I disconnected the 1080p external monitor again and started Spyder with just
the 4k laptop screen. This resulted in Spyder starting with correctly scaled GUI elements, and they stayed correct
once I reattached the external monitor dragged the spyder application window to the external screen.
So at least I know a workaround now, but I think it would still be preferable if the dedicated reset option would fix this issues as well.

Hi @WEB-FMA could you check if some of the options at Preferences > Application > Interface > Screen resolution like Enable auto high DPI scaling help?
Let us know!

Hi @dalthviz I tried all three different settings under ""Configuration for high DPI screens", all I was able to do was making the UI elements even bigger when I forced a custom high DPI scaling (as the value can't be set below 1, and everything above scales up).
The bug stemmed from Spyder already starting upscaled, or at least it appeared that way to me.
I'm used to spyder misaligning panels and scaling when the external monitor gets disconnected, my normal approach here is
to snap the application window fullscreen to the monitor I'm working on. Then Spyder resets itself correctly, sometimes panes get a little jumbled up but this is fixable in the worst case by resetting the window layout to default (normally it's not that bad).
With the bug I described this approach didn't help, that's why I wrote the bug report.
Once I was able to fix the layout again with the workaround I described in my second post, I did not have the problem I
described again. If I am able to recreate the behaviour again I will post it here.

Thanks for the details @WEB-FMA ! And yes please, If you have any other info that let us understand better this is greatly appreciated :+1:

I have the same issue with the scaling. Up until now I have followed what you have written in the comments and the scaling remains incorrect, despite the screen I use the IDE on. This has happened before but, I was always able to solve it by going to preferances.

I have this same scaling problem on my new setup with Windows 11 and Spyder 5.5.1.
The computer is a laptop with a 4K screen set at 225% upscaling.  The Main monitor is 4k set at 100% scaling.
The text in the editor and console are correct, but all the tabs and the address bar and everything in the preferences UI are about 3 times (maybe 2.25 times) bigger than they should be.  The spacing of the drop-down menus is also about 3 times taller than normal (even though the font there is correct).  I tried all the options in preferences, and nothing helped.
I do not have this problem on my old setup: a laptop 1440p at 150% and the same 4K main monitor at 100% using Spyder v5.4.1 on Windows 10.

Hey @bramson, thanks for letting us know that this is still a problem. Could you download and install our latest Windows application for Spyder 6.0 alpha4?
That application is completely independent from Anaconda/Miniconda, so it won't interfere with them. I'd like to know if this problem is solved in it because we've done several improvements to the way we handle fonts in that version.

I installed the Windows package from your link, and I confirmed that it does not have the scaling problem.
(It did start in Japanese, which took me a while to fix, but the scaling looks fine.)
I then installed v6.0.a4 through conda inside my mini forge environment.
And running it from my Python console gave me no problems.
It even kept the preferences I set after from running it from the Windows install package (which is a nice bonus).
Thank you for your hard work.  For me, v6 closes this issue.

That's fantastic news! Thanks for letting us know about it @bramson.

....but I spoke too soon.
First, even before, when it was mostly fixed, the fonts in linting pop-ups were still huge...but I usually turn those off so it didn't bother me.
Windows 11 just updated itself, and I don't know if that was the cause, but now when I open Spyder 6 the same way I did yesterday, the entire interface is scaled incorrectly...specifically, everything is about 2 times larger than it should be., but not inside the editor.  This is not the same as v5.4 when the elements affected were inconsistent.  Now it is a consistent enlargement.  I tried variations of the high DPI scaling and none had any effect.
And now, if I start Spyder without the external monitor attached, and then attach it, and move Spyder to it, the scaling is inconsistently correct.  Fonts and icons are now the correct size (even in the linting pop-ups!), but the spacing is 1.5 or 2 times too big.  It's fully usable in this state, but the layout is less efficient and less attractive.


Introduce tabs into the project details page

Introduce multiple tabs to switch between tasks, vulnerabilities and targets (and future content too)

done! with commit https://github.com/Reconmap/web-client/commit/d92ea21e4ecd64568868f4497a688760c4d521dc


feat: Add auto-shard support

What type of pull request is this?

[x] Non-breaking code change
[ ] Breaking code change
[ ] Documentation change/addition
[ ] Tests change

Description
Adds a new AutoShardedClient which will automatically create and manage all requested shards with minimal syntax changes, and importantly respecting discord rate limits in sharding and identification. This client allows all shards to share the same cache.
This PR also refines the guild caching step in the startup sequence, preventing erroneous "Not All Guilds Cached" warnings.
Changes

Create AutoShardedClient
Create ShardConnect and ShardDisconnect events
Change logging messages to include shard where relevant
Add _shard_ready event to ConnectionState
Add get_gateway_bot method to httpclient
Refactor Client.intents into a attribute rather than property
Add client/AutoShardedClient“.get_guild_web socket to allow getting the appropriate web socket for a given guild
Refactor Client._on_websocket_ready to avoid false warnings
Fix logger name

Checklist

[x] I've formatted my code with Black
[x] I've ensured my code works on Python 3.10.x
[x] I've tested my code

How many shard would you suggest the max be for this auto sharding?

Technically as many as you need. However in my opinion this form of sharding is only suitable for 2-10 shards, after that more processes would be needed.


Try http2 for the fetch

Is your feature request related to a problem? Please describe.
The kubernetes watch is brittle and we are silently missing watch events when using node-fetch, we want to try using http2 to the fetch call to check if we get more http events like disconnect and can use logic to reconnect, and ultimately more watch events.
Links:

https2 doc
http2 client side example

Describe the solution you'd like

Given someone does a watch
When the watch connects to the kube-apiserver
Then it uses the http2 library to connect.

Describe alternatives you've considered
(optional) A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Closing this, going to have a conversation with Jeff about the next steps.


[Bug]: In dark mode, the script and code snippet empty status prompt text is difficult to read.

ODC version
ODC 4.3.2
OB version
No related.
What happened?
In dark mode, the script and code snippet empty status prompt text is difficult to read.

What did you expect to happen?
Clearly display the text in dark mode.
How can we reproduce it (as minimally and precisely as possible)?
Switch to dark mode to view script and code snippets in empty state.
Anything else we need to know?
No response

Fixed In Commit 35a348fad2634216b838124b1ad984bd4808b1ec
------
xiaokang
Mon Sep 23 14:11:18 2024 +0800

pass


Typeahead Updates: Disable Filtering / Sorting + Full Theming Support

Jira
N/A
Summary
Subset of Typeahead updates from https://github.com/boltdesignsystem/bolt/pull/1542 that don't require any of the Preact v10 upgrade work from https://github.com/boltdesignsystem/bolt/pull/1567.

Details
This PR tackles 5 things related to Typeahead:

Adds the ability to optionally disable Typeahead's filtering / sorting
Allow different data structures to be passed along to Typeahead and rendered (see demo for reference)
Fix theming issues with Typeahead (theming is now fully supported)
Cleans up some of the internal Typeahead's UI logic to help keep keep things in a maintainable state.
Very minor docs-site related updates needed to support documenting #s 1-3.

How to test

Confirm tests pass as expected
Review the updated demos and docs for Typeahead

@colbytcook closing and not deleting. Typeahead is stable now, so we wouldn't want to introduce these changes, but if we want to add custom theming support in the future this work is here as a reference.


Task/operation integration tests

Closes #72

This change is

Codecov Report

Merging #98 into master will increase coverage by 4.58%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master      #98      +/-   ##
============================================
+ Coverage     73.86%   78.44%   +4.58%
- Complexity     1071     1147      +76
============================================
  Files           108      108
  Lines          3531     3531
  Branches        152      152
============================================
+ Hits           2608     2770     +162
+ Misses          848      681     -167
- Partials         75       80       +5

Impacted Files
Coverage Δ
Complexity Δ

...va/org/oscm/rest/common/SampleTestDataUtility.java
0% <ø> (ø)
0 <0> (ø)
:arrow_down:

...java/org/oscm/rest/service/PriceModelResource.java
100% <ø> (ø)
5 <0> (ø)
:arrow_down:

...rg/oscm/rest/service/TechnicalServiceResource.java
100% <ø> (ø)
7 <0> (ø)
:arrow_down:

...g/oscm/rest/service/CompatibleServiceResource.java
100% <ø> (ø)
3 <0> (ø)
:arrow_down:

...java/org/oscm/rest/operation/SettingsResource.java
100% <ø> (ø)
6 <0> (ø)
:arrow_down:

...java/org/oscm/rest/service/TSSupplierResource.java
100% <ø> (ø)
4 <0> (ø)
:arrow_down:

...in/java/org/oscm/rest/service/ServiceResource.java
100% <ø> (ø)
7 <0> (ø)
:arrow_down:

...t/service/data/TechnicalServiceRepresentation.java
68.18% <0%> (+3.4%)
46% <0%> (+3%)
:arrow_up:

.../oscm/rest/service/data/ServiceRepresentation.java
84.46% <0%> (+10.67%)
35% <0%> (+5%)
:arrow_up:

.../rest/service/data/OrganizationRepresentation.java
96.15% <0%> (+26.92%)
9% <0%> (+2%)
:arrow_up:

... and 9 more


Add plugin support to amok

add plugin support for amok.
this would allow for developers to create npm based plugins for amok and extend its capabilities based on their usecase.
amok would contain few default plugins for serving responses from flat files, swagger files, s3 buckets etc.

implemented since 2.x.x


5.0 | Proposal: add return types to select methods

Setting the scene
At this time, it has not yet been decided what the minimum supported PHP version will be for PHPCS 5.0, though it will be PHP 7.2 or higher (as PHP 7.2 is the new minimum supported PHP version for PHCPS 4.0).
PHP 7.2 allows for:

Adding parameter type declarations, with the exception of union/intersection/DNF/mixed types.
Adding return type declarations, with the exception of union/intersection/DNF/mixed types.

PHP 7.2 does not support:

Property type declarations (PHP 7.4+)
Constant type declarations (PHP 8.3+)
Union/intersection/DNF types (PHP 8.0, 8.1, 8.2 respectively).
Mixed type (PHP 8.0)
Null/true/false as in union types or as stand-alone types (PHP 8.0/8/2)

Parameter type declarations in PHP are contravariant, which means that overloaded methods in child classes can widen the type.
Return type declarations in PHP are covariant, which means that overloaded methods in child classes can make the type more narrow.
Full support for contravariance and covariance was only added in PHP 7.4.
Proposal
I propose to add return type declarations to the public/protected methods in PHPCS 5.0 as per the list below - providing the minimum PHP version will allow for it (to be decided and discussed in a separate issue).
If any of the signature changes can not be made yet due to the minimum PHP version not allowing for it (yet), those changes (and only those) will move to the first major release which has the required minimum PHP version.
Detailed proposal
Notes about the detailed proposal:

Signatures of private methods methods are excluded from the below list as they don't affect the public API, but these will be updated too.
Methods for which no legal return type can be added, have been left out of the list.
Methods which will need to be updated by extension and do not deviate from the change to the "parent" method, are not listed repetitively.
The return types have largely been based on the currently documented types, which may not be correct (in which case the type will be changed to the real type).
The list is based on the current state of the codebase and is subject to change.

In PHP_CodeSniffer\Config:

[ ] public function __get(...): mixed (* PHP 8.0+)
[ ] public function __set(...): void
[ ] public function __isset(...): bool
[ ] public function __unset(...): void
[ ] public function getSettings(): array
[ ] public function setSettings(...): void
[ ] public function setCommandLineValues(...): void
[ ] public function restoreDefaults(): void
[ ] public function processShortArgument(...): void
[ ] public function processLongArgument(...): void
[ ] public function processUnknownArgument(...): void
[ ] public function processFilePath(...): void
[ ] public function printUsage(): void
[ ] public function printPHPCSUsage(): void
[ ] public function printPHPCBFUsage(): void
[ ] public static function getConfigData(...): ?string
[ ] public static function getExecutablePath(...): ?string
[ ] public function setConfigData(...): bool
[ ] public static function getAllConfigData(): array
[ ] public function printConfigData(...): void

In PHP_CodeSniffer\Fixer:

[ ] public function startFile(...): void
[ ] public function fixFile(): bool
[ ] public function generateDiff(...): string
[ ] public function getFixCount(): int
[ ] public function getContents(): string
[ ] public function getTokenContent(...): string
[ ] public function endChangeset(): ? (under discussion, see #375)
[ ] public function rollbackChangeset(): void
[ ] public function replaceToken(...): bool
[ ] public function revertToken(...): bool
[ ] public function substrToken(...): bool
[ ] public function addNewline(...): bool
[ ] public function addNewlineBefore(...): bool
[ ] public function addContent(...): bool
[ ] public function addContentBefore(...): bool
[ ] public function changeCodeBlockIndent(...): void

In PHP_CodeSniffer\Reporter:

[ ] public function printReports(): bool
[ ] public function printReport(...): void
[ ] public function cacheFileReport(...): void
[ ] public function prepareFileReport(...): array

In PHP_CodeSniffer\Ruleset:

[ ] public function explain(): void
[ ] public function hasSniffDeprecations(): bool
[ ] public function showSniffDeprecations(): void
[ ] public function processRuleset(...): array
[ ] public function registerSniffs(...): void
[ ] public function populateTokenListeners(): void
[ ] public function setSniffProperty(...): void
[ ] public function getIgnorePatterns(...): array
[ ] public function getIncludePatterns(...): array

In PHP_CodeSniffer\Runner:

[ ] public function runPHPCS(): int
[ ] public function runPHPCBF(): int
[ ] public function checkRequirements(): void
[ ] public function init(): void
[ ] public function handleErrors(...): bool
[ ] public function processFile(...): void
[ ] public function printProgress(...): void

In PHP_CodeSniffer\Files\File:

[ ] public function setContent(): void
[ ] public function reloadContent(): void
[ ] public function disableCaching(): void
[ ] public function process(): void
[ ] public function parse(): void
[ ] public function getTokens(): array
[ ] public function cleanUp(): void
[ ] public function addError(...): bool
[ ] public function addWarning(...): bool
[ ] public function addErrorOnLine(...): bool
[ ] public function addWarningOnLine(...): bool
[ ] public function addFixableError(...): bool
[ ] public function addFixableWarning(...): bool
[ ] protected function addMessage(...): bool
[ ] public function recordMetric(...): bool
[ ] public function getErrorCount(): int
[ ] public function getWarningCount(): int
[ ] public function getFixableCount(): int
[ ] public function getFixedCount(): int
[ ] public function getIgnoredLines(): array
[ ] public function getErrors(): array
[ ] public function getWarnings(): array
[ ] public function getMetrics(): array
[ ] public function getListenerTimes(): array
[ ] public function getFilename(): string
[ ] public function getDeclarationName(...): ?string
[ ] public function getMethodParameters(...): array
[ ] public function getMethodProperties(...): array
[ ] public function getMemberProperties(...): array
[ ] public function getClassProperties(...): array
[ ] public function isReference(...): bool
[ ] public function getTokensAsString(...): string
[ ] public function findPrevious(...): int|false (* PHP 8.0+)
[ ] public function findNext(...): int|false (* PHP 8.0+)
[ ] public function findStartOfStatement(...): int
[ ] public function findEndOfStatement(...): int
[ ] public function findFirstOnLine(...): int|false (* PHP 8.0+)`
[ ] public function hasCondition(...): bool
[ ] public function getCondition(...): int|false (* PHP 8.0+)`
[ ] public function findExtendedClassName(...): string|false (* PHP 8.0+)`
[ ] public function findImplementedInterfaceNames(...): array|false (* PHP 8.0+)`

In PHP_CodeSniffer\Files\DummyFile:

[ ] public function setErrorCounts(...): void

In PHP_CodeSniffer\Files\FileList:

[ ] public function addFile(...): void
[ ] public function rewind(): void
[ ] public function current(): \PHP_CodeSniffer\Files\File
[ ] public function key(): ?string
[ ] public function next(): void
[ ] public function valid(): bool
[ ] public function count(): int

In PHP_CodeSniffer\Filters\Filter:

[ ] public function accept(): bool
[ ] public function getChildren(): \RecursiveIterator
[ ] protected function shouldProcessFile(...): bool
[ ] protected function shouldIgnorePath(...): bool

In PHP_CodeSniffer\Filters\ExactMatch:

[ ] abstract protected function getDisallowedFiles(): array
[ ] abstract protected function getAllowedFiles(): array

In PHP_CodeSniffer\Filters\Git*:

[ ] protected function exec(...): array

In PHP_CodeSniffer\Generators\Generator:

[ ] protected function getTitle(...): string
[ ] public function generate(): void
[ ] abstract protected function processSniff(...): void

In PHP_CodeSniffer\Generators\*:
All methods will get the void return type.
In PHP_CodeSniffer\Reports\Report:

[ ] public function generateFileReport(...): bool
[ ] public function generate(...): void

In PHP_CodeSniffer\Sniffs\Sniff:

[ ] public function register(): array

Note: the public function process(...) method will not get a return type as int|void is an illegal type, even in PHP 8.0+ and ?int is incompatible with a child class having a void return type.
Classes implementing the interface can still add a return type if the type returned from their implementation is consistently void or int.
In PHP_CodeSniffer\Sniffs\AbstractArraySniff:

[ ] public function process(...): void
[ ] abstract protected function processSingleLineArray(...): void
[ ] abstract protected function processMultiLineArray(...): void

In PHP_CodeSniffer\Sniffs\AbstractPatternSniff:

[ ] final public function process(...): void
[ ] protected function processPattern(...): array|false (* PHP 8.0+)
[ ] protected function prepareError(...): string
[ ] abstract protected function getPatterns(): array
[ ] protected function registerSupplementary(): array
[ ] protected function processSupplementary(...): void

In PHP_CodeSniffer\Sniffs\AbstractScopeSniff:

[ ] abstract protected function processTokenWithinScope(...): ?int
[ ] abstract protected function processTokenOutsideScope(...): ?int

In PHP_CodeSniffer\Sniffs\DeprecatedSniff:

[ ] public function getDeprecationVersion(): string
[ ] public function getRemovalVersion(): string
[ ] public function getDeprecationMessage(): string

In PHP_CodeSniffer\Tokenizers\Tokenizer:

[ ] protected function isMinifiedContent(...): bool
[ ] public function getTokens(): array
[ ] abstract protected function tokenize(...): array
[ ] abstract protected function processAdditional(): void
[ ] public function replaceTabsInToken(...): void

In PHP_CodeSniffer\Tokenizers\Comment:

[ ] public function tokenizeString(...): array

In PHP_CodeSniffer\Tokenizers\PHP:

[ ] public static function standardiseToken(...): array
[ ] public static function resolveSimpleToken(...): array

In PHP_CodeSniffer\Util\Cache:

[ ] public static function load(...): void
[ ] public static function save(): void
[ ] public static function get(...): mixed (* PHP 8.0+)
[ ] public static function set(...): void
[ ] public static function getSize(): int

In PHP_CodeSniffer\Util\Common:

[ ] public static function isPharFile(...): bool
[ ] public static function isReadable(...): bool
[ ] public static function realpath(...): string|false (* PHP 8.0+)
[ ] public static function stripBasepath(...): string
[ ] public static function detectLineEndings(...): string
[ ] public static function isStdinATTY(): bool
[ ] public static function escapeshellcmd(...): string
[ ] public static function prepareForOutput(...): string
[ ] public static function isCamelCaps(...): bool
[ ] public static function isUnderscoreName(...): bool
[ ] public static function suggestType(...): string
[ ] public static function getSniffCode(...): string
[ ] public static function cleanSniffClass(...): string

In PHP_CodeSniffer\Util\Standards:

[ ] public static function getInstalledStandardPaths(): array
[ ] public static function getInstalledStandardDetails(...): array
[ ] public static function getInstalledStandards(...): array
[ ] public static function isInstalledStandard(...): bool
[ ] public static function getInstalledStandardPath(...): ?string
[ ] public static function printInstalledStandards(): void

In PHP_CodeSniffer\Util\Timing:

[ ] public static function startTiming(): void
[ ] public static function getDuration(): float
[ ] public static function getHumanReadableDuration(...): string
[ ] public static function printRunTime(...): void

In PHP_CodeSniffer\Util\Tokens:

[ ] public static function tokenName(...): string
[ ] public static function getHighestWeightedToken(...): int|false (* PHP 8.0+)

In PHP_CodeSniffer\Tests\Standards\AbstractSniffUnitTest:

[ ] protected function getTestFiles(...): array
[ ] protected function shouldSkipTest(): bool
[ ] public function setCliValues(...): void
[ ] protected function getErrorList(): array
[ ] protected function getWarningList(): array

[*] The above list may still receive some updates over time. If/when updates are made, a comment will be left in the thread detailing the updates which were made.
New methods
Any new methods being introduced to PHPCS during the lifetime of the 4.x major should be considered outside the scope of this issue and can get a return type declaration at their introduction, providing the minimum PHP version allows for it.
New methods which need a type not yet supported via the current minimum PHP version should be added to the list above.
BC considerations
As return types are covariant, adding these type declarations in PHPCS 5.0 constitutes a backward compatibility break as the method signature of overloaded/implemented methods MUST be updated to include a compatible return type declaration.
Suggested Migration Path for External Standards
During the lifetime of PHPCS 4.x, external standards are encouraged to:

Add parameter type declarations to their code, in line with the proposal in #390.
Add return type declarations to their code, in line with the above proposal
Add strict_types=1 declarations to their files.
Use tooling like PHPStan and/or Psalm to find further type juggling issues.

None of this can/will be enforced, but external standards using the lifetime of PHPCS 4.x to action the above will be better prepared for supporting PHPCS 5.x when the time comes.
Opinions ?
/cc @asispts @dingo-d @fredden @GaryJones @greg0ire @kukulich @michalbundyra @Ocramius @sirbrillig @stronk7 @weierophinney @wimg

This sounds great! You mentioned private methods, what about final classes and methods? Should they benefit from return type declarations on 3.x already?

This sounds great! You mentioned private methods, what about final classes and methods? Should they benefit from return type declarations on 3.x already?

@greg0ire 3.x has a PHP 5.4 minimum, so that's a no-go. As for 4.x, I'd have to see, though there's enough to do for 4.0 already, so I think it might be better to leave it till 5.0.

I think the more types the better, so I'd vote yes for this.


Default programming language for syntax highlighting

Feature request: set a community default programming language for syntax highlighting.
This might be useful for e.g. the Julia zulip, where almost all the code posted will be in Julia. For code sections that should not be highlighted, users can use:
```text
blah
```

Discussed briefly here: https://chat.zulip.org/#narrow/stream/9-issues/topic/default.20programming.20language

This seems like a pretty reasonable feature, and probably should be implementable using the combination of our "New application feature" tutorial with a few changes in fenced_code.py to fetch the value.
In the database, we can use the setting variable default_fenced_code_language, a TextField with default=None.
@aero31aero do you want to see if you can make this happen in the next few days?  I feel like it should be fairly fast.

@zulipbot claim

would it be possible to auto detect the programming language based on what the user puts into a code block?
If the problem is that it's likely to be wrong, then we can just open the typeahead and put it there instead of filling it in.

I don't think we want auto detection for the Julia zulip. A default of Julia is unlikely to be wrong often for us, nor will it cause significant issue when it is.

Right, of course. And having a server wide default makes sense for your use case, and an obvious step in the right direction.
I was just wondering if we could add auto detection of languages as a feature that's helpful for generic programming communities and eg dev servers such as CZO.
If that's not possible, maybe we can allow stream wise defaults?
So in the example of CZO, frontend could use JS as a default and backend could use python as a default.
I want to prevent us from promoting the case where a default that helps some people annoys others.

Sure. Per-stream defaults sound like they might be good. I'm quite skeptical of accurate language detection from small snippets, but maybe it works better than I think.
Typeahead might be nice for showing what the default/current autodetection is, but I don't think I'd want to rely on users actually selecting the hint.

Thank you!

I was just wondering if we could add auto detection of languages as a feature that's helpful for generic programming communities and eg dev servers such as CZO.

We can do this pretty easily; there's a guess_lang feature of pygments that attempts to do this.  I think we could offer an option in the dropdown of "Guess" that uses that feature; users can decide if they like the results.
I agree with @cmcaine that it isn't accurate enough that we'd want to use it generally (at least, it wasn't a few years ago, which is why it's been turned off in Zulip since soon after we first added the fenced code block feature).


[CARBONDATA-2514] Added condition to check for duplicate column names

Duplicate columns check was not present.
indexFileReader was not being closed due to which index file could not be deleted.

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[x] Any interfaces changed?

[x] Any backward compatibility impacted?

[x] Document update required?

[x] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

[x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

retest this please

SDV Build Success , Please check CI http://<IP_ADDRESS>:8080/job/ApacheSDVTests/5052/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/6047/

Build Failed  with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/6066/

retest this please

SDV Build Success , Please check CI http://<IP_ADDRESS>:8080/job/ApacheSDVTests/5067/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/6068/

LGTM


1.20.1 support

amazing mod maybe 1.20.1 support in the future?

As soon as the mod is stable in this version and I get as many bug out as I can, I'll definitely backport if other mods still haven't updated.


Added install_scaleio_SLES.yml to support SLES rpm installs

We needed SLES support for installing and configuring ScaleIO. ScaleIO provides SLES rpms so this could be a useful change to any others who use them.
The only change necessary was copying the CentOS install task since rpm install follows the same process.

Thank you for the code...I'll merge.


Typo in content

On URL: https://learn.microsoft.com/en-us/training/modules/implement-manage-fslogix/1-introduction?source=learn
Under 'Manage user environments and apps for Azure Virtual Desktop' module 'Implement and manage FSLogix' -
There's a typo in the section 'Learning Objectives', third bullet, where it incorrectly says 'Install FXLogix.' It must be corrected to say 'Install FSLogix.'

Module: 00
Lab/Demo: 00
Task: 00
Step: 00
Description of issue
Repro steps:

That's nothing to do with the lab content - you need to raise the issue with Microsoft Learn by using the link on the page.

Pls follow the suggestion from Steve
rgds
Marcin


Does not work in CP?

Hi. I just tried your great plugin for my server for a maintenance operation. But I can still login and use the CP with a user profile that should not be able to do it (Patrol checkbox is not clicked in profile rights).
Did I forget someting to add in the config files ?

Hi, if the user is an admin, he/she will be allowed in even if that permission is not checked for the user. The permission is meant for non-admins.
If the user account you logged in with is not an admin, then that's probably something I need to look into.

One more thing... the CP is not affected during maintenance mode, only the front end to avoid lockouts.

Yes thats the point. I warn people in advance for a maintenance operation but I got many users on CP side so I d like them to be disconnecter automatically before there happen some weird things when they are in the middle of modifications...

Hey @migswd, sorry I didn't follow up sooner but it's still unclear to me what the idea. workflow would be for you. Do you feel that Patrol should protect the CP as well? Maybe have a config for that?

Hi.
Yes CP too.
I have 50 users in BO in CP using the website anytime, so it would be cool to be able to warn the guys and disconnect the automatically, after some email warning of course. But sometime some people do not see the email…
But I dont know if it can be possible.
Miguel Bocquier
<EMAIL_ADDRESS>http://silkwormdream.com
06 51 40 01 80

On 28 Oct 2015, at 08:16, Selvin Ortiz<EMAIL_ADDRESS>wrote:
Hey @migswd https://github.com/migswd, sorry I didn't follow up sooner but it's still unclear to me what the idea. workflow would be for you. Do you feel that Patrol should protect the CP as well? Maybe have a config for that?
—
Reply to this email directly or view it on GitHub https://github.com/selvinortiz/craft.patrol/issues/13#issuecomment-151749794.

@migswd While this is possible and probably non trivial to achieve, I don't think I'll be able to get to it until I update the plugin for Craft 2.5.

I'm gonna leave this one for Craft 3.0 because as of right now, nobody else is hitting this edgecase.


Rename clientDataJson to clientDataJSON

Fixes #338.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (15263ca) 77.50% compared to head (186dfc4) 77.50%.
Report is 1 commits behind head on master.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##           master     #450   +/-   ##
=======================================
  Coverage   77.50%   77.50%
=======================================
  Files          98       98
  Lines        2716     2716
  Branches      445      445
=======================================
  Hits         2105     2105
  Misses        492      492
  Partials      119      119

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@abergs, @aseigler this PR is ready to be reviewed.

This is one of those things that has been inconsistent from day one.  Should we take the call to rename all instances of clientDataJson to clientDataJSON and not just these two?

This is one of those things that has been inconsistent from day one. Should we take the call to rename all instances of clientDataJson to clientDataJSON and not just these two?

I was not sure if we want to use clientDataJSON with uppercase also in the c#. If you are ok with this, I can rename the property also in the c# contract.

@abergs , @iamcarbon , opinions?

My personal preference would be to use clientDataJSON during JSON serialization, and rename all the C# properties to "ClientDataJson" to remain consistent with .NET conventions.

My personal preference would be to use clientDataJSON during JSON serialization, and rename all the C# properties to "ClientDataJson" to remain consistent with .NET conventions.

So it's normally already the case:
https://github.com/passwordless-lib/fido2-net-lib/blob/11c78ce956ec2552bf122e2e14281080b5a8374e/Src/Fido2.Models/AuthenticatorAssertionRawResponse.cs#L43-L44
https://github.com/passwordless-lib/fido2-net-lib/blob/11c78ce956ec2552bf122e2e14281080b5a8374e/Src/Fido2.Models/AuthenticatorAttestationRawResponse.cs#L33-L34

For serialization/deserialization it should definitely be named clientDataJSON. That is what is used in the spec.
I would have tried to make the point on keeping JSON in .NET to make it less error-prone, but we rely on [JsonPropertyName] for many other fields, so that point is mute.
Go ahead and rename it to Json in .net @joegoldman2.

Go ahead and rename it to Json in .net @joegoldman2.

Thank you for the review. It's done.


Add nogil declaration to _run_paradiso function

This releases the Python GIL around the low level paradiso solver call. Hopefully this allows this library and dependent libraries (like SimPEG) to operate more smoothly in a multi-threaded environment.
This should be safe from a Python object perspective because no Python objects are created/destroyed/inc-reffed during this call.  It could still be that multiple concurrent calls of this function on the same data cause undesried results, but hopefully downstream users can apply concurrency primitives (like locks) as needed.
Fixes https://github.com/simpeg/pydiso/issues/8

Would it work to add the nogil decorator to the definitions imported from the header here https://github.com/simpeg/pydiso/blob/b31463f3b5cfb8b716a3f80b30c57440ffaed25e/pydiso/mkl_solver.pyx#L42 (and the pardiso_64 call as well)?

Maybe!  I should also add a disclaimer that I'm not an authoritative expert on this topic.
My guess is that your solution is a bit more general than mine.  It'll get used for example in the dealloc method and anywhere else these get used rather than just in the _run_paradiso method (although that seems to be pretty general purpose).

Ah!  CI does exist.  Sorry to doubt.  I'm not sure what's going on with the OSX number-of-threads failure though

I couldn't figure out why it was failing either on CI, and I couldn't reproduce that behavior locally either... So I've marked it as xfail on darwin systems so it won't error out the CI.


Support updating allowed_address_pairs on OpenStack ports

Support updating allowed_address_pairs on OpenStack ports
Description
Add support to OpenStack_2_NodeDriver for updating allowed-address-pairs on node ports.  This is a security extension that allows administrators to specify which IP and MAC addresses a node can use to communicate.
https://docs.openstack.org/api-ref/network/v2/index.html#allowed-address-pairs
Status

done, ready for review

Checklist (tick everything that applies)

[x] Code linting (required, can be done after the PR checks)
[ ] Documentation
[x] Tests
[ ] ICLA (required for bigger changes)

Codecov Report

Merging #1569 (5786481) into trunk (174f409) will decrease coverage by 0.08%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##            trunk    #1569      +/-   ##
==========================================
- Coverage   83.08%   83.00%   -0.09%
==========================================
  Files         394      394
  Lines       84795    84950     +155
  Branches     9004     9037      +33
==========================================
+ Hits        70456    70516      +60
- Misses      11274    11369      +95
  Partials     3065     3065

Impacted Files
Coverage Δ

libcloud/compute/drivers/openstack.py
85.84% <50.00%> (+0.10%)
:arrow_up:

libcloud/test/compute/test_openstack.py
94.45% <100.00%> (+0.01%)
:arrow_up:

libcloud/compute/drivers/equinixmetal.py
64.28% <0.00%> (-0.98%)
:arrow_down:

libcloud/compute/drivers/outscale.py
8.66% <0.00%> (-0.02%)
:arrow_down:

libcloud/common/base.py
90.59% <0.00%> (+0.02%)
:arrow_up:

libcloud/test/compute/test_ovh.py
97.34% <0.00%> (+0.02%)
:arrow_up:

libcloud/test/storage/test_azure_blobs.py
94.89% <0.00%> (+0.21%)
:arrow_up:

libcloud/storage/drivers/azure_blobs.py
86.84% <0.00%> (+0.60%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 174f409...5786481. Read the comment docs.

LGTM, thanks for contribution.
@micafer can you please also review this change?

It LGTM  also.

I'm tempted to also add methods to add and remove a single address-pair.  Would you be interested in this?

@dpeschman Yeah, that seems reasonable for - just please prefix those methods with ex_ since they are not part of the common / standard API.

I merged change into trunk - thanks for the contribution.
Please feel free to open a new PR for those two new additional methods.


[KMP] Memory leak found in AnyAddress

Describe the bug
I found a memory leak in AnyAddress within the Kotlin package of Wallet Core.
dependencies {
    implementation "com.trustwallet:wallet-core-kotlin-android:4.1.7"
}

To Reproduce
The following code can trigger a memory leak:
    println("Test started")
    val publicKeyHex = "044ba28b11af1561042b03b9d0f940446315af11358aa12d798050b3cf76265dab0f48b22ea1fc1f9560c969b966221f2821b746c4e56efaeaeec8689caf5843c9"
    var i = 1
    while (i <= 1_000_000) {
        val publicKey = PublicKey(publicKeyHex.hexToByteArray(), PublicKeyType.SECP256k1Extended)
        val anyAddress = AnyAddress(publicKey, Ethereum, Derivation.Default)   // This leaks memory!
        if (i % 10_000 == 0) {
            println("Test case $i run")
        }
        i++
    }
    println("Test completed")

Expected behavior
No memory leak.
Screenshots
Memory Profiler Screenshot:

Additional context
The above test was conducted on Android. Based on the generated code in directory wallet-core-kotlin, this issue should also exist on iOS.

Hi @10gic, good catch! I'll investigate what can be the reason for the memory leak.
Could you please advise if you caught the memory leak while using other WC bindings? For example, PrivateKey or HDWallet?

Hi @satoshiotomakan, I found that this is a common issue. It exists in HDWallet, PrivateKey, PublicKey, and others.
For swift binding (swift/Sources/Generated/AnyAddress.swift), we have deinit code to release memory:
    deinit {
        TWAnyAddressDelete(rawValue)
    }

For wasm binding (wasm/src/generated/AnyAddress.h), we have destructor to release memory:
    ~WasmAnyAddress() {
        TWAnyAddressDelete(instance);
    }

Kotlin don't support destructor, we need to think of other solutions to release memory.

Hi @satoshiotomakan, the pr https://github.com/trustwallet/wallet-core/pull/4031 only try to fix the issue https://github.com/trustwallet/wallet-core/issues/4030.
Issue https://github.com/trustwallet/wallet-core/issues/4030 and issue https://github.com/trustwallet/wallet-core/issues/4021 are two different types of memory leaks.
Issue https://github.com/trustwallet/wallet-core/issues/4021 hasn't been fixed yet.

Hi @satoshiotomakan, do you have any ideas on how to fix this issue (https://github.com/trustwallet/wallet-core/issues/4021)?

@10gic
I scheduled a call this Friday with Kotlin engineers from another team. One of the ideas is to use native Swift, Java, JS wrappers to free the memory.
Unfortunately, I can't fully dedicate to the memory leaks as I'm currently focused on BTC PSBT, but will do the research and PR review end of this week

Hi @10gic, we're trying to investigate, how it can be fixed. Will let you know once I have an update

Hi @10gic, we're trying to investigate, how it can be fixed. Will let you know once I have an update

I'm glad to hear this. Hope there is good news soon.

I performed the same test using two different versions (4.1.7/4.2.0-dev-rc2) and found that the new version(4.2.0-dev-rc2) is much better. However, the memory consumption keeps increasing with the number of PublicKey/AnyAddress.
Test case:
            println("Test started")
            val publicKeyHex =
                @Suppress("ktlint:standard:max-line-length")
                "044ba28b11af1561042b03b9d0f940446315af11358aa12d798050b3cf76265dab0f48b22ea1fc1f9560c969b966221f2821b746c4e56efaeaeec8689caf5843c9"
            var i = 1
            while (i <= 2_000_000) {
                val publicKey = PublicKey(publicKeyHex.hexToByteArray(), PublicKeyType.SECP256k1Extended)
                val anyAddress = AnyAddress(publicKey, Ethereum, Derivation.Default) // This leaks memory
                if (i % 10_000 == 0) {
                    println("Test case $i run")
                }
                i++
            }
            println("Test completed")

Result of version 4.1.7 (The native memory usage keeps increasing and has reached up to 759.8MB):

Result of version 4.2.0-dev-rc2 (The native memory usage keeps increasing and has reached up to 197.4MB):

So, I suspect that the problem may not be completely solved.

Hi @vcoolish, do you have any idea why the memory consumption keeps increasing in the test case above?

Hi @10gic, could you please clarify which WalletCore package do you use?
com.trustwallet.wallet-core (maven) or com.trustwallet.wallet-core-kotlin-jvm (maven) or com.trustwallet.wallet-core-kotlin-android (maven)?

Hi @10gic, could you please clarify which WalletCore package do you use? com.trustwallet.wallet-core (maven) or com.trustwallet.wallet-core-kotlin-jvm (maven) or com.trustwallet.wallet-core-kotlin-android (maven)?

Hi @satoshiotomakan, I used wallet-core-kotlin-android in previous tests.

Hi @vcoolish, do you have any idea why the memory consumption is still increasing in the test case above after the fix?

could you try adding delay(1) to your while loop?

Hi @vcoolish, do you have any idea why the memory consumption is still increasing in the test case above after the fix?

could you try adding delay(1) to your while loop?

I think adding a delay in the 'while' loop won't affect the test results. Could you please elaborate on your intention?

Hi @vcoolish, do you have any idea why the memory consumption is still increasing in the test case above after the fix?

could you try adding delay(1) to your while loop?

I think adding a delay in the 'while' loop won't affect the test results. Could you please elaborate on your intention?

GC doesn't clean the memory instantly and without 1 ms delay the test will throw outofmemory. Having 1ms delay simulates more real usage and allows GC to do the job

Hi @vcoolish, do you have any idea why the memory consumption is still increasing in the test case above after the fix?

could you try adding delay(1) to your while loop?

I think adding a delay in the 'while' loop won't affect the test results. Could you please elaborate on your intention?

GC doesn't clean the memory instantly and without 1 ms delay the test will throw outofmemory. Having 1ms delay simulates more real usage and allows GC to do the job

Do you mean that the GC has no chance to clean up memory without a 1ms delay? After I finish the while loop, I think the GC has enough time to clean up the memory. However, I don't see the memory consumption drop after the while loop finishes. Anyway, I will test it with a 1ms delay in each iteration of the while loop.

Hi @vcoolish,  I have retested wallet-core-kotlin-android:4.2.0-dev-rc2 with a 1ms delay in each iteration. Here are the results:

The native memory usage keeps increasing and has reached up to 200.1MB (almost the same as the previous result of 197.4MB). It seems that adding a delay doesn't work.

Hi @vcoolish, I have retested wallet-core-kotlin-android:4.2.0-dev-rc2 with a 1ms delay in each iteration. Here are the results:
The native memory usage keeps increasing and has reached up to 200.1MB (almost the same as the previous result of 197.4MB). It seems that adding a delay doesn't work.

ok, thank you for checking. Sergei has an idea what else can be done between C++ and rust we will investigate

Hi @vcoolish, I noticed that the code changes in PR https://github.com/trustwallet/wallet-core/pull/4070 are not included in version wallet-core-kotlin-android:4.2.0-dev-rc2. Could PR https://github.com/trustwallet/wallet-core/pull/4070 have resolved this memory leak issue?

Hi @10gic, https://github.com/trustwallet/wallet-core/pull/4070 doesn't resolve the memory leak issue for wallet-core-kotlin-android KMP package, but for wallet-core-android Android native package only.
We'll merge the PR into dev shortly

@10gic I debugged the test you provided above, and it turned out that the fix works fine in general. However PhantomReference is acting strange in some stress tests.
For example, in this case ReferenceQueue<PhantomReference> is triggered rare and with small butch of objects to be released. This leads to the fact that we don't release all objects allocated via WalletCore, but only a part of them.
Note that delay for 1ms or even 10ms doesn't help the queue to handle all objects.
        var i = 1
        while (i <= 200_000) {
            val publicKey = PublicKey(pk, PublicKeyType.SECP256k1Extended)
            val anyAddress = AnyAddress(publicKey, Ethereum, Derivation.Default)

            i++
            delay(1)
        }

However, if we move PublicKey generation out of the loop and leave AnyAddress only, all objects will be released without exceptions.
        var i = 1
        val publicKey = PublicKey(pk, PublicKeyType.SECP256k1Extended)
        while (i <= 200_000) {
            val anyAddress = AnyAddress(publicKey, Ethereum, Derivation.Default)

            i++
            // delay(1) // the memory clean up happening even without the delay
        }

@satoshiotomakan Thanks for your debugging. May I ask if there's a plan for when the dev branch can be merged into the master branch?

Hi @10gic, the memory fixes have been passed security review finally. But unfortunately, I can't merge dev branch into master before end of the year. Because we froze all codebases, and push only hot fixes into master since today.
I'll make a new dev branch release now, to make it updated with master.
Merry Christmas 🎅

Hi @satoshiotomakan, is it possible to merge the PR into the master branch in January 2025?
Merry Christmas 🎄.