Pile of issues for bakana v3

Overview
UI updates for bakana version 2.0.0 (currently in alpha, see latest version on NPM).
Major
Inputs
Input preflights have been completely overhauled to add more information. For example, the SE preflight now provides a list of experiments and assays, in addition to the row/column data. Even the MatrixMarket preflight now provides a list of feature types. This enables users to, e.g., pick which feature type corresponds to RNA, select the assay containing the count matrix, and so on. No more guesswork.
This is particularly important for choosing the "primary feature identifier", which will be used to define the intersection of common features when combining multiple datasets. Again, no need to guess the species or Ensembl/symbol nature of the identifiers, as we'll ask the user to confirm it all. (Of course, the initial defaults can be guessed, but at least then it's not so essential that we get it 100% correct.)
This requires a UI change to support dataset type-specific parameters. For example, SEs and H5ADs can have multiple assays and more rowData, whereas the 10X HDF5 and MatrixMarket formats do not. SEs also have multiple alternative experiments that can be used to define different modalities. Once parameters are chosen, they should be passed to the worker to configure the InputsState before runAnalysis.
Quality control
Quality control can no longer be skipped for any modality. That is to say, the QC metrics and thresholds are always produced for each (available) modality; but it is now the responsibility of the cell_filtering step to decide whether or not the filters for each modality are applied to the filtered matrix. This change means that it is now always possible for the UI to color the plot by QC metrics, and to produce other diagnostics that are arguably important even if the user doesn't want to use them for filtering.
I suppose the "skip" arguments should be renamed to "use for filtering".
Summaries
The summary() method no longer exists. Instead, we added many getters, and it is now the application's responsibility to format this information into a desirable message. This provides more flexibility for message formatting as we no longer have to update bakana whenever we decide to change the format a little.
So, the startFun and finishFun arguments to runAnalysis now accept the name of the step. This can be used to extract the appropriate State object, invoke the getters, and pull out all results required by the UI. Be mindful that the getters don't make copies, so we have to slice() any buffers that might get transferred.
Recovering the correct formatting probably requires migrating a few functions from bakana v1.2, e.g., formatMarkerResults, splitQCMetricsByBlock, etc.
Added CRISPR support
Added support for CRISPR quality control (based on the count of the most abundant guide), normalization (by library size) and PCA (just in case anyone wants to cluster by guide). It is also theoretically possible to do a combined embedding involving information from guides and other modalities, though this seems less useful to me. Anyway, the guide information also shows up in the marker tables, so that's another choice.
Many of the datasets also have a choice for the CRISPR feature type, e.g., TenxMatrixMarketDataset allows you to set featureTypeCrisprName, which defaults to "CRISPR Guide Capture".
Minor

[ ] sample_factor renamed to block_factor for choosing the blocking factor, to avoid confusion between samples and datasets when a dataset can contain multiple samples.
[ ] resolution renamed to multilevel_resolution for multi-level SNN clustering, to disambiguate between that and the corresponding resolution parameter for Leiden clustering.
[ ] add "compute AUC?" and "Log-FC threshold" options for marker detection.
[ ] add choice of SNN graph clustering algorithms (Leiden, Walktrap, Multilevel) + their params. This includes leiden_resolution and walktrap_steps, which are only relevant for their respective algorithms.
[ ] weights in combine_embeddings are now explicit per modality, e.g., rna_weight, crispr_weight. Removed support for the null special case, given that crispr_weight = 0 by default.
[ ] quality_control, pca and normalization all have an rna_* prefix now, to better disambiguate those steps for RNA from the corresponding steps of the other modalities.
[ ] fetchFilteredAnnotation no longer exists. Worker should instead call InputsState#fetchCellAnnotations(), pick the column of interest, and then run CellFilteringState#applyFilter() on the resulting column.
[ ] fetchNormalizedExpression no longer exists. Worker should instead call fetchNormalizedMatrix() for the relevant modality, and then call .row() with the row index of interest to get the row of expression values.
[ ] parseKanaFile now returns an object, and the load value must be passed as loadFun to loadAnalysis.

I am rewriting the entire app, haven't made much progress but focusing more on the inputs section

need a hackathon at ur plaece
will bring the toilet paper

changes happening in #158


Extension Auto-Update Mechanism Enhancements

Problem Statement:
Our team creates a few popular VS Code extensions. Within our organisation, we have a continuous release model where it's not uncommon for us to release daily, and sometimes even multiple times each day. We've had a number of users who have expressed dissatisfaction with the current mechanism for reporting extension updates, particularly regarding their frequency and the annoyance of the notification system. This issue has been observed with several of our extensions, exhibited in poor reviews in our marketplace review page as well as issues in our issue tracking system.
We've also observed the same for other extensions expressing similar dissatisfaction, for example, Github Copilot:

1 STAR: Updates are too frequent. It raises questions about their Q/A process. Even ArchLinux has fewer updates than this.

The marketplace reviews are used to highlight the user sentiment / dissatisfaction.
We think that the key issues are:

Users believe (incorrectly) that a pending update means that they must restart their editor to continue using the extension with pending updates vs. continuing to use the old version until the next editor restart.
Users (especially those with diverse sensory / cognitive needs) are distracted by the pending updates badge on the Extensions side-bar badge and want to clear the pending notifications.

Potential Solutions
We propose a possible set of improvements that could be implemented in part, or in whole. Where possible, we're happy to assist in contributing to improving this mechanism to expedite delivery (assuming it aligns with VS Code direction).

Allow extensions to opt-in for auto-update notifications (not auto-updates), either for a specific version of the extension, or for the extension entirely. This would allow extensions (both ours, and others) with continuous delivery models to not distract / annoy users. Perhaps a more general solution would be to notify for major.minor changes, but not patch releases (e.g. patch releases may auto-update without a badge indicator).
Allow the pending updates badge to be completely disabled for those with diverse sensory / cognitive needs.
Allow auto-update frequency and/or notification frequency to be controlled by users (e.g. every restart, once per day, once per week, etc.)

If an extension has updates, user should be notified.

I think this is valid but only IF there is value in notifying them. If there are no release notes, or there is no actionable information for the user (other than to restart the editor), then the notification is just noise and interrupts their flow. I think for the majority of extensions (definitely for ours) that most updates are not critical and only affect a small subset of users (we notify these users to update in response to an issue / feature request).

It should be up to users to opt out of notifications. I think the right approach here is to let users skip updates for this specific extension and also hide notification badge for this extension.

I still think that the providers of the extension should have the opportunity to have a say on this. For example, if there's a critical security update, then I think it's reasonable to notify users of that. I think it's also reasonable to notify users of updates that have actionable information for them (e.g. "we've added a new feature that you can use by doing X"). For our extensions, we do this via a custom webview with a LOT more information than we can fit in the release notes; we let users opt out of this. For example, most of our users don't care to know about a version upgrade for bleeding edge framework / node.js version support.

May I know how often you publish your extension and what kind of changes go in?

The frequency of updates (and what's in them) very much depends on what we're working on. There is a baseload of constant updates to support latest versions of node, TypeScript, frameworks we integrate with, etc. On top of that, there are bigger features releases that are followed by incremental enhancements and bug fixes. Often times, the number of releases aligns with changes to the underlying tech we integrate with.
Here are the number of extension updates for Quokka.js 2023 YTD:

Month
Number

Jan
5

Feb
3

Mar
2

Apr
4

May
9

Jun
5

Jul
2

Aug
13

Sep
4

Oct
9

Nov
23

To give you an idea of some of the November changes:

Add support for<EMAIL_ADDRESS>new way of loading ESM hooks
Fix node@v21 JSDOM compatibility issue
New Quokka Snaps feature
Fix bug with latest version of TypeScript; ignore diagnostics report where Option 'module' must be set to 'NodeNext' when option 'moduleResolution' is set to 'NodeNext'
Improve error reporting when ts-node transforms fail
Update to improve re-using ts-node cache for transformations
Fix bug in latest version of node with tsnode ESM integration
Fix bug not setting jsxImportSource for TypeScript files
Improve value output highlighting and formatting for string values
Improve line mapping for transformed files

Having provided that information, as a team, we definitely (collectively) feel that there's a lot of room for improvement in VS Code's update mechanism, and that its too noisy right now. We are getting the same feedback from our users, and we see the same update patterns and feedback from users on other extensions. We also think this specific issue increases in severity based on the number of extensions a user has installed. If you have 10 extensions installed, then you're going to get 10x the number of updates than if you only have 1 extension. I wouldn't be surprised if many users are being notified of updates every day, and that's just unnecessary noise.
I also think that at some point, there's a straw that breaks the camel's back with updates, and users will remember the last update that happened to annoy them, and will complain about that one, even if there were 10 updates before that one that they didn't notice.
As mentioned above, if there's no actionable information available about an extension update for users, then the update notification is less useful. Sure, a user may need to update VS Code to apply the update, but if there's no value in the update to them, then the notification is just noise and their development flow has been interrupted.
I picked a few popular VS Code Extensions to show that the issue of frequent updates is not just related to our extensions.
Here are some examples of other popular extensions that have frequent updates:

Python - details of changes out of date (last updated on November 2, 2023)

1/12/2023 7:49am
30/11/2023 8:18pm
30/11/2023 6:36am
29/11/2023 8:15pm
28/11/2023 8:19pm

Jupyter - details of changes out of date (last mentioned via VS Code release notes in October 2023).

29/11/2023 7:35pm
29/11/2023 11:00am
29/11/2023 7:06am
28/11/2023 7:24pm
27/11/2023 7:24pm

GitLens - details of changes out of date (last mentioned November 21, 2023)

30/11/2023 7:11pm
29/11/2023 7:12pm
28/11/2023 7:12pm
27/11/2023 7:12pm
24/11/2023 7:12pm

In addition to the GitHub Copilot example, some examples of negative feedback:
Pylance Extension

JavaScript / TypeScript Nightly Build
People even complain about the pre-release build...

In summary: we feel (as evidenced above) that the default way that updates are displayed to users is too noisy. We feel that the VS Code UX needs to change to address this. We propose something like important vs. not-important updates. As extension authors, we would like to define which updates are important to display to users. For example, in terms of possible UX, not-important updates to an extension would not change the badge number, all updates would still be visible in the Extensions View.

@sandy081 - do you need any more information from us at this stage?

@smcenlly Appreciated for detailed reasoning and feedback from users.
There are two kinds of notifications we have at present for extension updates

Number badge on the Extensions Activity Bar - This is considered as prominent notification to the user for extension updates.
Action item on the extension in the view - This is considered as secondary information presentation. This is shown only when user opens this view.

I tend to agree that the first mechanism should be able to consider the kind of extension transitions and should be configurable by the user. For eg., user can configure not to show update badge for a specific extension and then VS Code wont show the badge when there is an update for that extension. But when user opens the extensions view, VS Code should show the actions to update. VS Code should show the update badge for important updates like security fixes etc. This piece of information is missing that is to identify what kind of update it is.


bugfix fix glance list share resource

这个 PR 实现什么功能/修复什么问题:
RT
是否需要 backport 到之前的 release 分支:
release/2.10.0

/area glance
/cc @swordqiu

/area image

Codecov Report

Merging #1877 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #1877   +/-   ##
======================================
  Coverage     7.6%    7.6%
======================================
  Files         412     412
  Lines       63672   63672
======================================
  Hits         4844    4844
  Misses      58419   58419
  Partials      409     409

Flag
Coverage Δ

#aFlag
7.6% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6ef1c68...506005a. Read the comment docs.

/lgtm
/approve


Dataset dependency issue

Collecting pyicu>=1.9.3 (from normality>=0.5.1->dataset==1.1.0->-r requirements.txt (line 16))
  Downloading https://files.pythonhosted.org/packages/c2/15/0af20b540c828943b6ffea5677c86e908dcac108813b522adebb75c827c1/PyICU-2.2.tar.gz (211kB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-nisBrM/pyicu/setup.py", line 53, in <module>
        ''')
    RuntimeError:
    Please set the ICU_VERSION environment variable to the version of
    ICU you have installed.

Trying to pip install with dataset==1.1.0 is resulting in this error from the python:2.7-alpine Docker image.

You can either install py-icu in alpine or pin normality below 1.0.

@pudo do you really need the transliteration features of PyICU?
icu is great but this is a native dependency hence the confusion this may bring at times.
I use instead this in ScanCode https://github.com/nexB/scancode-toolkit/blob/9e04eb55cd4a049409b86dac0b76445d87474c3a/src/commoncode/text.py#L151 with @kmike text-unidecode and this is all Python so lighter weight... https://github.com/kmike/text-unidecode
Would you be opened to have the option to use either pyicu or something like what I use as pure python?
ICU is a tad painful to build on Linux, Windows and macOS so a pure Python way would be welcomed.

@pudo ping ...  this is a problem there https://github.com/nexB/scancode-toolkit/issues/1463 and not because of normality but because scancode depends on fingerprints. Making icu optional would be much welcomed. I can provide a patch for that either as an optional dep, and/or as a try/except to activate ICU features only if ICU is present.

First off, two clarifications: a) this issue should be closed, dataset no longer depends on normality. b) I didn't introduce the hard dependency on pyicu lightly, and I agree that it is a heavyweight dependency that will significantly impact the footprint of an app using this.
The reason that I eventually dropped support for text-unidecode is essentially that it made both normality and fingerprints hard to test and maintain. What normality does is pretty fundamentally a wrapper around transliteration and unicode normalisation. So icu is not tangential, it's the main feature. Using unidecode simply produces different results. This meant that each test assertion in both libraries was brittle. Going further: icu does the thing consistently, unidecode sometimes fails quite badly.
If you wish to commit to maintaining support for both in the long run and throughout both of these libraries, we can talk about it. But I'd really rather recommend you pin to the old version, adopt icu, or even consider forking a separate library that implements unidecode only. Doing both is really going to be ugly throughout.

@pudo thank you for the detailed explanation. As stated in my case this is for fingerprints and I get that you want the highest quality of transliteration. In my case I could deal with much less. Let me see if there is an easy way around thins, possibly with a fake, limited pyicu that I could install before I install fingerprint.
The 10+MB wheel and the pain to build pyicu on all three OSes I support are not worth the pain in my own small use case.

Great thread, just run into install issues with a colleague on a Mac with countrynames which depends on normality ...


Not clear the usage of the --extra-service-accounts flag option for operator-sdk generate bundle command

Bug Report
I am unable to make the flag added in https://github.com/operator-framework/operator-sdk/pull/4826 works. I cannot see my extra SA added to the CSV. Am I doing something wrong? Where this option is documented?
What did you do?
Following the steps to reproduce the issue scenario with testdata/go/v3/memcached/

Create the testdata/go/v3/memcached-operator/config/rbac/service_account_pod.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-pod-service-account
  # If we fixed here the ns then the following error is faced with operator-sdk bundle validate
  # operator-sdk bundle validate ./bundle
  # ERRO[0000] Error: Value my-pod-service-account: error validating object: metadata.namespace: Forbidden: not allowed on this type. &{map[apiVersion:v1 kind:ServiceAccount metadata:map[name:my-pod-service-account namespace:other-ns]]}
  namespace: other-ns

2 - Update the Makefile to use the flag and my extra service account to the CSV:
.PHONY: bundle
bundle: manifests kustomize ## Generate bundle manifests and metadata, then validate generated files.
	operator-sdk generate kustomize manifests --interactive=false -q
	cd config/manager && $(KUSTOMIZE) edit set image controller=$(IMG)
	$(KUSTOMIZE) build config/manifests | operator-sdk generate bundle -q --overwrite --version $(VERSION) $(BUNDLE_METADATA_OPTS) --extra-service-accounts "my-pod-service-account"
	# operator-sdk bundle validate ./bundle

3- Run make bundle
What did you expect to see?
The extra SA in the CSV
What did you see instead? Under which circumstances?
Nothing changed
Environment
$ operator-sdk version
1.19.0
$ go version
1.16
Possible Solution

The makefile has the information about the option and how to use it scaffolded via comments as we have other helpful info on this file to let user knows how to use the available options, e.g:

.PHONY: bundle
bundle: manifests kustomize ## Generate bundle manifests and metadata, then validate generated files.
	operator-sdk generate kustomize manifests --interactive=false -q
	cd config/manager && $(KUSTOMIZE) edit set image controller=$(IMG)
        # You can add the optional  --extra-service-accounts flag to the operator-sdk generate bundle command
        # to add extra ServiceAccounts to the CSV. CAVEATS: "When it is useful? Could this SA has an ns informed or not?"
        # is possible to add the extra SA that will get installed in another ns that is not the Operator ns?
	$(KUSTOMIZE) build config/manifests | operator-sdk generate bundle -q --overwrite --version $(VERSION) $(BUNDLE_METADATA_OPTS) --extra-service-accounts "my-pod-service-account"
	# operator-sdk bundle validate ./bundle

The help for the command operator-sdk generate bundle has a better explanation with some basic examples of how to use it. See that it is not very clear:

$ operator-sdk generate bundle --help
...
      --extra-service-accounts strings   Names of service accounts, outside of the operator's Deployment account, that have bindings to {Cluster}Roles that should be added to the CSV
...

What about amending it as an example as to when/why use this option?

3 - Did we have a further and detailed explanation about this option in the SDK docs?

#### Additional context

- When it is useful?
- Could this SA has an ns informed or not?
- Is possible to add the extra SA that will get installed in another ns that is not the Operator ns?

@estroz, you create this feature. So, could you please give me a hand and let me know if I am not trying to use it in the right way?  Also, could you provide further details and caveats to help us to doc it?

I was also unsure how to use this flag to add service accounts, and found that the missing step in my case was to include the file describing the new service account in the list of resources in rbac/kustomization.yaml.


Bad default value for IRC Charset

Describe the bug
Without Charset="utf-8", matterbridge will fail to recognize UTF-8 encoded characters and thus translate it incorrectly. It says in matterbridge.toml.sample that it will try to detect charset automatically if Charset is unset, but clearly it fails with the easiest case.
Interestingly, it is inconsistent with how it fails. I found that the string åså gets mangled while the string åaå works fine. I'm clueless as to why.
To Reproduce

Setup a bridge between IRC and Telegram like my config below.
Send the string åså to the IRC channel and see how it gets translated to Ã¥sÃ¥ in Telegram.

Expected behavior
Either that the default charset is utf-8 or that it's correctly detecting the charset.
Screenshots/debug logs

Environment (please complete the following information):

OS: linux (Docker)
Matterbridge version: version: 1.19.1-dev 0d997666

Additional context
Please add your configuration file  (be sure to exclude or anonymize private data (tokens/passwords))
[irc.test]
Server="irc.example.com:6697"
RemoteNickFormat="<tg-{NICK}> "
UseTLS=true
Nick="telegrambot"
Password="<my_nickserv_password>"

[telegram.test]
Token="<my_telegram_bot_token>"
RemoteNickFormat="<{NICK}> "
MessageFormat="HTMLNick"

[[gateway]]
name="test"
enable=true

[[gateway.inout]]
account="irc.test"
channel="#test"

[[gateway.inout]]
account="telegram.test"
channel="<negative_int_telegram_group_id>"

Afaik UTF-8 still isn't a standard for irc server, so setting this default will break other people setups.
The manual option is explicit there for when autodetect doesn't work, as in  your case.

https://github.com/42wim/matterbridge/issues/1118 is the same issue. I agree that defaulting to UTF-8 would work better in 2020 and maybe having an option for autodetection mentioned in the FAQ alongside it causing issues with valid UTF-8 could be better.

@42wim I think you are right that UTF-8 is currently not the only standard out there, and it ought to differ from server to server. But what's the most common one? Unless we have convincing statistics that tells us otherwise, I think it's a strange choice to set anything other than UTF-8 as the default.

At the moment matterbridge tries to autodetect your charset, if it fails, it falls back on ISO-8859-1.
If you have fixes for the autodetect, please send a PR.
But the problem with changing defaults is that it might work for you, but it breaks for other people.
The option to choose your charset is there, so when you need utf-8, you can configure it for utf-8.

But why should it fallback to ISO-8859-1? Surely that's the edge case, not UTF-8?


Implement support for action workflows

This PR adds support for managing the most recent update of Action Workflows.
The branch was created from PR #1402 .
Relates to: #1399

Thank you very much for the really great review, @gmlewis :slightly_smiling_face:

@martinssipenko  - if you are comfortable, please feel free to review this PR and then we can hopefully get it merged after you approve it and after we get #1402 merged.

@googlebot I consent.

LGTM

LGTM

Thank you, @martinssipenko!  Awaiting the merge of #1402 now before merging this one.

Thank you for the reviews, @martinssipenko @gauntface @gmlewis !! :tada: :tada: :tada: !!

Thanks, @gauntface.  I'm going to hold off merging this one until #1402 gets another LGTM and merged... as this is based off of that one.

Thank you, @joshuabezaleel!
Could you please check one more time for the opt to opts now that we merged #1402?

(You will find them in files that you didn't previously modify, by the way.... and you might need to perform a "git merge master" into your branch to see them.)

I hope that should do the trick. cc: @gmlewis


Controlling the locale: documentation + examples

This PR adds a new chapter 19.6 Controlling the locale to the manual, as proposed in  #1326.
Also sample files are added to the picocli-examples module.

Merged. Thank you very much! I will update the release notes tomorrow. Bedtime for me now. :-)


Revolver fixe

About the PR
Revvies deal more damage (34-36 to 40-42 i thimk)
Speedloaders (with bullets) are now printable in seclathes
Uplinks sell the proper speedloaders now
Screenshots

Changelog

:cl: BurninDreamer

tweak: Revolvers should be a little more punchy now, yes, even more than before.
fix: The Syndicate now distributes proper ammo in their uplinks for said revvies, uh oh.
add: NT has tweaked seclathes to be able to produce speedloaders with revolver ammo!

sloth fixed the main issues with another pr this can go now


JSON parsing error when attempting to execute most commands

Hello all, I've installed ZAT to my system (macOS Catalina) and I cannot execute any commands, as I get the same exception each time:
❯ zat update
/Users/marco.pagani/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:153: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/Users/marco.pagani/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:153: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/Users/marco.pagani/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:153: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/Users/marco.pagani/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:153: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/Users/marco.pagani/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/faraday-0.9.2/lib/faraday/options.rb:153: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
Traceback (most recent call last):
	11: from /Users/marco.pagani/.rbenv/versions/2.7.0/bin/zat:23:in `<main>'
	10: from /Users/marco.pagani/.rbenv/versions/2.7.0/bin/zat:23:in `load'
	 9: from /Users/marco.pagani/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/zendesk_apps_tools-3.8.1/bin/zat:13:in `<top (required)>'
	 8: from /Users/marco.pagani/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/thor-0.19.4/lib/thor/base.rb:444:in `start'
	 7: from /Users/marco.pagani/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/thor-0.19.4/lib/thor.rb:369:in `dispatch'
	 6: from /Users/marco.pagani/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/thor-0.19.4/lib/thor/invocation.rb:126:in `invoke_command'
	 5: from /Users/marco.pagani/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/thor-0.19.4/lib/thor/command.rb:27:in `run'
	 4: from /Users/marco.pagani/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/zendesk_apps_tools-3.8.1/lib/zendesk_apps_tools/command.rb:216:in `update'
	 3: from /Users/marco.pagani/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/zendesk_apps_tools-3.8.1/lib/zendesk_apps_tools/cache.rb:23:in `fetch'
	 2: from /Users/marco.pagani/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/zendesk_apps_tools-3.8.1/lib/zendesk_apps_tools/cache.rb:33:in `local_cache'
	 1: from /Users/marco.pagani/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/json-2.3.1/lib/json/common.rb:263:in `parse'
/Users/marco.pagani/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/json-2.3.1/lib/json/common.rb:263:in `parse': 784: unexpected token at '' (JSON::ParserError)

This occurs with any of the zat commands and I've had no luck zeroing in on the problem. I fully cleaned my ruby environment and reinstalled it (rbenv running 2.7.0) I also updated the json gem but I'm stuck as to where I can go from here. Any help is appreciated

After much pain and suffering I've found the issue. This error occurs if your root directory has a .zat file that's empty. It seems that I cleared the contents of that file at some point in my setup process. You can simply delete it and run zat validate to generate a new one and carry on.


Question: Refactoring Cucumber test Framework to Playwright BDD - passing a config around

I have successfully refactored the cucumber world functionality into a fixture, however I am having an issue when I try running in that the common config file is not being found
Error: No environment variable found for COMMON_CONFIG_FILE

      at ../src/env/parseEnv.ts:8

      6 |     const value = process.env[key]
      7 |     if (!value) {
   >  8 |         throw Error(`No environment variable found for ${key}`)
        |               ^
      9 |     }
     10 |     return value;
     11 | }

git@github.com:RichardCariven/react-app-playwright-bdd.git that is the repo
I think the issue is in how the cucumber project version passed in the config file and I need to change this for playwright.
As previously shown the command to run the original Cucumber runner was
const common = `./src/features/**/*.feature \
                --require-module ts-node/register \
                --require ./src/step-definitions/**/**/*.ts \
                -f json:./reports/report.json \
                --world-parameters ${JSON.stringify(worldParameters)} \
                --format progress-bar`;

const dev = `${common} --tags '@dev'`;
const smoke = `${common} --tags '@smoke'`;
const regression = `${common} --tags '@regression'`;

With the following scripts in Package.json
"scripts": {
    "precucumber" : "rimraf reports && mkdir reports && echo {} > reports/report.json",
    "transpile": "rimraf dist && babel --extensions .ts --out-dir dist src",
    "cucumber-compile": "yarn transpile && cucumber-js",
    "cucumber": "COMMON_CONFIG_FILE=env/common.env yarn cucumber-compile",
    "postcucumber" : "COMMON_CONFIG_FILE=env/common.env ts-node ./src/reporter/cucumber-report.ts"

Is there an equivalent way of defining and passing the location of COMMON_CONFIG_FILE in the playwright config, or do I need to do this another way? Just as a reminder below is the Fixture I have now written to replicate my Cucumber World functionality.
import dotenv from 'dotenv'
import { env, getJsonFromFile } from '../../env/parseEnv'
import {
    GlobalConfig,
    HostsConfig,
    PagesConfig,
    PageElementMappings, GlobalVariables,
} from '../../env/global';
import fs from "fs";
import {test as baseTest } from 'playwright-bdd'

type PageMapType = {
    worldParameters: GlobalConfig
    globalVariables: GlobalVariables
}

const pageMapfixture = baseTest.extend<PageMapType>({
    worldParameters: async ({}, use) => {dotenv.config({path: env('COMMON_CONFIG_FILE')});
        const hostsConfig: HostsConfig = getJsonFromFile(env('HOSTS_URLS_PATH'));
        const pagesConfig: PagesConfig = getJsonFromFile(env('PAGE_URLS_PATH'));
        const mappingFiles = fs.readdirSync(`${process.cwd()}${env('PAGE_ELEMENTS_PATH')}`);
        const pageElementMappings: PageElementMappings = mappingFiles.reduce(
            (pageElementConfigAcc, file) => {
                const key = file.replace('.json', '');
                const elementMappings = getJsonFromFile(`${env('PAGE_ELEMENTS_PATH')}${file}`);
                return { ...pageElementConfigAcc, [key]: elementMappings };
            },
            {}
        );
        await use({hostsConfig, pagesConfig, pageElementMappings})
    },
    globalVariables: async ({}, use) => {
        await use({currentScreen: ""})
    },
});

export const test = pageMapfixture

This is the env function
export const getJsonFromFile = <T = Record<string, string>>(path: string): T => {
    return require(`${process.cwd()}${path}`);
};

export const env = (key: string): string => {
    const value = process.env[key]
    if (!value) {
        throw Error(`No environment variable found for ${key}`)
    }
    return value;
}

export const envNumber = (key: string): number => {
    return Number(env(key));
};

Could you share the command that you use to run Playwright + playwright-bdd?
If it's the following:
COMMON_CONFIG_FILE=path/to/.env npx bddgen && npx playwright test

then COMMON_CONFIG_FILE is provided only to the first command (npx bddgen) and that is the reason of error.
To solve it you can use cross-env package:
npx cross-env-shell COMMON_CONFIG_FILE=path/to/.env "npx bddgen && npx playwright test"

I will add a note to the documentation as it is very valid use-case!

I was trying to pass the config file in through the Playwright Config by setting
require('dotenv').config({ path:path.resolve('../env/common.env')});
applying your solution works thanks


Multiple NICs case not possible

Per readme, I can have only one NIC:
The following devices are supported by the Solo5 virtio target:
a single virtio network device attached to the PCI bus
Is there intention to support multiple NICs? Firewall  is an obvious usecase.

This is on the roadmap and likely to be done within the next few months, however not in the next formal release (#217).

are there any updates on multiple NICs ?

Not at the moment.


Release signet 0.12.0

This version now requires Ruby 2.4.
Support array values of the "aud" field.
Normalize the version constant to match related gems.

Note: Autorelease may not be set up properly for this repo yet. I'm going to try to use it, but if it fails, I'll release manually after the PR is merged.

Is signet using google-style now?

No it isn't. I'm deferring moving things to google-style (and updating google-style) until after we've updated the gems with their version dependencies.

It should be as of https://github.com/googleapis/signet/pull/133

Do we want to bump the google-style dependency to ~> 1.24.0 to match the new minimum Ruby version?

Release is at https://github.com/googleapis/signet/releases/tag/signet/v0.12.0

The release build has started, the log can be viewed here. :sunflower:

The release build failed! Please investigate!

The release build has started, the log can be viewed here. :sunflower:

The release build failed! Please investigate!

The release build has started, the log can be viewed here. :sunflower:

The release build failed! Please investigate!

The release build has started, the log can be viewed here. :sunflower:

The release build failed! Please investigate!

The release build has started, the log can be viewed here. :sunflower:

:egg: You hatched a release! The release build finished successfully! :purple_heart:


Broken links for xml documentation

Hi,
I was looking up some examples and hoped that the new 2017 version had anything new, but reaching this point, the links were broken. I didn't found the right link location to correct, some simple approaches din't worked out, So... here is the site url
https://github.com/Microsoft/visualstudio-docs/blob/master/docs/ide/supplying-xml-code-comments.md
Kind regards

Hello,
When I go to the live topic for the link you supplied (Supplying XML Code Comments - https://docs.microsoft.com/en-us/visualstudio/ide/supplying-xml-code-comments), the links work okay for me. Are they still broken for you?


How to reference a new ec2 to a new TargetGroup dynamically

What is the best approach to create a new ec2 instance, and then reference this ec2 instance to TargetGroup ?
I can see here  that there is targetType that I can use for static ec2 instance id or  ip (and more), but there is no any ec2IdRef or something similar (like vpcIdRef or other object references).
Do you plan to support it? any other idea of how to referenca a new ec2 instance to a target group dynamically ?

No referencers for spec.forProvider.targetType have been implemented so far. We have no plan to work on it by ourselves but we are open to accept a PR if you want to work on it.


[Bug] Consistent: invalid_request_error - Anthropic API

Bug Description
Claude 3.5 chats/API were working, then I encountered what I presumed was a cache corruption issue with NC, so I reset, restarted and NC worked again.
After restarting NC anthropic API did not work, soI got a new API, pasted it, and this recurring error:
{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "max_tokens: 10000 > 4096, which is the maximum allowed number of output tokens for claude-3-5-sonnet-20240620"
  }
}

As per Anthropic docs:
400 - invalid_request_error: There was an issue with the format or content of your request. We may also use this error type for other 4XX status codes not listed below.
Steps to Reproduce

Paste new API in "model provider/Anthropic" .
Attempt Claude 3.5 chat
Error as above

Expected Behavior
Claude 3.5, chats work.
Screenshots
No response
Deployment Method

[ ] Docker
[ ] Vercel
[ ] Server

Desktop OS
macOS m1 14.5
Desktop Browser
No response
Desktop Browser Version
2.12.4
Smartphone Device
No response
Smartphone OS
No response
Smartphone Browser
No response
Smartphone Browser Version
No response
Additional Logs
No response

It tells you where the problem lies: "max_tokens: 10000 > 4096, which is the maximum allowed number of output tokens for claude-3-5-sonnet-20240620".
Try to modify the max_tokens parameter.

I changed Anthropic plan to Build Plan (had $5 free credit before), changed Max Tokens in NextChat to 4096, didn't work.
Deleted Anthropic API, recreated, pasted in NC, tried again, worked.
Changed back to 10000 in NC, worked.
Max tokens in Anthropic free vs Build Plan (Tier 1)? (Rate Limits in account doesn't specify)


fix: migrate LastRewardsAuctionEndTime

Description
Migrate LastRewardsAuctionEndTime to NextRewardsAuctionEndTime when upgrading x/liquidfarming.
This is done by using same key for both values.
Also re-orderer store keys in x/liquidfarming.
Tasks

[x] Fix migration
[x] Write store migration test

References

According to internal discussions, instead of migrating the existing liquidfarming, a new liquidfarming module with a new name will be added, and the existing liquidfarming will be safely removed from v6 to reduce the migration risk.  Therefore, I think this PR can be changed or closed

I'll close the PR.


Update dependabot.yml

Codecov Report
Patch and project coverage have no change.

Comparison is base (518b7d4) 0.00% compared to head (69f679a) 0.00%.

Additional details and impacted files
@@          Coverage Diff          @@
##            main     #90   +/-   ##
=====================================
  Coverage   0.00%   0.00%
=====================================
  Files          1       1
  Lines          2       2
=====================================
  Misses         2       2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Some questions and issues.

Hello, I would like to ask you several questions.

It is still maintained that it is necessary to record the IQ signal with the .fc32 file in GNURadio for Octave to process the process_file.m? The bursts that are stored when you record the signal in temp/droined_debug, which are obviously smaller files, aren't they still useful to obtain the result?

The versions that are in the branches of rewrite and update-3.10 do not work for me in octave when I execute the process_file. It gives an error that it cannot find the Turbo decoder (remove_turbo). Any idea why it gives me this error?

Thanks

Those smaller files are great if the GNU Radio logic successfully detected the burst.  Definitely a lot faster than chewing through gigabytes worth of IQ samples.  Apologies if I am misunderstanding the question.  Side note, you can use whatever format you want so long as you tell process_file.m what the type is [1]
At this time you need to build the remove_turbo binary yourself.  I didn't realize there was nothing in the docs stating that you needed to build the program.  See [2] for information about how to build the remove_turbo application.

[1] https://github.com/proto17/dji_droneid/blob/main/matlab/updated_scripts/process_file.m#L36
[2] https://github.com/proto17/dji_droneid/blob/main/cpp/remove_turbo.cc#L1-L7

Thanks for answering.
This is what I have done so far:

I managed to compile the remove_turbo as you told me to do.

I have some doubts about why there are so many branches or which one I should use specifically. I am currently testing with update-3.10.

I have seen that just about 3 seconds of recording in IQ at 30.72 MSps implies almost 400 MB of file, therefore almost 1M samples when applying octave - process_file.

With the drone_id_test.grc, when I put the central frequency in the center of the burst transmission, the gnuradio informs me in the console:
-Found burst @ 358 / 357
-Sending message with 27952 elements (27952) starting at 357
And so on continuously increasing the numbers each time.
Thousands of burst_# files appear in the /temp/droined_debug folder.
I don't know if this would be right or if you mean that the gnuradio was able to find the bursts automatically with the logic that uses.

I am saving with the file_sink block a .fc32 file (a few seconds) which I then use in process_file, giving it the path, the sample rate of 30.72e6 and the offset in 0e6.
When running it in octave it takes about 10 minutes (more than 86 thousand samples) to finally get this on the screen:

warning: Skipping burst at offset 612 as the beginning of the burst has been clipped
-warning: called from
extract_bursts_from_file at line 62 column 13
process_file at line 49 column 8
And so on consecutively increasing the numbers in the offset. Until the end shows a "Killed" ending the octave.

I keep trying but I'm running out of ideas.
Greetings.

I’m using the update 3.10 branch now and playing back my previously recorded IQ file at the settings you describe yields bursts in the temp directory. I can then either use the 2GB+ file itself or the smaller burst files (I cat them together into one file) in Octave and the end result is Frames that have been decoded. I’ll test my other IQ files and see if I can replicate what you’re seeing. Maybe it’s possible the initial capture was bad.
I was less then a few feet away in a very open area without WiFi etc around.

Yay!
The branch situation is terrible.  I wanted to get the code up on GitHub as quickly as possible and ended up making way too many confusing branch names in the process :(
Yup, the process_file.m route with multi-second collects is not ideal.  To help with this I created a GNU Radio graph that uses gr-fosphor to watch for DroneID bursts (they happen at an even cadence and are 10 MHz wide, so pretty easy to spot) and only record samples when the signal is active on the current frequency
I don't have the graph in front of me right now, but that sounds right.  There will be one file for each burst found.  But, if your RX parameters aren't right you can get loads of false positives.  As an example, using a HackRF I can get mountains of false positives by setting the gain too high.  Might be due to an issue in the cross correlation normalizer.  Thousands is definitely too many and there's likely something else happening.
Any particular reason for the high sample rate?  You only need to use 15.36 MSPS

The most likely reason for your issues is that the GNU Radio branches (aside from the rewrite branch) are behind the main branch for the MATLAB code.  This is somewhat by design as the GNU Radio code was tested against the MATLAB that's in it's branch if that makes sense.  Check out the main branch if you want to use the MATLAB code.  It's a cluster for sure, but that's the solution for the moment.  I hope to be able to fix the other branches once the rewrite is done.

The burst don't show every 600ms? I see the signal is there, 10 MHz BW. This is not the burst with the remote id data?
You said: "and only record samples when the signal is active on the current frequency"
I get loss at this point :(

It's hard to explain, but basically there are at least 15 or so frequencies that DroneID hops around in.  It only stays on one frequency for a few seconds.  So, if you just tune to a frequency and collect samples for a second or so, you don't have much chance of actually catching a DroneID burst because it could be at one of the other frequencies.  What I do is use gr-fosphor to watch the spectrum live at a specific frequency.  When the bursts show up on that frequency I start recording those samples.  I stop once the drone moves off to another frequency.  Then I know the collect has DroneID samples and I won't be wasting a lot of time processing noise or random other signals (WiFi, Bluetooth, etc).  But, you have to know what a DroneID burst looks like in the power spectrum for this to work, so it might not be a good path for you at this point.
The bursts are sent every 600 ms, but the signal hops around after 20 or so bursts, so your odds of catching it are slim.
I forgot that the graph uses 30.72 MSPS.  That can be dropped to 15.36 MSPS if processing power is too high.  Also, the higher sample rate causes increased runtime for Octave/MATLAB, so lower is better.  Just happens that 15.36 MSPS is the lowest you can go.

I have a strong signal with 10MHz BW, sometimes hops and move to other freq, but most of the time stay there. When I turn on the control, the spectrum shows a really fast freq hops around all the band (80 MHz of Wifi and with almost 15 hops), and the signal has less than 2 MHz of BW. But when I turn on the dron, then the signal of 10 MHz appears and like I said, stay in the same freq for a long time, but of course, change from time to time to another freq.
So the signal of RemoteID is on that one, 10 MHz BW? When I record my IQ signal I put the freq in the middle of the signal. The gr-fosphor shows in the waterfall like a burst shows up and down really fast.

There are three signals that come from the Ocusync drones:

Video downlink: can be 10 or 20 MHz in bandwidth, are very frequently transmitted, and tends to stay on one frequency
Control uplink: usually 1.8 MHz wide, only sent once per frequency it hops across
DroneID downlink: always 10 MHz wide, 8 to 9 OFDM symbols long, hops around, stays on one frequency for ~ 20-30 bursts, and sends bursts at a rate of 1 every ~ 600 ms

What you're describing sounds like you are looking at the downlink.  To be extra sure that's not happening, you can use the DJI app to lock the video signal (downlink) to 5.8 GHz, and only look in 2.4 GHz for DroneID signals.

Thanks once again.
Now if my doubt about this question is clarified, thanks to the masterful explanation that you have just given.
Obviously I'm only looking at the Video Downlink and the Control Uplink.
I have never seen another signal in the spectrum like the DroneID, maybe because I had not paid enough attention. I'll have to look more closely.

I think I already have the answer and it is not the one I would like. I have continued reading and investigating and I have realized (too late I think) that my drone does not have OcuSync, it is a Phantom 4. Could you confirm if it could be done with this drone model or is it simply for the Mavic? as in your case the Mini 2.
I read in an article that the OcuSync Protocol is used by the DJI Mavic series, Air series and Mini.
My drone does not have a way to go to 5.8 GHz. Also in the spectrum I see the 10 MHz Video Downlink signal from BW, and those from the Uplink Control with a freq hop as you say. Apart from this, no other signal is evident in the spectrum. According to what you say, should another 10 MHz burst come out every 600 ms and hop in frequency every 20-30 bursts? This means at least a permanence of this RemoteID signal of about 10 seconds in each frequency. Would it be similar to the Video Downlink, at least visually in the spectrum?

So it is impossible or very difficult to see the DroneID bursts with gr-phosphor in real time? In a graph of frequency against amplitude and with waterfall included?

I was looking at the video that I found now on Youtube from the cemaxecuter channel:
https://youtu.be/wAnU-4JQPXQ
Here he identifies with the phosphor the burst of the dronID. Watch it if you can. He is showing how to use your project.

If the drone is with Lightbridge 2, then it is not possible? This is only for OcuSync?

Forgive my lack of knowledge. I've seen the images and I've analyzed everything lobart has been asking and the answers you guys give it to him. With what software were these images obtained? To process the IQ signal previously recorded by the GNURadio.

Thanks.

I didn't mean to imply that you couldn't identify DroneID bursts live, just that you need to know what you're looking for.  Also, it's pretty easy when you have a single drone active, it's a bit more difficult with multiple drones since the cadence of the bursts might be goofy.
I don't know if the previous generation drones send out DroneID frames or not.  I suspect all of the WiFi based drones do, but I don't know about the non-WiFi drones.
My analysis tool of choice is baudline.  Doesn't do demodulation, but does do a wonderful job for analysis of new signals.


 v2 vs v1?

I have successfully connected my LG refrigerator, however it's been added as a v2 device (I had to use the beta build to get it to work.) This means that the door shows up as a sensor, but I can't see any of the other data from the refrigerator. I don't quite understand the difference between v1 and v2, and am wondering if there is something I can do to get more data out.
Thanks for the hard work - this software is fantastic!

I'm also seeing this, got everything connected, see the V2 refrigerator, but only see it as a door sensor.  Validated the device is using the proper device handler.

I'm also seeing this, got everything connected, see the V2 refrigerator, but only see it as a door sensor.  Validated the device is using the proper device handler.

There are two lg server.
v1. One server is communicated with https. (polling every min)
v2. Other one is communicated with https & mqtts (no polling, notify immediatley)
Refrigerator function.

door
set a temperature  (I just remove this)


Structured exceptions

PS: This uses https://github.com/p1c2u/openapi-core/pull/89 as a base just in hope not to have to deal with conflicts twice :)

@p1c2u could you help me why these exceptions are left out? https://github.com/p1c2u/openapi-core/blob/master/openapi_core/shortcuts.py#L34
The problem is, they get raised on validation of response.

@domenkozar it's to filter out body validation errors (OpenAPIRequestBodyError, OpenAPIMediaTypeError, OpenAPISchemaError)

@p1c2u why does it need to filter out body errors? Those are also part of the schema validation?

I'm confused about the following:

InvalidSchemaValue (in schema validation) gets caught and reraised as InvalidMediaTypeValue (which is a "body error")
MissingSchemaProperty (in schema validation) gets propagated normally and is not considered as a body error

@domenkozar
It filter out body errors because validate_parameters should return only parameters errors.
Of course MissingSchemaProperty should be caught too. Didn't notice it.

OK, I'll fix that. @p1c2u do you otherwise think this is good? I'd like to avoid PR long too open as it will often get into conflicts (as it already does).

LGTM. Can you resolve conflict and we can merge it.


Add an option to control the shard states in the placement

1, Added ShardStateType in placement/types.go:

By default we will use TransitionShardState, which includes Initializing, Available, Leaving states, for M3DB and M3Agg
SimpleShardState will only have shards in Available state, can be used by the cache or indexer

2, Algo will pick up the new option in tryCleanupShardState()
@xichen2020 @robskillington

Coverage increased (+0.2%) to 83.948% when pulling 60acbfa40c5b6863bb78936895c6d24106a0b335 on chao-only-available into 356ac9438874dabebfb0a4707580b42d8c865aa2 on master.


Pause/Resume

Is it possible to add pause/resume download button?

It is possible, willing to create a PR?

What is a PR?

It is possible, willing to create a PR?

We can agree on the work and payment, please text me on Telegram @ZiedAmmar


Use InfinispanTestResource in the Infinispan integration tests

Quarkus 2.5.Final is introducing newer version of Infinispan but also the Infinispan test resource. It could be relevant to use this testing class helper in our test integration class
More information here : https://github.com/quarkusio/quarkus/blob/main/docs/src/main/asciidoc/infinispan-client.adoc#testing-helpers

Hi @zbendhiba , can i work on this issue i'm new to open source.

InfinispanTestResource does not seem to support setting a random port

Seems you can set the port via initArgs if needed.

I can only set a constant

Yes, I think you're right...

So can we live with a fixed port, or we propose some change to InfinispanTestResource to use a random port (ideally by default)?

So can we live with a fixed port

Probably in the short term. We do it for some other things out of necessity.

propose some change to InfinispanTestResource to use a random port

I guess we should do that. Looking again at InfinispanTestResource I don't actually see the port arg being used at all to set up the container. I guess because InfinispanContainer hard codes the port to use.

I don't actually see the port arg being used at all to set up the container. I guess because InfinispanContainer hard codes the port to use.

That's interesting! We should definitely report that.

I may be out of context and in such case, please discard my message, but isn't the test resource already using a random port ?
My understanding is that the call to withExposedPorts(11222); setting the port from the point of view of the container but  for the host, then it will be a random port and you can get it with container.getMappedPort(11222). The test resource seems to do so here: https://github.com/quarkusio/quarkus/blob/main/test-framework/infinispan-client/src/main/java/io/quarkus/test/infinispan/client/InfinispanTestResource.java#L45-L46

I may be out of context and in such case, please discard my message :) , but isn't the test resource already using a random port ?

Yeah, my bad. It should work as you describe.

@ppalaga can you please assign to me!

FYI - I already did some hacking on this when I recently added a bunch of new Infinispan tests:
https://github.com/jamesnetherton/camel-quarkus/commit/b9469594d8dd595e05d37251bbbb0608723a41a6
The reason I did not proceed with it is because <metrics accurate-size="true"/> is not set in the container Infinispan configuration, so the statistics tests fail. So either we need the Inifnispan / Quarkus folks to add an option to enable it, or we just live with asserting that the statistic values are all returned as -1.

Thanks @jamesnetherton I ll check with someone from the Quarkus team.

Yes, I assumed at the beginning this could be easy.
Removing the "good first issue" flag

@svkcemk  FYI thread

@zbendhiba @jamesnetherton we can totally add an option for the statistics size check in metrics. however size operation should be working and retrieving the accurate size. Using another assertion to check the size instead of the statistics, would work for this case?

@zbendhiba I created this issue https://issues.redhat.com/browse/ISPN-13810

Thanks @zbendhiba

@zbendhiba @jamesnetherton we can totally add an option for the statistics size check in metrics. however size operation should be working and retrieving the accurate size. Using another assertion to check the size instead of the statistics, would work for this case?

We actually already do that. The Infinispan component supports a size and stats operation. They're two different things as I guess stats can provide some additional stuff outside of just getting cache size etc.
So if there's some way to enable <metrics accurate-size="true"/> in the container, it'd be useful for our testing.

In my opinion, assertions based on statistics values should never be done. We may (and we do) cache stats for performance reasons, so they may never be accurate for tests. My suggestions is to change the tests not to rely on stats.

This was done for 3.8.0 in https://github.com/apache/camel-quarkus/commit/22b77edd4bc3bc7d8de3e1c0c0e751e6dbf81daf.


MS COCO

If i skip MS COCO step, the error will occur or only the performance will reduced??

If you still have a question please re-post with enough details that it is possible to answer. Closing for now, thanks.


Supporting 'Consumption' Azure API Management SKU

Terraform has recently added support for newer Consumption SKU for Azure API Management it seems.
https://github.com/terraform-providers/terraform-provider-azurerm/issues/6256
Would it be possible to include this in Pulumi? Thanks.

We track the upstream releases, so everything through 2.7.0 is released. We typically push out an update of our providers within 48 hours of upstream releases, so you'll see this get picked up once that happens.


RT id values out-of-sync across profiles

WAU.profile.BF2.BFLC.json
RT "WAU RT BF2 BFLC Admin Metadata" -->
"id": "WAU:RT:BF2:BFLC:AdminMetadata"
This RT is referenced in the RDA profile and monographs profile,
but, for example
WAU.profile.RDA.v04.json
RT "WAU RT RDA Work" -->
PT "Administrative Metadata for Work" -->
"valueConstraint"/"valueTemplateRefs": "["WAU:RT:BF2:AdminMetadata"]
(no match)
and, also for example
WAU.profile.RDA.Monograph.v04.json
RT "RT RDA Work for monographs" -->
PT "Administrative Metadata for Work" -->
"valueConstraint"/"valueTemplateRefs": ["rt:bf2:AdminMetadata"]
(no match)

:question: @gerontakos, since XSLT will generate future-version WAU.profile.RDA.Monograph.v0X.json, id edits are only needed in the RDA and BFLC profiles?

@CECSpecialistI
@alexiskeely
2019-07-26:
RT IDs still seem out of sync.

WAU:RT;RDA:Manifestation
WAU:RT:RDA:MANIFESTATION
etc.

Not attempting to fix IDs/references/etc. today
BUT
We need to identify what RT IDs should be (capitalization) and fix moving forward.
Thank you!


[Portal] Migrate to Angular signals

[ ] upgrade to Angular v16
[ ] migrate to signals

Related package: https://github.com/Azure-Samples/contoso-real-estate/tree/main/packages/portal

Can i take this up @manekinekko?

Sure @sumitparakh. Thank you 🙏

Thank you @manekinekko 🤩


All indices of an element in an array.

Related Issue

Added all indices of an element in an array.

[x] DevIncept Participant

[ ] Contributor

Closes: #33
Describe the changes you've made
I added solutions to this problem.
Type of change
What sort of change have you made:

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Code style update (formatting, local variables)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Describe how have you verified the changes made
Checklist:

[x] My code follows the guidelines of this project.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly whereever it was hard to understand.
[x] I have made corresponding changes to the documentation.
[x] My changes generate no new warnings.
[x] I have added test cases that prove my fix is effective or that my feature works.

PR approved. Fab work. Keep rocking. 👌🙌@siddhi-244

Thank you maam!


Segmenting Files with customed data segmentations - Using SegmentAnnotation Class Or must I need to feed XML files?

Let's consider the use case when there is a database that comes already with segments descriptions, but lacking labels.
Is it possible to feed a SegmentAnnotation Class, or dictionary, into the segment_files_parallel() function ?
/pykanto.utils.types.html#pykanto.utils.types.SegmentAnnotation
https://nilomr.github.io/pykanto/_build/html/contents/segmenting-files.html
Looking at your tutorial, it seems one must have a copy of XML files, that are used to prepare the files to segment:
files_to_segment = get_wavs_w_annotation(wav_filepaths, xml_filepaths)
and then feed the list of tuples (wave_paths, xml_paths) in:
segment_files_parallel(
    files_to_segment,
    ...
)

Instead, I would like to pass tuples of (wave_files, info_about_segments) where info_about_segments could be a dictionary, or SegmentAnnotation object.
As an example, consider this workflow:

I may load a dataset in pandas:

From there, I already have info about the segments starts/ ends, but not the labels (which are the things I want to classify by clustering)

I would like to avoid to create temporary xml files on disks, and instead extract directly a dictionary, or a class, for each row on the fly, and then pass it to segment_files_parallel( ).

Is it possible?

Hi,
That is how things work, yes. You can either

Prepare your dataset so that it can be ingested using pykanto (i.e., have a metadata file per audio file), or
Modify the relevant functions to do what you want, e.g. like you mention you could pass a SegmentAnnotation object or similar to https://github.com/nilomr/pykanto/blob/2f92a9406f3618d8f6c8ca5acc6bb3b0326704fb/pykanto/signal/segment.py#L261


Implement scoped spawn

An attempt on implementing scoped spawn.
Using regular tasks is inconvenient because they require all references to be static and this is difficult when you construct a lot of stuff in the main function. So far I've been using a huge select_biased! block to poll all of the anonymous futures, but this is very inefficient, because all of them share the same waker, which means that on wake up each of the futures has to be polled. This PR makes it possible to spawn anonymous futures that use references to the stack and schedules each independently.
I wasn't sure if it's possible to cancel a spawned task, so dropping an unfinished guard results in a panic.
Getting this to work was kind of tricky and I'm not sure if it is the best approach, so I'm open for suggestions.

Interesting! Some general thoughts:
1- This is adding two separate features:

Allow spawning tasks that borrow data from another.
Allow tasks returning a result that can be collected by the parent task.

IMO they are mostly orthogonal, for example you might want to collect a result from a non-scoped task. The result feature should be available everywhere, or we should add only scoped tasks for now and defer the result for later.
(The result feature has some tricky edge cases. For example, if the parent task doesn't collect the state, the child task stays in STATE_RESULT_PENDING forever, and can't be reused. Especially for non-scoped tasks, this means they all need to have a "JoinHandle" and track more info, like "is the JoinHandle live?" to avoid this issue, which adds nontrivial code size and complexity)
2- Forcing the use of dyn is unfortunate, this should be avoidable with TAIT + macros or perhaps a different design that allows good old Rust's type inference to do it.
3- The TaskStorage and ScopedTaskStorage duplication feels a bit ugly to me, maybe it'd be possible to unify them especially if we hold off on the "task result" feature.


Atom command line opens up a Windows shell in Git Bash.

Description
The atom command line no longer opens files in the editor in Windows 10. Instead it opens up a type of shell and expects input. This happens in Git Bash but not cmd. It's happening on multiple Windows 10 machines around the office.
Steps to Reproduce

Open up Git Bash on Windows
Cd to a folder you want to open in atom
Run atom . or atom foldernamehere to open up a folder

Expected behavior:
Atom will open up a new window with the folder added as a project folder
Actual behavior:
A type of shell opens up and expects input from the user. See screenshot.

Versions
Atom 1.7.4
Windows 10
Git Bash 2.8.3 MINGW64
Workarounds
These workarounds may or may not work depending on your bash versions.
Running atom.cmd
Running atom.cmd . will produce the expected behavior. (@mikew)
Aliasing atom.cmd
Add an alias in your .bashrc alias atom='atom.cmd' (@DGempler)

Just started happening here today too
Atom 1.7.4
Windows 10
Git Bash 2.8.2

Can confirm as well
Atom 1.7.4
Windows 10
Git Bash 2.7.2

Happening on Windows 7 too.
Atom 1.7.4
Windows 7
Git Bash 2.8.2

I have got same situation
Atom 1.7.4
Windows 8.1
Git bash 2.8.2, 2.8.3
I guess atom . works on atom 1.7.3 because I alway use the command.
atom . works on Git cmd, windows command line.

Running atom.cmd . works as a workaround

Thanks @mikew . It works in my case.

@damieng :scream:

Yeah, looks like more subtle differences between the various flavors and versions of Bash on Windows. Sigh.

So it seems MSYS variants want //C on line 11 and one backslash on line 3.
Cygwin wants /C on line 11, double-backslash on line 3...

I have a workaround for the path from cygpath ending or not ending with a backslash between different flavors.
Not seeing a single method that's going to handle the /C however.  Open to any suggestions that are better than hard-coding uname -a signature based logic.

Apparently MSYS helpfully 'detects' /C everywhere and maps it to the C: drive path. This is what is causing cmd.exe to open an Window and not actually run the command on Git Bash.

Using @mikew 's suggestion, a temporary solution would be to add
alias atom='atom.cmd'
to your .bashrc or aliases file.

Some of the older bash shells won't actually run atom.cnd directly without
going via cmd.exe
On Mon, May 23, 2016, 16:28 Dmitry Gempler<EMAIL_ADDRESS>wrote:

Using @mikew https://github.com/mikew 's suggestion, a temporary
solution would be to add
alias atom='atom.cmd'
to your .bashrc or aliases file.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/atom/atom/issues/11817#issuecomment-221125382

Thanks for the quick fix @DGempler! This bug was driving me nuts, it's hard to change such an ingrained habit.

Fix went out in 1.8 and 1.9 beta.

Problem Still Persists
Atom: 1.8.0
Windows 10
Git bash 2.8.4

Yes, it looks like the fix didn't make the 1.8 branch.
Can you copy the contents of https://github.com/atom/atom/blob/dc6388323c55a577669eb92fcef5131477d34aad/resources/win/atom.sh into your atom.sh in resources/cli and see if that works for you?

It seems that resources/cli/atom.sh of app-1.7.4 and app-1.8.0 are the same.
After changing atom.sh as @damieng 's, atom . works in my case
Windows 8.1
Git bash 2.8.3

Fix provided by @damieng works +1

using windows 10, atom 1.8 and using the provided update works for me as well.
I was a little confused on what file to update at first because of the path of the file to update in git isn't the same as the installed path.
For others similarly confused update
/c/users/{{YourAccount}}/AppData/Local/atom/app-1.8.0/resources/cli/atom.sh
I then closed all instances of atom and git bash (had to use the task manager to kill a git for windows task as well) then started git bash and atom worked from the command line again.

Am having a problem opening Atom from Git bash on windows 7.
Windows 7
Atom 1.8.0
Git bash 2.9.0

@ ^ any help please!

@giresse19 The answer is right there above.  Copy the atom.sh from https://raw.githubusercontent.com/atom/atom/dc6388323c55a577669eb92fcef5131477d34aad/resources/win/atom.sh and replace the contents of /c/users/{{YourAccount}}/AppData/Local/atom/app-1.8.0/resources/cli/atom.sh with it.

@damieng Thanks but am still having some probs and will be grateful if u help out:
After typing: cp dir Desktop/file1/filetocpy.txt(location of the item to copy,i.e the link u send to me )  dir  c:/users/kasutaja/AppData/Local/atom/app-1.8.0/resources/cli/atom.sh.
i get the RESULT: cp: target 'app-1.8.0/resources/cli/atom.sh' is not a directory.
QUESTION: did I type in a wrong command on the Git bash, if yes, what should use?
Thanks in advance!

^what should I use**?

Try this from a PowerShell command line;
invoke-webrequest https://raw.githubusercontent.com/atom/atom/master/resources/win/atom.sh -OutFile $env:LocalAppData\atom\app-1.8.0\resources\cli\atom.sh

Can you paste the contents of your atom.sh here?

Can you paste the contents of your atom.sh here?


[UI] clr- prefix for css classes

Using postcss to generate two versions of css files:

one where every classname is prefixed with clr-
one where old classname and prefixed version coexist to allow the client app to migrate at their own pace

Note that the script order in build:ui matters, since one of them change the css files in place as opposed to generating a new file.
Signed-off-by: Jeeyun Lim<EMAIL_ADDRESS>

Just to follow up, we discussed keeping this open until Matt has the other PRs ready so we can see them all together first, but otherwise this is good to me.

this solution is problematic. we are pushing past 1.0 to find a better way to do this.


Added datasets for Predicting Credit Risk notebook series

opening a new request with an additional dataset.

@RMDK looks like this got prematurely "closed" without "merged". if you reopen, I can merge it in.


Adding QED, QEM, QEY and similar commands.

Hi!
For some time I'm using esphome-pipsolar, and it's awesome! The only thing I'm missing is the ability to count how many Wh did the inverter generate (obviously for some HA-graph greatness and tracking PV production).
Would it be possible to add here? I can test it, if i ever learn how to use a branch on docker...

Could you provide some more details about the commands? Do you have an idea which models does support these commands?

I don't advise to use the builtin statistics capacities of these inverter. There are not really accurate and you can only log infos up to the 01/01 of the current year.. Use a dedicaced PZEM or similar device.... and use a total_daily_energy sensor to integrate data in kWh ....

Hiho, thanks for your responses :)
My inverter is like this:

It shows daily/monthly/yearly/total generation, and I thought that since this is an additive value, it should be simple to push it into HA for graphing.
And I was thinking about getting just a rudimentary reading for now - since it's all in software, it could give me an approximate idea when/how much is the inverter producing, without any additional costs.
I also saw those commands listed in PDF document linked here:
https://github.com/syssi/esphome-pipsolar/issues/3#issuecomment-1036986944


Unable to add aws session token?

Hello,
I'm trying to add the aws session token to bosh create-env as "-v session_token" but I only get the message "AWS was not able to validate the provided access credentials".

Is this the correct way to inject the session token? I need to use the session because the account hast mfa activated...

@jhuntoo
I had to modify the "/instance_groups/name=bosh/properties/aws?" (file bosh-deployment/aws/cpi.yml):
path: /instance_groups/name=bosh/properties/aws?
value: &aws
access_key_id: ((access_key_id))
secret_access_key: ((secret_access_key))
session_token: ((session_token))
default_key_name: ((default_key_name))
default_security_groups: ((default_security_groups))
region: ((region))

Thankyou @TimonB that did the trick!
I've created this PR to address it the repo that's referenced in the getting started tutorial.
https://github.com/cloudfoundry/bosh-deployment/pull/246


Update current and past interns on the "About" page

Updated the About page to include some current and past SIParCS interns.
Also:

changed SIPARCS to SIParCS
removed a no longer valid NCAR username from the alumni list

Closes #108.

Ah, yeah.  I had intentionally left the positions as the most recent role (e.g. with Philip and Heather) since it seemed like that might be the norm (although perhaps that wasn't the case).
I don't have a terribly strong opinion on this, but it does make me a little bit nervous to keep track of some of this publicly without explicitly asking if people would like to opt out.  That said, I like the ideas of erring on the side of transparency and inclusivity/credit as well.  A little torn.  Not sure if other folks have thoughts on this @clyne @jukent.
I do think the multiple entries solution is a potentially bit duplicative and/or confusing (e.g. is a person really an alum if they are still staff, but in a different role?).  Happy to give the "multiple roles in the same designation" option a shot to see how that looks though if there's interest in including this info.
And I'll make sure to add Clare in - thanks for catching that!  Somehow missed her.

Oh I hadn't looked at that team page or realized interns were listed there. I know that they are on this page: https://vast.ucar.edu/siparcs/ which posts VAST SIParCS projects that are already publically available on the SIParCS site. Other internships could go here too.

I agree! One internship entry in the About page misled us but let us remove that one and instead have our interns listed only under the internship page

Ok, in that case I think I'll just close this, make a note in the corresponding issue, and make quick PR cleaning up the staff list.


[MonacoEditorReactComp] - Correct approach to store code in state

Any idea how to store editor text in app state? Because when state changed onTextChanged event is being triggered twice because of having code also in userConfig and lost cursor position in editor.
Examples and readme doesn't show usage with react state.
const [code, setCode] = useState('');

const onTextChanged = (text: string, isDirty: boolean) => {
  setCode(text);
};

<MonacoEditorReactComp
  userConfig={{
     wrapperConfig: {
      editorAppConfig: {
        ...
        code,
        ...
      },
    },
  }}
  onTextChanged={onTextChanged}
/>

I don't think Monaco should be used as a "controlled input"; I don't think it's designed to be used that way. Instead, you can use onLoad to store a reference to the editor itself in your component's state, and then call getValue() to get the current value whenever you need it:
const [editor, setEditor] = useState();

// example of getting the current value from the editor
const onClick = () => alert(editor?.getValue());

<MonacoEditorReactComp
  userConfig={{
     wrapperConfig: {
      editorAppConfig: {
        ...
      },
    },
  }}
  onLoad={(wrapper) => setEditor(wrapper.getEditor())}
/>

@vrama628 provided the answer. Thank you. I will close this issue.


Incorrect "string literal incompatible with empty" error when destructuring function params

Flow version: 0.98
Expected behavior
Destructuring in a function component parameter list works the same way as destructuring from the body of the function.
Actual behavior
The first scenario gives an error and the second succeeds.

Flow try link

The issue seems to be fixed a while ago.


[feat][ci] Check PR title with semantic

Signed-off-by: mango<EMAIL_ADDRESS>Master Issue: #15311
Motivation
Follow the guide of PIP 198 and Guide, we support a job to standardize PR naming convention using GitHub Action, but this PR finished only one task(Create “check PR title” job in GitHub check), other task involves modifying docbot, so we will commit another PR to solve.
Modifications
Add a job to check PR title with semantic GitHub Action
Verifying this change

[x] Make sure that the change passes the CI checks.

This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation
Check the box below or label this PR directly.
Need to update docs?

[ ] doc-required
(Your PR needs to update docs and you will update later)

[x] doc-not-needed
Modify github workflow

[ ] doc
(Your PR contains doc changes)

[ ] doc-complete
(Docs have been already added)

And I test it in my repo, please see https://github.com/mangoGoForward/C/pull/3 , more screenshot like blow:
Normal:

Missing scope:

Error scope:

Missing type:

Error type:

Please have a review when you have free time
@maxsxu @Anonymitaet

If this job fails, a reminder/error msg should be shown as below, similar to https://github.com/apache/pulsar-test-infra/blob/a30bb7884b796e9abb294adc8548406471bc3175/docbot/main.go#L18
Please follow [Pulsar Pull Request Naming Convention Guide](https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit#) to write a PR title.

Seems the action-semantic-pull-request cannot meet this requirement currently. @mangoGoForward might need to do more custom work here.

Yes, I will improve it.

Hi @mangoGoForward any blockers?

Hi @mangoGoForward any blockers?

Yes, seems we should write an new action in pulsar-test-infra to customize the error message instead of using semantic GitHub Action， like:
Please follow [Pulsar Pull Request Naming Convention Guide](https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit#) to write a PR title.

I will test it well and continue to commit

Hi @mangoGoForward any blockers?

Yes, seems we should write an new APP in pulsar-test-infra to customize the error message instead of using semantic GitHub Action， like:
Please follow [Pulsar Pull Request Naming Convention Guide](https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit#) to write a PR title.

I will test it well and continue to commit

Thanks @mangoGoForward Please feel free to reach out to me if you need any help.

Thanks @mangoGoForward Please feel free to reach out to me if you need any help.

Hi, @Anonymitaet @maxsxu @tisonkun ,I have been finished coding and tested it successful in my public repository. see the action and workflow, if it possible to move this action to pulsar-test-infra?

@mangoGoForward Thank you! I think it's pretty close.
May need to make the TitleUnmatchPattern as an issue comment to achieve the effect mentioned by @Anonymitaet before.

@mangoGoForward Thank you! I think it's pretty close. May need to make the TitleUnmatchPattern as an issue comment to achieve the effect mentioned by @Anonymitaet before.

OK

Closed by https://github.com/apache/pulsar/pull/17961, thank you all for your contributions!


Implement Lazy Loading of Submodules for faster import

Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Type of change
Please add a line in the relevant section of CHANGELOG.md to document the change (include PR #) - note reverse order of PR #s. If necessary, also add to the list of breaking changes.

[ ] New feature (non-breaking change which adds functionality)
[ ] Optimization (back-end change that speeds up the code)
[ ] Bug fix (non-breaking change which fixes an issue)

Key checklist:

[ ] No style issues: $ pre-commit run (or $ nox -s pre-commit) (see CONTRIBUTING.md for how to set this up to run automatically when committing locally, in just two lines of code)
[ ] All tests pass: $ python run-tests.py --all (or $ nox -s tests)
[ ] The documentation builds: $ python run-tests.py --doctest (or $ nox -s doctests)

You can run integration tests, unit tests, and doctests together at once, using $ python run-tests.py --quick (or $ nox -s quick).
Further checks:

[ ] Code is commented, particularly in hard-to-understand areas
[ ] Tests added that prove fix is effective or that feature works

Tried handling wildcard (i.e. *) imports with this function but wasn't able to, have to explicitly define each attribute to be imported separately (for expression_tree mostly), Happy to have some suggestions here.

I think the importlib.util or pkgutil modules from the Python standard library might have something that we could use. Ideally, we should define everything that should be available to the user namespace in a list named __all__ for every package and module, we haven't done that but this PR could be a good start for that, PyBaMM-wide.
This might or might not help to construct a function whose output can provide lazy_loader with importable paths for all submodules:
import pkgutil
import pybamm

print([i for i in pkgutil.walk_packages(pybamm.__path__)])

returns
paths to modules under pybamm/

[ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='batch_study', ispkg=False),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='callbacks', ispkg=False),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='citations', ispkg=False),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='discretisations', ispkg=True),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='doc_utils', ispkg=False),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='experiment', ispkg=True),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='expression_tree', ispkg=True),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='geometry', ispkg=True),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='input', ispkg=True),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='install_odes', ispkg=False),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='logger', ispkg=False),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='meshes', ispkg=True),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='models', ispkg=True),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='parameters', ispkg=True),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='parameters_cli', ispkg=False),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='plotting', ispkg=True),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='settings', ispkg=False),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='simulation', ispkg=False),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='solvers', ispkg=True),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='spatial_methods', ispkg=True),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='util', ispkg=False),
 ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm'), name='version', ispkg=False)]

You should be able to run this on packages under pybamm.* too. This is under pybamm.expression_tree as an example:
print([i for i in pkgutil.walk_packages(pybamm.expression_tree.__path__)])

returns
For the expression tree

ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='array', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='averages', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='binary_operators', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='broadcasts', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='concatenations', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='exceptions', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='functions', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='independent_variable', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='input_parameter', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='interpolant', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='matrix', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='operations', ispkg=True)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='parameter', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='printing', ispkg=True)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='scalar', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='state_vector', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='symbol', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='unary_operators', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='variable', ispkg=False)
ModuleInfo(module_finder=FileFinder('/Users/agriyakhetarpal/Desktop/PyBaMM/pybamm/expression_tree'), name='vector', ispkg=False)

and you can now manipulate this import machinery to describe your own module paths by joining the module_finder and name attributes for pybamm.util.lazy_loader! This might or might not work for our case (but at least TBH this is a great start for diving into all of the core details and the fundamentals of Pythonic imports, and I managed to learn something too!).

I was reading up on making imports faster and I think we don't need tor essentially redefine the import pybamm mechanism by using lazy_loader everywhere, just making it faster would be great in itself. i.e., we can segregate what is causing a slower import and target that module in specific for a lazy-import, while keeping the other imports as is. For example, something like
from .expression_tree.operations.jacobian import Jacobian

or
from .util import (
    get_parameters_filepath,
    have_jax,
    install_jax,
    have_optional_dependency,
    is_jax_compatible,
    get_git_commit_info,
)

will never be slow, but something like this
from .models.submodels import (
    active_material,
    convection,
    current_collector,
    electrolyte_conductivity,
    electrolyte_diffusion,
    electrode,
    external_circuit,
    interface,
    oxygen_diffusion,
    particle,
    porosity,
    thermal,
    transport_efficiency,
    particle_mechanics,
    equivalent_circuit_elements,
)

has too many things going on in one statement. I figure that it will be just the *-imports and those for the battery models like the above one that are currently causing the bottleneck, so it would be great if we can do some profiling in this PR to gauge what parts are slow – we can explicitly focus on those and leave the rest (and leave out the parts that cause unit and integration tests to fail – just a case of tedious trial and error).
It might be worth scouting SciPy to see how they manage it (despite being a large monorepo, import scipy takes just about a second at most). In the footnotes for SPEC-0001, I found this project that can help automate a few things related to *-imports: https://github.com/Erotemic/mkinit
N.B. For profiling to be effective, we will have to delete the __pycache__ directories and their files (compiled bytecode) everywhere since Python imports get cached very efficiently.

Thanks a lot @agriyakhetarpal, i'm looking accordingly to your suggestion into the segregation first and getting back in some time.

I might have solved it: for modules that further have sub-modules (pybamm.models.*), we might need to modify every __init__.py file, not just the base one. I tried this command in the root directory which set everything up for me automatically:
mkinit --lazy_loader --inplace --recursive pybamm  # also accepts --noall so as to not add __all__ in the files

and I presumably removed the import caches (all __pycache__ folders) recursively by doing
rm -rf $(find . -type d -name '__pycache__')

But as I understand, this still does not remove all caches – so %timeit importlib.import_module("pybamm") in a interactive REPL might be returning an incorrect calculation and could be unreliable. It does feel faster, though – but it could be a case of placebo.
Outputs from importing PyBaMM
Without lazy_loader
%timeit importlib.import_module("pybamm")

The slowest run took 13.77 times longer than the fastest. This could mean that an intermediate result is being cached.
2.33 µs ± 3.05 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

With lazy_loader
%timeit importlib.import_module("pybamm")

The slowest run took 11.64 times longer than the fastest. This could mean that an intermediate result is being cached.
2.29 µs ± 2.84 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

This isn't helpful TBH, so I started deleting the entries from the sys.modules dictionary:
import sys
keys_to_delete = [key for key in sys.modules if 'pybamm' in key]
for key in keys_to_delete:
    del sys.modules[key]

in addition to removing the __pycache__ directories, but I haven't found a substantial increase yet. Maybe this can help act as a precursor for your experiments as we go further. A reliable method, though a bit tedious, would be to re-clone the repository, re-install it from source, apply the lazy_loader changes, and then benchmark the import pybamm statement – maybe try that and see what you get?
P.S. You can consider putting out a testimonial (https://github.com/scientific-python/lazy_loader/issues/50) after we manage to do this

This does the work, thanks for suggesting @agriyakhetarpal.
The import for me is superfast now as compared to before 🎉
Without lazy-import :
The slowest run took 7.07 times longer than the fastest. This could mean that an intermediate result is being cached.
58 µs ± 51.7 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

With lazy-import :
The slowest run took 14.75 times longer than the fastest. This could mean that an intermediate result is being cached.
2.74 µs ± 3.74 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

However, there are a number of CI failures raised due to big change which would be fixed iteratively.

But as I understand, this still does not remove all caches – so %timeit importlib.import_module("pybamm") in a interactive REPL might be returning an incorrect calculation and could be unreliable. It does feel faster, though – but it could be a case of placebo.

I am also encountering an issue with %timeit importlib.import_module("pybamm") returning an incorrect calculation(might be due to cache), however I can easily recognize a clear difference with or without lazy-import while importing PyBaMM.

Now as I'm trying to fix CI failures, I have first tried to resolve imports in the __init__.py files but I am realizing that it is leading to again defining imports like before recursively plus it is not ideal.
Other way around is to fix the imports in code i.e

pybamm.Symbol should be pybamm.expression_tree.Symbol
pybamm.Parameter should be pybamm.expression_tree.parameter.Parameter
pybamm.multiply should be pybamm.expression_tree.binary_operators.multiply
& so on

But this approach is leading to big API change. So I'd be more than happy to have suggestions here & also would like to know if i'm missing any page and can try something else instead too.

Definitely don't make that API change. Also having to keep the __init__.py files up to date like that will make development more challenging. What are the benefits of lazy loading that would justify this?

Usually import pybamm can take you 15 to 20 seconds in a Jupyter notebook (mostly the case for me) – lazy loading is supposed to just import PyBaMM and then import the modules under it at runtime dynamically when they are first used in a script.
I don't think we are using lazy_loader correctly, since it was defined for scenarios like this (to not bring a breaking change in terms of the public API). Can we skip lazy loading pybamm.Symbol and others (whatever ones break the tests)?

Without lazy-import :
The slowest run took 7.07 times longer than the fastest. This could mean that an intermediate result is being cached.
58 µs ± 51.7 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

With lazy-import :
The slowest run took 14.75 times longer than the fastest. This could mean that an intermediate result is being cached.
2.74 µs ± 3.74 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

One known benefit of lazy-loading is reduced import time and it is also expected to improve performance as Some of the Calculated attributes or attributes that are loaded are using an expensive operation i.e. from .expression_tree.symbol import *

Can we skip lazy loading pybamm.Symbol and others (whatever ones break the tests)?

I've tried doing that but it was leading more imports to skip lazy loading in a chain but yes we can try this to reach at point where there are no or minimum failures and least import time.

Closing this as discussed in the last developer meeting.


Fine tuning

Is that possible to configure the recognition engine with custom vocabulary for terms, names and abbreviations?

Right now, it's theoretically possible if you're willing to write some NLP code. Buzz supports Hugging Face models, so you can train your models (https://discuss.huggingface.co/t/open-to-the-community-whisper-fine-tuning-event/26681) and import them into Buzz. But, yes, there isn't a way to load your own custom vocabulary in a way that is practical for most people.


424 add more binary contingency table metrics

This now replaces https://github.com/nci/scores/pull/425 as I am now merging in from my fork.
I have made updates to the docstrings and rebased onto the latest develop since the last review.

@tennlee can you please push this to 228 for checking?

Looks good at a glance - I'll double-check formulae and build the docs etc tomorrow


Monitoring of the server

I need to improve the page http://qbuissondebon.info:10041/zabbix/screens.php?sid=b8e415c489a9eb38&fullscreen=1&elementid=22&groupid=0&hostid=10110 wich is available on http://minetestforfun.qbuissondebon.info/ > LE JEU > Statut du serveur
Need to be addded
The actual graphs/screens aren't suffisant, i need to add a graphs/screens for :

[x] service Minetest-server uptime (with trigger if down more than 30minutes)
[x] number of players online (with a special command, the players online in a 5minutes refresh is available on the file minetest@/home/quentinbd/minetest/worlds/minetestforfun/online-players, the players online begin at line 4 and the pseudo is the first varchar, 1 player per line)

Need to be tweaked

[x] In a second time, i need to improve the title of some graphs/screens to be more explicite

Other details
This graphs/screen need to be available publicly with the user "guest"

Done, new link : http://qbuissondebon.info:10041/zabbix/screens.php?sid=1472310e18f8c756&fullscreen=1&elementid=22&groupid=9&hostid=10110


Rendering of buttons in Firefox broken

When opening the docs examples in Firefox on iOS/iPadOS, all buttons are not being rendered correctly:

In mobile Safari the buttons show up as expected.

Thanks for reporting ... Sorry, I could not reproduce this error so far, but it looks like a CSS problem, where the webfonts could not be loaded. I guess you already tried a hard reload?
If the problem persists, could you please share a screenshot of the developers console? I don't own an IPad, so maybe following some of the mentioned steps here:
https://discourse.mozilla.org/t/how-do-i-debug-firefox-on-ios/111414/4
could shed a light on the problem :see_no_evil:
Thanks a lot ...

Ah, the culprit was the (default?) setting: "block web fonts" has been active. Once this is deactivated, everything works as expected. Sorry for bothering you. Thx for this interesting project!


UG spacing

Codecov Report

Merging #353 (536a803) into master (699c022) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 536a803 differs from pull request most recent head 1d12b6e. Consider uploading reports for the commit 1d12b6e to get more accurate results

@@            Coverage Diff            @@
##             master     #353   +/-   ##
=========================================
  Coverage     71.23%   71.23%
  Complexity      807      807
=========================================
  Files           129      129
  Lines          2632     2632
  Branches        327      327
=========================================
  Hits           1875     1875
  Misses          667      667
  Partials         90       90

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 699c022...1d12b6e. Read the comment docs.